Just sharing this MIT Technology Review article on digital archives, a cause that’s close to our hearts: https://www.technologyreview.com/2024/08/19/1096284/data-archives-archeologists-tiktok-future-wayback-machine/?src=longreads
It is always a good occasion to mention: donate to the Internet Archive, it is such an important resource, and it’s constantly under attack. Donating to them was the first thing I did when I got my first wage (rapidly followed by Wikipedia).
Read the latest developments on the rules regarding Open Access and cultural heritage in the Italian legislative system: https://www.wikimedia.it/news/il-cammino-verso-lopen-access-in-italia/
Our own position is that eventually openness will win out, simply because the state is trying to charge for something that people are not willing to pay for, and the state has very little enforcement power over the open web (they should have even less).
People expected big changes after the release of the Apple Vision Pro, and some are visibly disappointed. Apple tends to have a big effect on the mainstream, lots of people seem to think that they invented the smartphone, when the iPhone came out I was using my third smartphone.
XR technology has been commercially available for 30 years, it tends to progress in the same way that people go bankrupt: gradually, then suddenly.
We’re seeing some pretty interesting gradual changes, particularly in terms of software platforms. The Apple partnership seems to really have pushed Unity into high gear, they’re releasing a lot of resources, both on the editor side and on the design side. Sadly some of the best stuff is reserved for paying customers, and it’s pretty expensive.
This seems to also have changed the thinking on the Sony (with the opening up of PSVR2 to PC users) and Meta. In just a few days Meta announced a radical opening up of their efforts.
First of all they announced a new educational project, which will probably increase the demand for the kind of work we do quite a bit. The fundamental problem is that they keep using the “metaverse” word, and quite a few people have a very negative reaction to it. The concept is really not that clear, but the educational uses of VR are real, hopefully if there is an educational sales push it will also benefit us.
The most important announcement is the opening up of their operating system. It will be possible to buy compatible headsets from other manufacturers, which should keep prices affordable without selling hardware at a loss, and even more importantly they’re opening up the app store. Multiple stores will be available, which could several important advantages. We could have a cross-platform cultural heritage app store which encourages reuse. Or more prosaically, maybe I could use some of my SteamVR purchases.
Zuckerberg is explicitly aiming at providing the Windows of the XR market. At the same time, these changes should make them entirely Digital Markets Act compliant.
Last year a mandatory fee regime was introduced for images of publicly-owned cultural heritage artifacts. This is morally wrong, but it is also impractical.
A new regulation came out in March modifying the rules and introducing exemptions. Wikimedia Italia wrote about it on their Arkivia blog.
Academic works and news are now exempted from the fees, and so are catalogues printed in less than 4000 copies.
The bureaucratic abomination under-girding the system, however, remains in place.
There are several use-cases for algorithmically generating music, and indeed there is a long history of both hardware and software in this field. Back in the 80s there was even a significant professional backlash against MIDI and drum machines.
For example, it would be quite handy to generate anechoic sounds for acoustics listening tests, so that they can be convolved with the Impulse Responses of simulated environments.
The current mania is randomly generated media, which is an offshoot of a longer-standing mania for using media as filler instead of as a signifier. A lot of what we see and read is just Lorem Ipsum: posts need an image to do well, so people just slap a random image in them. It’s algorithmic garbage causing other algorithmic garbage. One of our deepest worries regarding our own work is that we could be hired to do it just because immersive technology sounds cool, and not because it genuinely adds value to the project at hand.
Anyway, of course there are going to be attempts at using the newest technology for generating music, and here are three of them:
There is also an OpenAI product, but it’s not generally available.
The increased repetitiveness in commercial music is definitely not an hindrance for these efforts.
Fundamentally, all of the do what they say they will do, but we do wonder why they focus on directly generating sounds instead of generating MIDI.
Check out this Ars Technica syndacation of a Financial Times article: https://arstechnica.com/ai/2024/04/power-hungry-ai-is-putting-the-hurt-on-global-electricity-supply/
This is fundamentally an effect of computing getting increasingly centralised in massive data centers (just think of government over-reliance on Microsoft), which has noticeable economies of scale.
With AI there is also a significant divergence between the cost of executing the software, and the cost of training the software. The current AI boom is largely focused on just throwing resources at relatively “dumb” approaches, which is working better than it could be reasonably be expected for now, but that will inevitably run into resource limitations. For now, that bottleneck is energy, which is very bad news in environmental terms.
The silver lining is that if we managed to make companies pay for their environmental impact we would make smarter, more correct, approaches even more competitive.
The always excellent 3Blue1Brown has released two new videos in his Deep Learning series, explaining Transformers. Here is a link to the whole series of videos: https://www.youtube.com/watch?v=aircAruvnKk&list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi&index=1
We’re working on the immersive component of the BCC Innovation Festival: a startup event that we won back in 2022.
There are still around two weeks to participate, and we’re aggregating a little bit of media coverage. Mostly it’s because we like seeing our own work (the 3D model of the low-poly Rodin statue) featured so publicly.
The Gazzetta di Napoli newspaper wrote about it on the 6th of March: https://www.gazzettadinapoli.it/economia-2/industria-4-0/al-via-la-terza-ediz
ione-del-bcc-innovation-festival/
Emilia-Romagna Startup, the regional government’s startup promotion agency, wrote about the festival today: https://www.emiliaromagnastartup.it/it/innovative/articoli/2024/04/bcc-innovation-festival-2024-percorso-progetti-e-idee-innovative
There is time until April 30th 2024 to sign up. The whole process was very useful for anyone starting a company, they provide plenty of educational resources and mentorship.
I’ve learned about this amazing exhibit about reversos at the Prado: https://hyperallergic.com/869083/prado-show-reveals-the-hidden-artworks-on-the-backs-of-masterpieces/
It is truly exhibit design at its best, but it gave me the idea of digitizing not only the back of paintings, but the very structure. I’d like to build an exhibit in which we show how the painting is made, how it looks from every direction, how the frame is built, and even how the canvas has been constructed.
There is also this great video from Christie’s about the back side of paintings, and all the information they can reveal: https://www.christies.com/en/stories/what-to-look-for-on-the-back-of-a-painting-821e1638846944fb9a2b05aad53854fe
The Apple Vision Pro launch seems to have brought a lot of of interest, which is to be expected when Apple does, well, anything.
They even came up with their own marketing-infused grammar, they want people to say Apple Vision Pro, never the Apple Vision Pro.
Some of the discourse has veered into worries about people wearing visors while driving, luckily those are just a handful of attention seekers.
First of all, how is it built? iFixit to the rescue:
It would seem that finally the screen resolution and clarity are sufficient for the “lots of virtual screens” use case. Last year Karl Guttag evaluated the angular resolution of the Apple Vision Pro as between 35 and 40 PPD (pixels per degree). In the above iFixit video it’s measured at 34 PPD, so that was spot on.
Basically if you are at all interested in the optics side of thing, just read everything Guttag writes: https://kguttag.com/tag/apple-vision-pro/
Also check out the Ars Technica review.
On the development side, Unity support is now out of beta, but it is only available for Pro users ($1800/year). So for most developers the choice is between the native Apple SDKs and WebXR. The latter enables us to develop once, and run on every device, so it is clearly superior.
There has also been some debate on Spatial Video, namely whether it is simply a stereoscopic image, or if there is some parallax magic. As always with immersive video, Hugh Hou has the last word:
Another important issue with the Apple Vision Pro are the optical inserts, Eric Cheng wrote a status of optical accommodation in the whole industry.
Casey Neistat’s failure to activate travel mode was very funny, but it was not a very good review, so we’re not going to link it.
Joanna Stern tried some interesting “spatial” uses. In particular the multiple timers are a classic Ambient Computing idea.
The Verge’s review is great at emphasizing all the ways in which the Apple Vision Pro pushes the limits of our current computing environment.
Stratechery, Daring Fireball, Wait But Why, Hypergrid on VR Gentrification, Road to VR,
Personas demo:
MKBHD did a four-parter:
The fundamental distinction is between “windowed” apps, that are limited on a virtual screen, and actual VR apps. The beauty of Reality Kit is the ability of having spatially-aware elements in apps that still work with passthrough.
Soul Spire, Shortcut Buttons, Now Playing,
Aro.work: A webXR workspace
Everyone seems very enthusiastic about Juno, a third party YouTube client. The whole thing reminds me of Windows Phone.
From Road to VR: 8 Great Vision Pro Apps to Download First
The Apple Vision Pro orders just opened (US only), and they also uploaded a new guided tour, which gives us a chance to reflect on what kind of experiences Apple is putting front and center.
In the VR community there has been much ado about Apple’s choice to entirely refuse the industry jargon: there is no VR, MR, XR, AR, it’s all spatial computing. They’ve been accused of making up new words to obfuscate what they’re offering. We think it’s more subtle than that.
First of all, naming things matter. Some of the experiences we build can be considered part of the “metaverse”, but that’s not a term we use, because it doesn’t have a good technical definition, and because it reminds people of sleazy operations. Some think of the Meta advertisements, some think of literal scams. Those scams that are so pervasive that I don’t dare to mention them, lest I attract their spam.
Apple systematically chooses feature names that represent the value added for the user’s experience, while simultaneously obfuscating the technical details. They don’t want us to know the DPIs of a screen, they just tell us it’s Retina. It doesn’t matter very much to Apple that competitors make screens with better resolution, if it’s Retina it’s good enough, you’re not going to see the individual pixels. It’s annoying for us nerds, but it seems to work just fine.
In some ways it’s similar with Spatial Computing, it’s technically indisputable that the Apple Vision Pro is a VR headset with a Mixed Reality passthrough function, just like the Meta Quest 3. But what Apple is doing is not only obfuscation, and they certainly didn’t come up with the term. What they’re trying to communicate is that the advantage of computing on a headset rather than on a laptop is going to be spatial in nature. Spatial computing is not a new term at all, the essays written by Timoni West at Unity were an inspiration for my dissertation, and then for the founding of this company. The fundamental idea is to use the dimensionality of the interface for communication between the user and the computer. Information doesn’t have to be limited to a 2D screen, and most importantly user input doesn’t have to be limited to discrete actions. It’s still hard work, but we can enable natural interactions that allow us to think gesturally. Let me gesture that I want a piece of machinery to be ✋ this 🤚 big, and have the computer figure out how big that is in numbers. This is the kind of use case that Sony is targeting with their new VR headset.
Looking at the Apple developer documentation, a lot of work has gone into making the creation of this kind of experience possible, but they’re not focusing their presentation on that at all. All that the user is seen doing is positioning virtual windows around them.
This is extremely similar to what was possible with Windows Mixed Reality or with the Quest. On the Hololens the visual fidelity was just not enough, and the field of view was tiny. On the Quest 2 the visual fidelity was still too low, it is much improved on the Quest 3, but it is still somewhat worse than just staring at a screen. Looking at the specs the Apple Vision Pro might finally be able to pull of this use case.
However, it’s not at all the kind of spatial computing that I described above.
Instead, it harkens back to the old days of the classic Mac Finder, and in some ways to the design of Jef Raskin. The idea is that rather than following the hierarchical organization of the file system, the interaction between the user and the computer will determine a spatial organization of the information that is being worked on, that will mirror a spatial conceptualization in the user’s mind.
A lot of the modern window management solution that Apple has added to both the Mac and the iPad, like Stage Manager and Mission Control, have enabled users to experiment with this kind of paradigm. There is also a $9.99 Mac spatial desktop environment called Raskin as an homage. Raskin’s son Aza also developed Tab Candy, which brought spatial tab management to Firefox.
I think the verdict is still out on how useful this kind of spatialization is, but I think it’s clear that it is part of what Apple is going for. With any immersive technology we must always remember that the real immersion is in the user’s mind, we can feel present in a novel as much as in VR. As a company we are sticking to the other kind of spatialization, where the spatial information is intrinsic to the problem domain, whether it’s reproducing historical artifacts, simulating the propagation of sound in an environment, or assembling a machine in the correct order. Apple Vision Pro makes both of them much easier to develop. You know, when we’ll actually get one here in Italy 😀
This ended up being a pretty weird post, juxtaposing technical releases and legal developments, we don’t know if we really managed to give it a coherent shape, but this is very much the complex space in which the future of digital humanities, and indeed the future of everything we do, is being shaped.
There are a lot of powerful actors that are working to shape it in their favor, and they are not being secretive at all.
We’re working on a couple of audio-guide projects, and in one of them it would be really convenient to automate some of the work with an AI. Our current attempts are technically presentable, but honestly pretty boring. Think something like the Spot guide-dog from Boston Dynamics (what is up with their microphones?), but on your phone and without installation. We have built it, it works, we do not think it’s good enough, just like automated translations aren’t really good enough, compared to a good professional translator. Needless to say, we’re keeping our eyes peeled for the latest advancements. After all AI is guaranteed to disrupt us.
Large Language Models are tools, and it’s up to us how we use them, from Giant Robots Smashing Into Other Giant Robots.
The EU Council and Parliament have reached an agreement on the new AI ACT. Before the agreement OpenFuture had written about copyright opt-outs, about friction and governance and about self-regulation.
Sticking to AI, it seems that Google is catching up to ChatGPT, confirming our “there is no moat” position. However, just like Threads and Horizon Worlds, it is not coming to the EU yet, which we consider a worrying trend.
We suspect it’s going to end up just like cloud computing: there is a lot of money to be made, but it’s going to be so capital-intensive that only a few big players will dominate the market, developing useful products that are hamstrung by the imperative to build in vendor lock-in. Luckily there are some really interesting developments on the local deployment side, and we think that the transparency and user control of locally run Open Source software are going to be extremely important.
Mozilla published a guide for new AI developers last month, but last week they also published Llamafile, which combines llama.cpp and the brilliant Cosmopolitan library to distribute Large Language Models as single files.
In a similar vein Noiselith allows us to run Stable Diffusion XL and generate images on local hardware, with a user-friendly setup.
Voicemod allows to change your voice in real time, both to existing voices and to newly generated ones. We do not agree with The Verge’s disregard for the legal aspects of voice cloning, rights limiting the reproduction of one’s likeness are well established, copyright is not the issue.
Apple released a Machine Learning framework for Apple Silicon, and they literally just pushed it on GitHub without announcing it. Of course nothing that Apple does passes unnoticed, the tech press was instantly on it.
The ex-Apple employees behind Shortcuts have a new desktop AI startup.
Meta, IBM, Intel, and around 50 other organizations launched an alliance for Open Source AI.
Mistral AI, the French company that releases Apache-licensed models (their benchmark scores are pretty amazing), has received a $2B evaluation.
Meta has a new AI trust and safety initiative, but at a glance it seems like it’s mostly focused on spotting content that does not align with what the owners want, which is obviously super important.
The most important article on AI trust you’ll read this week: AI and Trust by Bruce Schneider.
Getting back to EU legislation, Felix Reda (of Pirate Party fame) and Justus Dreyling wrote about the need for a Digital Knowledge Act, highlighting the need to achieve something that is very much in line with our mission: allowing every research question to be done online. It should never happen that a document or a resource is held in a public archive or is made with public money (we’re thinking of scientific papers) and not be made available, online and for free.
Cory Doctorow wrote on the evils of DRM, and that is always important to keep in mind when building digital archives and collections.
The EU Data ACT is also moving forwards, with some good stuff (harmonization and foreign transfers, in particular) and some really concerning bits. Offering legal protection to trade secrets favors some specific players, but it is opposed to the hard bargain that makes patents exist. The current discourse seems to have lost sight of the fact that “Intellectual Property” is not property at all, in the abstract all knowledge should belong to every human being, we have setup a legal system that exchanges a temporal monopoly for technical information (in the case of patents) or as an incentive for the creation of more cultural works (in the case of copyright). Trade secrets should not be legally protected, if you want protection you should use patents. The current Data Act agreement also restricts reverse engineering, which is terribly harmful for innovation and competition.
Work has also been progressing on the EU Cyber Resilience Act. It seemed to go in a weird direction for what concerns the intersection between security and Open Source, but they seem to have already fixed the most glaring issues. This is a good chance to recommend following Bert Hubert, he always has insightful articles and up to date news. For example that’s how we learned about this case in which Sony attempted to strong arm a DNS provider.