Apple's WWDC 2023 Keynote Highlights Innovation In Machine Learning

When you think about tech companies and their role in advancing machine learning (ML) and artificial intelligence is Apple on your list? Let’s be honest, the biggest and splashiest research in the last eighteen months has come from companies like OpenAI (the GPT family of models, DALL-E 2), the Google Brain Team (Bard, Imagen), and Meta (SAM, LLaMA). But after watching Apple’s WWDC 2023 Keynote and thinking about how Apple applies ML in their software I’ve decided they deserve a spot on the innovators list as well. Let’s go over some of the announcements from the keynote and I will explain my reasoning. FYI, there’s a lot more to it than the reveal of the Vision Pro.

New Mac Hardware

Apple has always had strong ties to the liberal arts and their new lineup of hardware reflects that continued relationship. New models of Mac Studio and Mac Pro were announced that can be configured with Apples latest M2 Max and M2 Ultra chips. Clearly these machines are being marketed primarily to companies in the media industry. They said as much as they went over performance enhancements in video editing software like Adobe After Effects and dropped names of high profile customers like NBC’s Saturday Night Live.

But there was also a brief moment where Apple boasted about the new M2 Ultra chip and applications in ML.

“… in a single system it can train massive ML workloads like large transformer models that the most powerful discrete GPU can’t even process because it runs out of memory.”

— WWDC 2023 Keynote

For those not entrenched in the latest GPU specs, NVIDIA’s top of the line consumer GPU, the RTX 4090, has 24GB of VRAM. When put along side a maxed out M2 Ultra with 192GB of unified memory (shared across all compute tasks) it’s clear you can train larger models with these new machines, but was Apple trying to dangle a carrot to ML practitioners?

There is a lot to unpack here. Just three years ago Apple was still relying on Intel for their CPUs and AMD for their GPUs. I had contemplated using my iMac and an external GPU for ML at that time, but with little software support I ended up building a custom Ubuntu rig instead. However, the technological landscape has changed dramatically since that time. Metal, Apple’s graphics library, has been incorporated into Pytorch and Tensorflow, the two most popular deep learning frameworks, adding GPU accelerated training to the Mac. Apple has also completely transitioned to their own silicon and the latest M2 Ultra chip can be configured with a 24-core CPU, 76-core GPU, and 32-core Neural Engine. So the real question is how does the M2 Ultra stack up against NVIDIA’s GPUs and can I get real ML work done on a Mac?

To answer these questions what we really need are benchmarks, and it turns out, we have some! The folks over at Weights and Biases have been tracking the performance of Mac silicon for a while now. Here is a chart comparing speeds when training a Resnet50 model (23.5M parameters) on the Oxford Pets dataset.

The M1 Ultra 64 GPU core system, Apple’s fastest in this benchmark, is about 3x slower than NVIDIA’s RTX 4090, and approaching the same speed as an RTX 3050 Ti laptop GPU. Before you get too disappointed let me point out a few things. The Ultra series of chips are actually two chips stuck together, so the M1 Ultra 64 is two M1 Max chips and the new M2 Ultra will actually be two M2 Max chips. Using the data from our benchmarks I calculated a 1.7x speed-up going from the M1 Max to M1 Ultra. If we see the same performance improvement going from the M2 Max to the new M2 Ultra, then I think we can expect the M2 Ultra to sit nicely between the Tesla T4 and the GTX 1080Ti. Now we are getting somewhere. In addition, Apple’s silicon is designed to be extremely efficient and uses only a fraction of the wattage usually drawn by NVIDIA GPUs.

To finish the hardware part of the discussion let’s circle back to the quote from the keynote. Unified memory will indeed allow you to train large models on Apple’s silicon as suggested in the keynote, but the benchmark data shows we should be realistic about how long it might take on a Mac. I think most folks are going to stick with NVIDIA for ML workloads, but fine tuning and tinkering with smaller models are definitely things you can do on this new hardware. I believe Apple’s true interest in ML is less about fast hardware and more about applications in software, which we will go over next.

New Software

The keynote this year was literally chalked full of technology utilizing machine learning. I had trouble keeping track so I made a list:

• realtime transcription of incoming voice mail in the Phone App

• transcription of audio messages in the Messages App

• pet recognition added to Photos App

• keyboard autocorrection is now powered by a transformer model…what the duck

• dictation also uses a new transformer model

• ML powered inspiration for writing entries in the new Journal App

• automatic textfield identification in PDFs

• presenter overlay in video conferencing

• adaptive noise canceling and conversation awareness on AirPods

• AirPlay learned preferences makes hardware suggestions about device pairing

• Smartstack on WatchOS uses ML to show relevant info when you need it

As you can see the list is fairly long, but only covers what was mentioned in new features for this year. ML has become heavily incorporated across Apple’s software. It’s not just sprinkled in a few applications anymore. What’s more, although Apple is much less prolific in terms of academic contributions, they have been busy innovating in their own way. Apple’s ML models run locally on-device where energy efficiency, size, and speed are paramount. And of course we, as end users, also want accuracy. It’s really hard to get all of these things at once and there are always trade-offs.

Direct evidence of innovation, aside from a phone that can do cool things, is hard to come by. Apple tends to keep a lot of their tech and code private, but for those that are interested there is an area on their website highlighting publications and advancements in ML. One of their posts titled On-device Panoptic Segmentation for Camera Using Transformers is a perfect example.

Here, they developed a model that can separate elements from a scene (people, sky, etc.) along with subcomponents such as skin and hair. To paraphrase the article, their technique is fast enough to run in realtime, compact enough to run on mobile, and has a minimal impact on battery life. Work like this isn’t drawing headlines, but it’s being used in millions of devices.

Apple VisionPro

Generated using Imagine

And now we get to the elephant wearing VR goggles in the room. If you watched the keynote I think you will agree that the VisionPro looks like something from a sci-fi movie. The interface uses your voice, eyes, and hands to control applications that appear to hover in front of your furniture. It looks amazing. Clearly ML is being used everywhere. Gestures, pose, and voice in the UI are powered by ML. Detecting people entering your field of vision requires ML, and reconstructing your face so you don’t look like the guys from Daft Punk during FaceTime calls requires ML.

Are the underlying ML models innovative? In terms of the basic building blocks probably not. But as I mentioned earlier, getting models that run in realtime on a device with resource and power constraints is.

Summary

It wasn’t the palpable notes of ‘machine learning’ and ‘neural networks’ sprinkled in with the announcements, or the reveal of the Vision Pro that put Apple on the ML innovators list for me today. It was the realization that after years of quietly investing, developing, and experimenting, Apple has become a leader in deployment of ML across their software and hardware ecosystem.

Parsing Character Entities from HTML/XML Content In Swift

The Web today is a wonderfully addictive mashup of culture, commerce, and technologies, both old and new. As an iOS developer, interacting with the Web is usually trivial. Make an endpoint request of a REST API from a web-server, get back data, decode data. Boom. Done. At least, that’s what I thought, until I ran into odd substrings like ‘"’ and ‘&’ in an xml formatted response I was parsing. These curious bits of text are known as character entity references (CERs) and in this case, stand for the quote ( “) and ampersand (&) characters respectively. In this article I will provide a bit of background about why CERs exist for us non-web developers and give a few practical methods for decoding them in Swift.

What are Character Entity References and why do they exist?

As I mentioned at the start of the article CERs are basically character codes within a string that are sandwiched between an ampersand and semicolon. Your web browser recognizes these codes and automatically replaces them with the appropriate character for rendering on your screen. So, HTML with ‘5 &lt; 6’ renders to ‘5 < 6’. The Worldwide Web Consortium (W3) has a spiffy interactive chart you can have a look at if you are interested in seeing more. For those that want to dig deep you can read the wiki entry or have a close look at the official html spec.

Character Entity References exist for a number of reasons:

1. To allow for inclusion of reserved characters in HTML. Just like any programming language there are characters that are reserved for the language itself…most of us have at least seen at least a little HTML. Each tag begins with < and ends with >. Is it any surprise these characters are reserved?

2. To allow for characters not included in the encoding format of the document (90% of web page are UTF-8 these days, so I think this is mostly for edge cases).

3. As a convenience to the document writers (web devs) for characters that aren’t included on a standard keyboard. Writing ‘&copy;’ is a lot faster and more efficient that going fishing for the copy-write symbol in a special characters library.

Dealing with CERs in Swift

Now that we have the background out of the way I will show you two methods you can use to ‘find and replace’ CERs in Swift.

Option 1: NSAttributedString

If we dip into ObjectiveC (not very Swift-like, I know), NSAttributedString already has a lot of functionality built around parsing html. Here is an alternate initializer for String that handles CERs:

extension String {
    init?(htmlEncodedString: String) {
        guard let data = htmlEncodedString.data(using: .utf8) else {
            return nil
        }
        let options: [NSAttributedString.DocumentReadingOptionKey: Any] = [
            .documentType: NSAttributedString.DocumentType.html,
            .characterEncoding: String.Encoding.utf8.rawValue
        ]
        guard let attributedString = try? NSAttributedString(data: data, options: options, documentAttributes: nil) else {
            return nil
        }
        self.init(attributedString.string)
    }
}

Here we briefly initialize an attributed string specifying the DocumentType as .html. Character entity references are automatically substituted for the appropriate character on initialization, so all we have to do is return the .string property and we are done! The new initializer can be used like:

let htmlString = "Easy peasy lemon squeezy. 🍋"
let fixedString = String(htmlEncodedString: htmlString)
print(fixedString)

Easy peasy lemon squeezy. 🍋

Option 2: Regular Expression Matching

For the second technique we will write our own function that use a dictionary of CER -> Character mappings and regular expressions to perform character substitution manually.

Our dictionary will look like this:

let characterEntities : [String: Character] = [

    // XML predefined entities:
    """     : "\"",
    "&"      : "&",
    "'"     : "'",
    "&lt;"       : "<",
    "&gt;"       : ">",

    // HTML character entity references:
    "&nbsp;"     : "\u{00A0}",
    "&iexcl;"    : "\u{00A1}", ...]

I’ve left out the full list of CER : character mappings, but you get the idea. As for the rest of the implementation, let’s write a new function as an extension on String so character substation is available whenever we need it. Here is the full code:

extension String {
    func replacingCharacterEntities() -> String {
        func unicodeScalar(for numericCharacterEntity: String) -> Unicode.Scalar? {
            var unicodeString = ""
            for character in numericCharacterEntity {
                if "0123456789".contains(character) {
                    unicodeString.append(character)
                }
            }
            if let scalarInt = Int(unicodeString),
               let unicodeScalar = Unicode.Scalar(scalarInt) {
                return unicodeScalar
            }
            return nil
        }

        var result = ""
        var position = self.startIndex

        let range = NSRange(self.startIndex..<self.endIndex, in: self)
        let pattern = #"(&\S*?;)"#
        let unicodeScalarPattern = #"&#(\d*?);"#

        guard let regex = try? NSRegularExpression(pattern: pattern, options: []) else { return self }
        regex.enumerateMatches(in: self, options: [], range: range) { matches, flags, stop in
            if let matches = matches {
                    if let range = Range(matches.range(at: 0), in:self) {
                        let rangePreceedingMatch = position..<range.lowerBound
                        result.append(contentsOf: self[rangePreceedingMatch])
                        let characterEntity = String(self[range])
                        if let replacement = characterEntities[characterEntity] {
                            result.append(replacement)
                        } else if let _ = characterEntity.range(of: unicodeScalarPattern, options: .regularExpression),
                                  let unicodeScalar = unicodeScalar(for: characterEntity) {
                            result.append(String(unicodeScalar))
                        }
                        position = self.index(range.lowerBound, offsetBy: characterEntity.count )
                    }
            }
        }
        if position != self.endIndex {
            result.append(contentsOf: self[position..<self.endIndex])
        }
        return result
    }
}

So what is this function doing? In essence, we take our original string, look for substrings that match our pattern, iterate over the matches, and build up the result string by using the ranges found in each match to replace any CERs. For those unfamiliar with using NSRegularExpression there is an excellent article written by Matt on NSHipster that offers background, examples, and explanations. And while I’m directing you off of this article I should also recommend regex101.com, an interactive website I use all the time for prototyping regex patterns.

This new function can be called on any string as in:

let htmlString = "Easy peasy lemon squeezy. 🍋"
print(htmlString.replacingCharacterEntities())
Easy peasy lemon squeezy. 🍋

Conclusion

Thanks for reading. If you found this article interesting and aren’t already a member of Medium, please consider signing up! You will be supporting me (disclosure: I get part of the membership dues) and get access to tons of great content.

References

• https://nshipster.com/swift-regular-expressions/

• https://www.w3.org/TR/html4/cover.html#minitoc

• https://gist.github.com/mwaterfall/25b4a6a06dc3309d9555

• https://www.swiftbysundell.com/articles/string-literals-in-swift/
  
  

Making A Bucket Caddy Using OpenSCAD, A CNC Router, And Some Plywood

Over the past few years my wife and I have really gotten into gardening. We enjoy planting flowers and vegetables, getting outside, and rearranging our plants to enhance the aesthetic appeal of our yard. But having a garden is a lot of work and maintenance, especially when it comes to pulling weeds. On many occasions I would find myself losing track of my weeders and trowels while working a flower bed. I finally decided enough was enough and designed a small bucket caddy out of plywood that fits around any standard 5 gallon bucket.

Prep Work

Image from http://fivegallonideas.com/bucket-buddy-wheelbarrow-buckets/

Caddies for 5 gallon buckets aren’t a new thing. There are a variety of commercial cloth covers out there, but most of them have a billion pockets, which is overkill for what I need. For gardening I wanted something relatively simple that would strap onto the bucket and provide spots to hold my hand tools. A quick google of 5 gallon bucket dimensions led me to fivegallonideas.com, a site devoted to blinging out 5 gallon pails. Here I learned two things. First, the top diameter of most pails is 30cm. Second, people have figured out how to turn these things into base guitars, beehives, backpacks, and even air cannons. Why have I only just discovered this site?!

3D Modeling

Screenshot of OpenSCAD. The script is on the left and a rendering of the output is on the right.

There are a ton of free and commercial CAD packages out there. Some of the more popular ones for hobbyists are Funsion360 , SketchUp, TinkerCAD, and FreeCAD. Early in my prototyping phase I used SketchUp, but quickly realized that making small adjustments to shapes, spacing, etc. was absolutely painful using the free version. Then I found OpenSCAD. OpenSCAD ( Solid Computer Aided Design) is an open source project that uses scripts to define 2D and 3D geometries that can be exported as STL files for machining. It only took a small amount of tinkering to realize the power of OpenSCAD. You can parameterize whatever you want and create reusable parts (called modules), which makes tweaking dimensions in the design a lot easier than working with a GUI. I can’t say enough nice things about OpenSCAD. If you are a programmer interested in 3D CAD check it out.

CNC Routing, Finishing, and Assembly

The final design wraps a quarter of the way around the bucket and measures 198mm (~7.8 in) square x 13mm (0.51in). I used half-inch plywood and my Shapeoko for the cutout. With a bit of sanding and two coats of polyurethane here is the finished product. A 21” rubber tie down provides a snug fit with no slippage. Mounting the plywood centered and just below one side of the bucket handle keeps tools at a reasonable height and prevents the bucket from tipping during transport.

Make Your Own

If you have a CNC router you can make your own bucket caddy with the following:

• 1 x half-inch plywood sheet 8.5 x 8.5 in (large enough to clamp down and allow space for your cutter to travel)

• 1 x 21” rubber tie down

• This STL file (units are in mm)

Fixing A Broken Rocking Horse

My two-year-old son decapitated a rocking horse at our in-laws earlier this spring and rather than throw it out I decided to fix it while the stay-at-home order from Covid-19 was in effect.

So where to begin? For staters I thought it might be a good time to give the old chap a more handsome look. After doing a bit of digging on the internet for .svg files I found one that I liked. I smoothed out the mane using InkScape to make it easier to cut out on a bandsaw, and then proceeded to print out the new image and pasted it onto a piece of oak.

The bandsaw made quick work of the cutout, but left rather course edges that aren’t suitable for a kids toy, so I also added a 1/4 inch round-over using a trimming router.

A new hole was drilled into the head and the old handle was recycled. The project was finished off by affixing the newly constructed head to the rocker with a few screws and some wood glue!