Expose new tvOS 18 Companion Remote features to API (e.g. now playing images, insight, upnext)

[thiccaxe]

What feature would you like?

see #2325 (comment)

Describe the solution you'd like

The "now-playing" endpoint needs to be exposed. if I PR I would also want to throw in some of the things from the linked issue, namely

• MCC and MCF requests/responses

As for the now-playing stuff, there are a few points:

1. interest packet for now playing c->s (response useless)
2. FetchCurrentNowPlayingInfoEvent c->s (response useless)
3. NowPlayingInfo s->c
4. FetchUpNextInfoEvent c->s (need to figure out how "pagination key" works) (response important)

for 3) we can give back the response as raw json (parsed with plistlib). This is because the response uses the NSKeyedArchiver format, for which there exist a couple unarchivers/archivers, but I do not know the quality of them. (Make one in-house?). The best option for now might very well be to leave this to the end user. I'd love some input on this.

sending 2) seems to be a request for the server to process the data needed for 3) and send it back later (a couple seconds)

we actually don't know the exact timestamp, but the tv client likely extrapolates the current time stamp from the last time the apple tv send a now playing info packet.

Any other information to share?

---

[postlund]

We should never expose any raw values from the protocol, but rather come up with a stable API. I assume most of the "now playing" data should be exposed via Metadata.playing, like other protocols do. For more complicated data, we likely should come up with another interface to use. Please paste some example messages and we can go from there.

[postlund]

We have a basic implementation for NSKeyedArchiver here:

https://github.com/postlund/pyatv/blob/master/pyatv/protocols/companion/keyed_archiver.py

It was implemented for keyboard support, but might be useful here too?

[thiccaxe]

We have a basic implementation for NSKeyedArchiver here:

https://github.com/postlund/pyatv/blob/master/pyatv/protocols/companion/keyed_archiver.py

It was implemented for keyboard support, but might be useful here too?

It could be, but that one extracts one key at a time, whereas we might want to generally dump a large amount of data.

Netflix title:

{
    "$version": 100000,
    "$archiver": "NSKeyedArchiver",
    "$top": {
        "root": 1
    },
    "$objects": [
        "$null",
        {
            "playbackRate": 3,
            "captionsEnabled": 16,
            "imageData": 14,
            "imageDataIsPlaceholder": 15,
            "metadata": 6,
            "playerIdentifier": 5,
            "$class": 17,
            "identifier": 2,
            "expectsTimedMetadata": 16,
            "rawTimedMetadata": 0,
            "playbackState": 4,
            "hasValidCaptionOptions": 16
        },
        "com.apple.avkit.1234.githash",
        1.0,
        1,
        "avkit-{UUIDV4dashupper}",
        {
            "canonicalID": 7,
            "currentlyPlayingSongID": 0,
            "showProductPageURL": 0,
            "title": 8,
            "episodeTitle": 0,
            "episodeNumber": 0,
            "productPageURL": 0,
            "duration": 9,
            "mainContentStartTime": 0,
            "releaseDate": 0,
            "rottenTomatoesReview": 0,
            "imageURLTemplate": 0,
            "kind": 0,
            "showID": 0,
            "$class": 13,
            "seasonNumber": 0,
            "isAppleOriginal": false,
            "extendedDescription": 0,
            "bundleID": 0,
            "timestamp": 12,
            "timeOffset": 11,
            "audioLanguage": 0,
            "ratingDescription": 0,
            "programID": 0,
            "genre": 0,
            "iTunesStoreIdentifier": 10
        },
        "com.apple.tvremote.canonical-identifier.unknown",
        "Netflix Title Name",
        4567.8, // "<title length>",
        "0",
        1234.5, // "<the current progress?>",
        743908511.717014, //"<for synchronization? not unix time??>",
        {
            "$classname": "TVRCNowPlayingMetadata",
            "$classes": [
                "TVRCNowPlayingMetadata",
                "NSObject"
            ]
        },
        "0xJPEG",
        false,
        true,
        {
            "$classname": "TVRCNowPlayingInfo",
            "$classes": [
                "TVRCNowPlayingInfo",
                "NSObject"
            ]
        }
    ]
}

Apple TV+ title with the timed metadata

{
    "$version": 100000,
    "$archiver": "NSKeyedArchiver",
    "$top": {
        "root": 1
    },
    "$objects": [
        "$null",
        {
            "playbackRate": 3,
            "captionsEnabled": 23,
            "imageData": 20,
            "imageDataIsPlaceholder": 21,
            "metadata": 6,
            "playerIdentifier": 5,
            "$class": 24,
            "identifier": 2,
            "expectsTimedMetadata": 23,
            "rawTimedMetadata": 22,
            "playbackState": 4,
            "hasValidCaptionOptions": 23
        },
        "com.apple.avkit.{somenumber}.{githash}",
        1.0,
        1,
        "avkit-{UUIDV4dashupper}",
        {
            "canonicalID": 7,
            "currentlyPlayingSongID": 0,
            "showProductPageURL": 0,
            "title": 8,
            "episodeTitle": 10,
            "episodeNumber": 4,
            "productPageURL": 0,
            "duration": 11,
            "mainContentStartTime": 14,
            "releaseDate": 0,
            "rottenTomatoesReview": 0,
            "imageURLTemplate": 0,
            "kind": 0,
            "showID": 0,
            "$class": 19,
            "seasonNumber": 4,
            "isAppleOriginal": false,
            "extendedDescription": 0,
            "bundleID": 18,
            "timestamp": 17,
            "timeOffset": 16,
            "audioLanguage": 0,
            "ratingDescription": 12,
            "programID": 15,
            "genre": 9,
            "iTunesStoreIdentifier": 13
        },
        "umc.cmc.a0b1g4hijkl",
        "Title",
        "Genre",
        "Episode Name",
        duration: float,
        "Rating",
        "iTunesStoreIdentifier",
        mainContentStartTime: float,
        "programID",
        timeOffset: float // the current location in the title,
        timestamp: float,
        "com.apple.TVWatchList",
        {
            "$classname": "TVRCNowPlayingMetadata",
            "$classes": [
                "TVRCNowPlayingMetadata",
                "NSObject"
            ]
        },
        "0xJPEG",
        false,
        "timed encoded data, json",
        true,
        {
            "$classname": "TVRCNowPlayingInfo",
            "$classes": [
                "TVRCNowPlayingInfo",
                "NSObject"
            ]
        }
    ]
}

I see what mean about interface.Playing.
we can fill the following data, with device state coming from another companion event, TVSystemStatus, and a corresponding request.

media_type: const.MediaType = const.MediaType.Unknown,
device_state: const.DeviceState = const.DeviceState.Idle,
title: Optional[str] = None,
genre: Optional[str] = None,
total_time: Optional[int] = None,
position: Optional[int] = None,
series_name: Optional[str] = None,
content_identifier: Optional[str] = None,

the question is, what to use as content identifier. There is canonicalID, bundleID, iTunesStoreIdentifier, and maybe for music currentlyPlayingSongID

the timed data comes as JSON in this form:

{
  data: { timedMetaData: {
    entities: {
      "/v/id": {
        type: "Person",
        id: "id",
        url: "https://",
        title: "Person name",
        subtitle: "charactername",
        isPrimary: boolean, // (isMainCharacter?)
        image: { //optional
           height, width, url, joeColor: "tagged rgb values",
           supportsLayeredImage: bool
        }
      },
     "/v/id": {
        type: "Song",
        songType: "iTunes", // I assume all Apple TV+ music is on itunes
        mediaApiData: {songData}
     }
    },
    occurences: { audio, video: { frames: [{
       start, end, entities: []
    }] } },
    sortedEntityIds: ["/m/id", "/v/id", "alpha?"]
  }}
}

[thiccaxe]

interface.RemoteControl#skip_backward and #skip_forward needs to be updated to support a defined skip time, or create new methods that allow it, maybe just #skip

Also, we can probably use interface.Stream#play_url with the companion PlayMediaEvent request. I will need to test this.

[thiccaxe]

not sure exactly what the PlayMediaEvent is doing as compared to passing in a url to launch_app. It doesn't actually start playing the media, just opens to its page. And it only works on apple tv. maybe apple is transitioning to this new endpoint? I don't know.