Trouble migrating from Gracenote Mobile Client to GNSDK for Mobile

Question

I'm working on migrating my Android app from the legacy Gracenote Mobile Client to the newer GNSDK for Mobile SDK, and I've hit a few snags:

1. In mobile client, I used GNOperations.recognizeMIDStreamFromRadio(GNSearchResultReady, GNConfig, samplePCMBuffer) to initiate a fingerprint and lookup operation on a PCM buffer. My application can only supply prerecorded audio to Gracenote (as opposed to simply pointing Gracenote at a streaming audio source), ideally as raw PCM though I could encode to standard compression formats if necessary. What should I use from the GNSDK for Mobile API to effect the same fingerprint and lookup operation on supplied prerecorded audio data, which can hopefully still be raw PCM?
2. The class GnMusicId looks like it might be a handy generic fingerprint generator and query issuer class, so it might be the answer to #1 above. However, I haven't found a way to determine when it finishes writing a fingerprint and, therefore, we are ready to issue a query. How can I get a callback letting me know that GnMusicId has finished writing a fingerprint from the GnMusicId.fingerprintWrite(byte[] audioData, long audioDataSize) method, and that the fingerprint is ready to be used in a query via GnMusicId.findAlbums(fingerprintDataGet(), GnFingerprintType.kFingerprintTypeStream6)?
3. In Mobile Client I was able to cancel ongoing Gracenote operations using GNOperations.cancel(GNSearchResultReady) -- I've read that the new architecture requires that specific operations be canceled individually due to a more modular design, but I haven't found a standard cancellation API on the various operations the GNSDK for Mobile can perform -- how should I cancel fingerprint and song lookup operations in GNSDK for Mobile?

Answer 1

Turns out you can recognize a given PCM array in GNSDK for Android with the following three GnMusicIdStream API calls:

1. GnMusicIdStream.audioProcessStart(sampleRateHz,pcmBitcount,channelCount) to prepare the recognition engine for the incoming PCM
2. GnMusicIdStream.audioProcess(pcmArray,pcmArray.length) to pass in the PCM array that you want to recognize
3. GnMusicIdStream.identifyAlbumAsync() to generate the fingerprint and then use it for a lookup operation. This will cause callbacks to the IGnMusicIdStreamEvents object passed to your GnMusicIdStream instance, and musicIdStreamAlbumResult() will deliver any results.

As far as I can see, using this approach you don't need to wait on fingerprint generation etc. explicitly -- you just call those three methods in order and then GNSDK handles the rest and will eventually issue a callback. The full id operation ends up looking like this:

try {


        mGnMusicIdStream = new GnMusicIdStream(mGnUser, GnMusicIdStreamPreset.kPresetRadio, new IGnMusicIdStreamEvents() {
            @Override
            public void musicIdStreamProcessingStatusEvent(GnMusicIdStreamProcessingStatus gnMusicIdStreamProcessingStatus, IGnCancellable iGnCancellable) {
                Log.d(TAG,"gracenote gnsdk -- musicIdStreamProcessingStatusEvent(); event is: "+gnMusicIdStreamProcessingStatus);
            }

            @Override
            public void musicIdStreamIdentifyingStatusEvent(GnMusicIdStreamIdentifyingStatus gnMusicIdStreamIdentifyingStatus, IGnCancellable iGnCancellable) {
                Log.d(TAG,"gracenote gnsdk -- musicIdStreamIdentifyingStatusEvent(); event is: "+gnMusicIdStreamIdentifyingStatus);
            }

            @Override
            public void musicIdStreamAlbumResult(GnResponseAlbums gnResponseAlbums, IGnCancellable iGnCancellable) {

                Log.d(TAG,"gracenote gnsdk -- musicIdStreamAlbumResult();  responsealbums matches: "+gnResponseAlbums.resultCount());

                if (gnResponseAlbums.resultCount() > 0) {
                    try {
                        final GnAlbum albumResponse = gnResponseAlbums.albums().at(0).next();

                        final GnTrack trackResponse = albumResponse.trackMatched();

                        if (trackResponse != null) {
                            mEvent.postOnGNSearchResult(new ISongRecognitionResponse() {
                                @Override
                                public
                                @NonNull
                                String extractTrackTitle() {
                                    // seems that track title comes reliably from GnTrack and much of the rest is locked
                                    // up in the GnAlbum?
                                    if (trackResponse.title() != null) {
                                        return trackResponse.title().display();
                                    } else {
                                        return "";
                                    }
                                }

                                @Override
                                public
                                @NonNull
                                String extractTrackArtist() {
                                    if (albumResponse.artist() != null) {
                                        if(BuildConfig.RULE_DEBUG_LEVEL>0)
                                            Log.d(TAG,"gnsdk -- album artist says "+albumResponse.artist().name().display());
                                        return albumResponse.artist().name().display();
                                    } else {
                                        return "";
                                    }
                                }

                                @Override
                                public long extractTrackPosition() {
                                    return trackResponse.currentPosition();
                                }

                                @Override
                                public long extractTrackDuration() {
                                    return trackResponse.duration();
                                }

                                @Override
                                public byte[] extractCoverArtImageData() {
                                    // seems that base64 string of the image is not always/commonly available
                                    // at least as we're trying to access it here.  The sample app downloads the image
                                    // asynchronously from the URL, which seems more reliable
                                    String img64 = albumResponse.coverArt().asset(GnImageSize.kImageSizeSmall).imageDataBase64(); //trackResponse.content(GnContentType.kContentTypeImageCover).asset(GnImageSize.kImageSize220).imageDataBase64();

                                    if(img64 != null && !img64.isEmpty()) {
                                        return Base64.decode(img64, Base64.DEFAULT);
                                    }else{
                                        return null;
                                    }
                                }

                                @NonNull
                                @Override
                                public String extractCoverArtImageURL() {
                                    // beware: asking for specific image sizes has been known to cause
                                    // no cover art to come back even if there might be cover art at another size.
                                    // The sample app uses the categorical size qualifier constant kImageSizeSmall
                                    String httpURL = albumResponse.coverArt().asset(GnImageSize.kImageSizeSmall).urlHttp();

                                    return httpURL;
                                }
                            });
                        }//end if track response data is non-null
                        else {
                            mEvent.postOnGNSearchResult(null);
                        }
                    }catch(GnException e){
                        Log.e(TAG, "we received a response clbk, but failed to process it", e);
                    }
                }//end if greater than 0 results
                else{
                    //no results, so pass a null result to indicate a miss
                    mEvent.postOnGNSearchResult(null);
                }
            }

            @Override
            public void musicIdStreamIdentifyCompletedWithError(GnError gnError) {
                Log.e(TAG,"gnsdk -- musicIdStreamIdentifyCompletedWithError(); we received a response clbk, but failed to process it");
                mEvent.postOnGNSearchFailure(gnError.errorDescription());
            }

            @Override
            public void statusEvent(GnStatus gnStatus, long l, long l1, long l2, IGnCancellable iGnCancellable) {
                Log.e(TAG,"gnsdk -- statusEvent(); status is: "+gnStatus);
            }
        });

        //configure the options on the gnmusicidstream instance
        mGnMusicIdStream.options().lookupData(GnLookupData.kLookupDataContent, true);
        mGnMusicIdStream.options().lookupData(GnLookupData.kLookupDataSonicData, true);
        mGnMusicIdStream.options().lookupMode(GnLookupMode.kLookupModeOnline);
        mGnMusicIdStream.options().preferResultCoverart(true);
        mGnMusicIdStream.options().resultSingle(true);

        //configure audio processing params on gnmusicidstream
        mGnMusicIdStream.audioProcessStart(sampleRateHz,pcmBitcount,channelCount);

        //pass the pcm array to the gnmusicidstream for processing
        mGnMusicIdStream.audioProcess(pcmArray,pcmArray.length);

        //initiate the lookup operation based on the processed pcm
        mGnMusicIdStream.identifyAlbumAsync();


    }catch(GnException e){
        Log.e(TAG,"gnsdk -- failed recognition operation",e);
    }

The return data is kind of confusing, with multiple potential ways to extract metadata about the track which may be null or empty when queried certain ways and not when queried in other ways. Interesting points I've found about the GnResponseAlbums response object so far (I'm not sure about the nullability contract of the return values mentioned below, so watch out for nullpointerexceptions):

• gnResponseAlbums.resultCount() will be 0 if nothing explicitly went wrong but there were no matches found.

• the matched GnTrack can be retrieved with albumResponse.trackMatched()

• the track title can be retrieved as a String with albumResponse.trackMatched().title().display()

• the track artist can be retrieved with albumResponse.artist().name().display()

• the current track position in time can be extracted with albumResponse.trackMatched().currentPosition(), which seems to be pretty accurate in determining the time the song will end as {endTime = currentTime + duration - currentPosition}

• the duration of the track can be extracted with albumResponse.trackMatched().duration()

• the cover art URL can be extracted with albumResponse.coverArt().asset(GnImageSize.kImageSizeSmall).urlHttp().

I haven't had any luck getting the image bundled as a base64 string via albumResponse.coverArt().asset(GnImageSize.kImageSizeSmall).imageDataBase64(), but the GNSDK provided a simple GnAssetFetch class which can be used to pull down cover art data as follows

GnAssetFetch assetData = new GnAssetFetch(mGnUser,coverArtUrl);
byte[] data = assetData.data();

As for cancelling an operation in progress, the GnMusicIdStream instance's identifyCancel() method can be used. If the cancellation is going to occur in the IGnMusicIdStreamEvents callback methods, the provided IGnCancellable canceller should be used instead.