If you have an iOS device you probably search the App Store regularly for apps you have heard about. Following the recent AppGratis ousting from the AppStore, there were claims that the App Store search is broken. That was our trigger to try something ourselves that could serve both as a good demo and help us to explore new use-cases! Check it out!

Obtaining the data

So first, we needed to obtain the data. Apple provides an API to accredited developers, but given that this can be fairly difficult to attain, we considered other solutions. Crawling was our second option, but that approach has its own caveats: you need to play nice with their servers or you get banned (very) quickly. We didn’t want to spend days implementing our own distributed crawler and definitely didn’t have the time to do a sequential and polite crawling. It is in these moments that you are glad to have an external team to do the job for you.

We chose to perform the crawling with grepsr, a service we found via a simple Google search. After a few exchanges we were confident that they were up to the job, and they ended up exceeding our expectations. Not only did they crawl the pages, but they also scraped the apps’ attributes to provide us with a clean dataset. After a few days we had our full dataset ready for indexing.

Indexing

Indexing was actually the easiest part. We uploaded the data in JSON format to our backend and used these simple settings:

{
"attributesToIndex": ["name", "author", "category"],
"attributesToHighlight": ["name", "author","category", "description"],
"customRanking": ["desc(score)", "asc(name)"]
}

Our dataset included the 630k applications currently published in the US app store. For each of them we index the name, author and category, but also include their icon, score, and description for display and sorting.

The score is a simple computation between the number of comments and the average ranking: rating * log2(nbComments) * 10000.

Searching

Similar to our CrunchBase demo, we trigger a query directly after page load and again after each keystroke. Additional queries are automatically triggered when scrolling to the bottom of the page.

Guillaume Esquevin did the front-end for us and a first version of the demo was up and ready in no time. Take a look at how simple and fast it is to search for an app!

In the end we did receive access to the Apple API, which we may use later on to keep the data in sync.

We have great news for our iOS and OS X users: our Offline SDK is now available as a CocoaPods dependency.

Cocoapods is a popular dependency management tool that lets you specify the libraries (dependencies) you want to use for your project in an easy-to-edit text file (Podfile). It then fetches all the required libraries and sets up your Xcode workspace.

You can now set up Algolia Search Offline in your iOS project with this line in your Podfile:
pod 'AlgoliaSearchOffline-iOS-SDK'

You can also set up Algolia Search Offline in your OS X project with this line:
pod 'AlgoliaSearchOffline-OSX-SDK'

Once you’re done, simply use the “pod install” command to set up Algolia Search Offline in your project. Now it’s easy to manage library dependencies for iOS and OS X projects!

If you are a Ruby developer and have an iPhone, chances are you already now about the cool project that is RubyMotion! I quote:

“It lets you quickly develop and test native iOS applications for iPhone or iPad, all using the awesome Ruby language you know and love.”

And it rocks! It’s actually used by the Ruby on Rails sponsor 37signals.

What if you could use your favorite search engine along with your favorite language to create iOS apps? That’s exactly what we propose with a RubyMotion Algolia Search gem that seamlessly integrates in your Ruby project. You know the trick:

 
gem 'motion-algolia-search'
 

The gem code is open source. You can fork it from github.

Special thanks to the RubyMotion team and especially to Joffrey for this integration!

While our latest news focused on the Algolia Search cloud offer (you can still join the beta), we’re pleased to introduce a major new version of Algolia Search offline: V2! It is available today for iOS, Android and OS X. Windows Phone and Windows versions will be released as soon as they are ready. A few months in the making, we used our early customers’ feedback to introduce new features while simplifying integration.

Algolia became the easiest way to search by geolocation!

The ease of integration is a constant concern for us and that’s why we carefully consider every new feature. Two important features made it in this version:

• Geo-search means the ability to search around a location or inside a bounding box. Results can be sorted by distance and of course geo-queries can be combined with textual ones. We added a dedicated tutorial in the doc to get up to speed with this new feature in no time (for iOS and Android).
• Tag filters enable restriction of results to specific tags. We received this demand a number of times in order to avoid the creation of too many specialized indexes.

These new features are also available in the beta of our cloud version!

Improved performance and ranking

With some hard work… and a lot of profiling, we have been able to get a 10% gain in performance on every query.

In V1, name matches were always considered more important than other attributes, but we didn’t consider differences between other attributes. This changed in V2: ranking priority now respects the order in which you indicate attributes in the textToIndex method. It’s more powerful while actually being more consistent with no specific processing of the name field.

But this improvement comes at the cost of a slightly bigger index and longer computation. If index size is important or if you need to earn a few nanoseconds more, you can optimize it away with the increaseCompression method. You’ll get a 10 to 30% reduction in index size and an additional 20% boost in performance (that’s 30% total compared to V1!).

Easy just got easier

Integrating search in an app has never been so easy. For V2 we took into account all the excellent feedback we received, and wherever it was possible we simplified the API:

• No distinction between suggest and search methods. We wanted to match the expected use-cases of the SDK but it was causing more confusion than anything else. So there is now only one way to send queries to an index: the search method.
• With the addition of geo-search, the index class was becoming crowded. We simplified this by decoupling the search approach and query definition. A small set of search methods enable the developer to choose if the search will be synchronous, use a callback, or batch several queries. And a simple SearchQuery class defines the nature of the queries themselves: geolocation, use of prefixes, tag filters, etc.
• Out of simple strings for which we provide a helper, every indexable object now has a UID. Our use of a “name” for this role led to a few difficulties when collisions were possible (persons for example). There are no longer any privileged attributes.
• License key initialization is now done using a static method. It is a best practice that was actually necessary to build a RubyMotion gem.

Specific to Android, we also added an AbstractIndexable abstract class. Instead of implementing the Indexable interface, you now have the option of directly extending AbstractIndexable that takes care of optional methods for you.

Specific to iOS, you can now directly index core data entities with the setCoreDataEntityDescription selector. No need to create a wrapper.

Still able to read V1 indexes

If for any reason you cannot replace or reindex your data, V2 is still able to search in a V1 index. However, as the name attribute was removed you do need to implement the IndexableLegacy interface. If you then publish changes, the new index will be in the V2 format.

We’re really sorry to make our Windows Phone and Windows customers wait. Feel free to torment us with your needs, great motivation to finish quicker

If you’re still reading, I guess it’s time for you to test this new version of the Algolia Search Offline SDK. Get started!

The NEXT Berlin Conference will take place on April 23rd & 24th in one of the most active ecosystem in Europe!  More than 100 experts will share their knowledge with an expected audience of 2000 attendees.

If you follow our newsletter or social presence, chances are you saw our call for help to participate in the NEXT Berlin startup competition. We sincerely thank you for your votes! They placed us in the 30 top startups to be considered for the final. The judges then selected us as one of the 12 finalists to pitch during the conference. We are thrilled to be able to present Algolia to the Berlin ecosystem!

If you happen to be attending NEXT Berlin, ping me if you’d like to meet! (@dessaigne on twitter, or nicolas at algolia dot com).

I recently read Mattt Thompson’s post about Apple Search Kit and he’s right about it being rather unknown, at least I had never heard of it before. So my very first thought was “how does it compare to our own SDK for OS X?” I immediately asked Julien if he could perform a few tests. Here are our findings.

Scope of use

Before examining the technical differences, it is important to stress the different intents behind the creation of both tools.

SearchKit has been designed as a document search engine, in what I call the “traditional approach”. It is able to index documents directly from disk (including conversion for a few common formats like pdf), and at the same time enables tokenization or stop words to be customized.

Algolia was designed differently. Even if it can be used to index 100-page documents, we targeted small texts, focusing on ease-of-use and performance. Tweaking a search engine requires deep expertise so we chose to hide that complexity by making sensible choices that work 99% of the time. You may see it as the “we know what’s best for you” approach taken by Apple for many of their products! On the other hand, we provide a Search SDK and only that. If you need to index PDF files, you need to manage the conversion with other libs.

Test setup and scenario

To get up to speed quickly, we performed our test with an existing dump of city data we had already grabbed from geonames for a tutorial (available here: http://www.algolia.com/doc/GeonamesCities.json). It contains 150k cities with their name, country, known population, and geoposition.

Search Kit doesn’t support multi-fields (a document is a single collection of terms) so we chose to only index the city names in this test. (Teaser: upcoming Algolia SDK V2 will support geosearch).

Results

Here are the raw results on a MacBook Pro Retina:

The actual syntax of Algolia queries does not need the ‘*’ character, as we use prefix queries by default.

Our observations

We discovered a few weak points of Search Kit during the test:

• It is a low level API and it doesn’t provide an Objective-C wrapper. It may explain why almost no one has heard of it.
• It doesn’t provide any helper to highlight matching characters or terms.
• It doesn’t support approximations, nor stemming. That means you need to query the exact words that are in the documents you index (prefix queries can also help in some cases).
• There is no notion of multiple fields. A document is a simple set of terms. This is particularly problematic when you want to tune ranking, for example by boosting the title.
• You must use a URI as identifier, even if you are indexing a simple string or if you are indexing an object that already has its own unique identifier. No big deal, but it complicates the code. Four our test, we had to write something like the following:

NSURL *objectURL = [NSURL URLWithString:[NSString stringWithFormat:@"file:///%@", [cityName stringByAddingPercentEscapesUsingEncoding: NSUTF8StringEncoding]]];
SKDocumentRef document = SKDocumentCreateWithURL((__bridge CFURLRef)objectURL);
SKIndexAddDocumentWithText(index, document, (__bridge CFStringRef)cityName, true);

However, there are a few characteristics that Algolia does not provide:

• It enables customization of tokenization and of stop words. Make sure you know what you’re doing however, as it can have unexpected impact. For example, if you add ‘-’ as a kSKTermChars, you won’t be able to find “Saint-Etienne” with the “etienne” query.
• It is able to convert some common file types (PDF, HTML, RTF, DOC).
• It supports more advanced boolean queries.

Conclusion

There is no one-size-fits-all search engine out there, especially on OS X. I would say that if you’re building a search-as-you-type feature like Spotlight, you should first give our SDK a try. It is much faster to integrate, has better response times, and handles typos. And don’t forget it is also available for your iOS apps.

If, on the other hand, you want to build a more traditional document search, you may need the document conversion and customization of Search Kit.

We are no Search Kit gurus and we’re happy for any errors in this post to be corrected.

Here is the code we used for indexing and searching on Search Kit & Algolia.

Indexation with Search Kit

CFTimeInterval before = CFAbsoluteTimeGetCurrent();
SKIndexRef index = SKIndexCreateWithURL((__bridge CFURLRef)fileURL, NULL, kSKIndexInvertedVector, NULL);
assert(index != nil);
NSString *citiesPath = [[NSBundle mainBundle] pathForResource:@"GeonamesCities" ofType:@"json"];
NSData* citiesData = [NSData dataWithContentsOfFile:citiesPath];
NSDictionary* dict = [NSJSONSerialization JSONObjectWithData:citiesData options:0 error:nil];

NSUInteger cnt = 0;
for (NSDictionary* k in dict) {
  NSString *cityName = [k objectForKey:@"Name"];
  NSURL *objectURL = [NSURL URLWithString:[NSString stringWithFormat:@"file:///%@", 
      [cityName stringByAddingPercentEscapesUsingEncoding: NSUTF8StringEncoding]]];
  assert(objectURL != nil);
  SKDocumentRef document = SKDocumentCreateWithURL((__bridge CFURLRef)objectURL);
  SKIndexAddDocumentWithText(index, document, (__bridge CFStringRef)cityName, true);
  ++cnt;
}
SKIndexFlush(index);
SKIndexCompact(index);
CFTimeInterval after = CFAbsoluteTimeGetCurrent();
NSLog(@"Time to compile %zdms", (NSUInteger)((after - before) * 1000));

Indexation with Algolia Search

CFTimeInterval before = CFAbsoluteTimeGetCurrent();
ASIndex* index = [[ASIndex alloc] initWithName:@"GeonamesCities" delegate:nil
                                  userDataClass:nil licenseKey:@""];
NSString *citiesPath = [[NSBundle mainBundle] pathForResource:@"GeonamesCities" ofType:@"json"];
NSData* citiesData = [NSData dataWithContentsOfFile:citiesPath];
NSDictionary* dict = [NSJSONSerialization JSONObjectWithData:citiesData options:0 error:nil];

for (NSDictionary* k in dict) {
    [index setEntry:nil forName:[k objectForKey:@"Name"]];
}
[index publishChanges];
CFTimeInterval after = CFAbsoluteTimeGetCurrent();
NSLog(@"Time to search %zdms", (NSUInteger)((after - before) * 1000));

Search with Search Kit

NSString *query = @"l*";
CFTimeInterval before = CFAbsoluteTimeGetCurrent();
SKSearchOptions options = kSKSearchOptionDefault;
NSUInteger limit = 1000; // Maximum number of results
NSTimeInterval time = 10; // Maximum time to get results, in seconds
SKDocumentID documentIDs[limit];
CFURLRef *urls = calloc(limit, sizeof(CFURLRef));
float *scores = calloc(limit, sizeof(float));
SKSearchRef search = SKSearchCreate(index, (__bridge CFStringRef)query, options);
CFIndex count;

SKSearchFindMatches(search, limit, documentIDs, scores, time, &count);
SKIndexCopyDocumentURLsForDocumentIDs(index, count, documentIDs, urls);

CFTimeInterval after = CFAbsoluteTimeGetCurrent();
NSLog(@"Time to search %zdms", (NSUInteger)((after - before) * 1000));

Search with Algolia Search

NSString *query = @"l";
CFTimeInterval before = CFAbsoluteTimeGetCurrent();
[index setNbHitsForSuggest:1000]; // Maximum number of results

[index suggestSync:query];

CFTimeInterval after = CFAbsoluteTimeGetCurrent();
NSLog(@"Time to search %zdms", (NSUInteger)((after - before) * 1000));

After launching the Beta Cloud version of Algolia, we wanted to demonstrate what it can do. We built a search engine using CrunchBase data, so entrepreneurs can easily search for their company or themselves. Check it out!

You can search for companies, people and financial organizations, using multiple attributes. Results are updated after each keystroke and matching characters are highlighted. And of course it tolerates typos. In this post we’ll explain in more detail how it works.

Indexation

The CrunchBase API is unfortunately rather poor. There is no way to know the latest updates. So we dump it regularly and push it to our servers in json format after pruning unnecessary attributes and adding images encoded in base64.

We actually create 3 indexes for companies (117k+ entries), persons (152k+ entries) and financial organizations (9k+ entries). The json files, including images, are respectively 150MB, 70MB & 7MB.  The full indexation takes about 5 seconds (excluding upload time). Resulting index sizes are respectfully 124MB, 85MB and 7,5MB.

Indexation is done simultaneously in 3 datacenters: US-West, US-East & Europe. Additional datacenters are on the roadmap.

Instant Search

We trigger a query directly after page load and again after each keystroke. To simplify communication with the server, we created a javascript client (contact us if you want to use it before its release). We then simply call the search function indicating the callback that will handle resulting hits asynchronously. More details to follow once we’ve written the doc!

We automatically choose the server closest to your location by using Amazon Route 53. Once the DNS lookup is resolved, it lets us get low enough latencies that the response feels nearly instantaneous (if you test it from North America or Europe). From DSL connections, we obtain search latencies of about 90ms in San Francisco, 75ms in New York and 65ms in London. About 20ms are used for querying the index, 5ms for compressing the data and 5ms for uncompressing. The remaining time is the actual transfer of the data and depends of your location and the quality of your connection.

If you’re a hacker, you may also remark the presence of an API key in the javascript. It cannot be hidden as we directly query our servers from the browser. The operations it enables are however restricted to search only, you would need to use a different key to update entries for example. You can create and revoke as many API keys you need directly from the API.

Hits Display

No designer worked on the demo, but we hope it doesn’t show! We execute the 3 queries simultaneously and display the results by blocks of 20 hits. Additional queries are automatically triggered when scrolling to the bottom of the page.

We display approximate results with a transparent background to clearly differentiate them.

You can use the arrow keys to navigate inside the results.

Ranking

We use the standard ranking order. By descending priority:

• Exact matches before approximate matches;
• User-defined order of attributes;
• Distance between the matching term and the beginning of the attribute;
• Proximity between terms in multi-word queries;
• User defined score.

For the order of attributes, we use {name, twitter, organization or people, description}. This translates into very simple settings. Here are for example the settings of the persons index:

{
  "attributesToIndex": ["name", "twitter", "unordered(companies)", "description"],
  "attributesToHighlight": ["name", "twitter", "companies", "description"],
  "customRanking": ["desc(size)", "asc(name)"]
}

By default, all attributes are indexed and highlighted. “attributesToIndex” & “attributesToHighlight” enable to precisely define what to index (and in what order), and what to highlight. The “unordered” modifier disable ranking between values of a multi-valued attributes.

For the user defined score (“customRanking” in settings) , we sort by decreasing order of CrunchBase entry size and then by alphabetical order.

Help us

This is just a demo but we’d like to continue improving it! Please tell us what you think and send your suggestions: contact at algolia dot com

Imagine all the power and simplicity of Algolia Search, but available online, from any connected device. Welcome to our Cloud offer!

Up to now, our instant, typo-friendly search was only available locally on your device. You could only index data stored directly in your app. While that was great for offline apps, it was not so fantastic for rapidly changing data server-side. All that has just changed! You can now use our search engine online, enabling you to change your data at any frequency and return consistantly up-to-date search results in your apps. Interested? Request your invitation to the Beta from the features page.

And don’t forget you can combine it with our offline SDK, for great search performance whatever the connectivity!

If you have visited our website in the past few days you may have noticed a few changes. We relaunched our website with a brand new design! Our main objective was to convey a clearer message about what Algolia is about and what we have to offer. In order to achieve this we:

• Completely redesigned the site to be less cluttered and hopefully better looking
• Rewrote most of the content aiming for clarity and conciseness
• Added illustrations to support our messages
• Added a use-cases section to illustrate what can be built with Algolia Search.

And finally, we made a new short video that should get you up to speed in barely a minute

Hope you like it! Let us know how we could continue to improve!

PS: Don’t forget to check out our new FREE offer!

Yes, you read that right! We’re not talking about a free trial, but of a completely FREE version of our SDK. It is often difficult to implement a good search experience, so we want to democratize access to easy-to-integrate and first class search features! All you have to do is to display non-intrusive Algolia branding.

This branded offer is already available on our website. Don’t delay, register today! Check out our use-cases section to discover possible implementations and see how Sharypic, Offline dictionaries, and Sush.io have integrated Algolia in their apps.

Help us spread the word. We want every app developer to be able to take advantage of this offer!