== Dev Week -- u1db: synced data for your apps on many platforms -- aquarius -- Thu, Aug 30th, 2012 == {{{#!irc [18:01] first, thanks to the Alexes :-) [18:01] The subtitle for this talk on the schedule is "synced data for your apps on many platforms", and that's what u1db does. [18:01] You build an app, and use u1db to store its data, and you can then sync that data to your app on other machines and other devices and other platforms. [18:02] So if you built an Ubuntu app, you could have the same data sync between all your Ubuntu machines, meaning that your data is everywhere. [18:02] You could also sync to other platforms, so an Android app could share the same data, and so on. [18:02] Imagine your shopping lists and your notes and your movie watching habits and your music ratings in your Ubuntu desktop and your smartphone and on the web available from anywhere. [18:03] So far, this is just syncing, like Ubuntu One file sync: why have a database? [18:03] U1DB is good at dealing with data; file sync isn't. [18:03] Some people have thought in the past "hey, my app stores things in SQLite; I'll just use U1 to sync my app's data folder and then my data is synced everywhere, woo!" [18:03] This is great, until it isn't. [18:04] If you make changes to your SQLite database on two different machines, they'll conflict, because SQLite isn't designed for syncing. [18:04] So you'll get a conflict from Ubuntu One file sync, which you'll have to resolve. [18:04] U1DB is cleverer than that; changes to your data won't conflict unless they really, really need to, so it's much more suitable for syncing. [18:04] You probably have questions at this point, but let's see some code first and then we can discuss. [18:04] First, you'll need u1db, of course. [18:05] For this talk, you'll need to get u1db from launchpad, with "bzr branch lp:u1db" [18:05] (U1DB itself is now available in Ubuntu Quantal, but for this demo please get the Launchpad version because it contains the example app as well!) [18:05] Once you have u1db, this should work and not throw errors: PYTHONPATH=u1db python -c "import u1db" [18:05] (I pause a little to give people a chance to grab u1db :)) [18:06] Let's try a simple example of a working app first: the u1db distribution comes with an example app called "cosas", which is a small todo list. [18:07] (Everyone writes a todo list now: it's like the Hello World of the 21st century) [18:07] cd u1db/cosas [18:07] PYTHONPATH=.. python ui.py [18:07] and you should see a todo list window pop up, looking like http://ubuntuone.com/7aOvtpIljWwbwB1FEFbs5L [18:08] (you might need to apt-get install python-qt4 if you don't have it) [18:08] calmi asked: python is the recommended language? [18:09] calmi: I'll get to that. It is *a* language you can use :) [18:09] there are many others :) [18:09] Add a couple of tasks, tick a couple [18:09] Then from the app menu, do File > Synchronize [18:09] and choose Ubuntu One and Synchronize Now [18:09] (this demo is for people who have an Ubuntu One account; if you don't have one, just skip doing this and take my word for it. You can sync u1db without using Ubuntu One at all by running your own server) [18:10] Your little todo list is now synced with your U1 account! [18:10] You can prove this: quit cosas and then delete the file ~/.local/share/cosas/cosas.u1db, as if you're a second machine [18:11] If you now restart cosas you'll have no todo list items... just File > Synchronize again and they'll be synced to you! [18:11] Obviously you could be running cosas on many different Ubuntu machines and the data could be synced to all of them. [18:12] So, let's see how this actually works. I'll show using U1DB from Python on Ubuntu, but it's also available in other platforms and languages too, which I'll talk about later. [18:12] Start with some simple examples, taken from the documentation at http://packages.python.org/u1db/ [18:12] Start a python interpreter with "python" [18:12] >>> import u1db [18:12] >>> db = u1db.open("mydb.u1db", create=True) [18:13] We've now created a U1DB database named mydb.u1db. [18:13] Next, we'll create a document in it. U1DB is a document-based database: you save JSON documents into it. So, a simple document naming a person: [18:13] >>> content = {"name": "Alan Hansen"} [18:13] >>> doc = db.create_doc(content) [18:13] And the Document is saved in the database. You can still see the Document's content: [18:14] >>> doc.content [18:14] {'name': 'Alan Hansen'} [18:14] We can edit the content of that document, of course: [18:14] >>> doc.content = {"name": "Alan Hansen", "position": "defence"} [18:14] After changing the content, we need to save that updated Document: [18:14] >>> rev = db.put_doc(doc) [18:14] And now the updated document is saved in the DB, ready to be retrieved or queried or synced. [18:15] Let's create a couple more documents: [18:15] >>> doc2 = db.create_doc({"name": "John Barnes", "position": "defence"}) [18:15] (and we'll change the content before saving: Document.content is a dictionary) [18:15] >>> doc2.content["position"] = "forward" [18:15] >>> db.put_doc(doc2) [18:15] >>> doc3 = db.create_doc({"name": "Ian Rush", "position": "forward"}) [18:15] and we now have three documents. [18:15] Retrieving documents from the database with a query is done by creating an "index". [18:15] To create an index, give it a name, and the field(s) in the document that you want to query on: [18:15] >>> db.create_index("by-position", "position") # create an index by passing a field name [18:16] so we have an index, named "by-position", indexing all documents on the field "position". [18:16] And now we can query that index for a particular value. If we want to get everyone with position="forward" in our list of people: [18:16] >>> results = db.get_from_index("by-position", "forward") [18:16] And our results is a list of two Documents: [18:16] >>> len(results) [18:16] 2 [18:17] And you can manipulate that list just like a standard Python list, which is what it is: [18:17] >>> data = [result.content for result in results] [18:17] >>> names = [item["name"] for item in data] [18:17] >>> sorted(names) [18:17] [u'Ian Rush', u'John Barnes'] [18:17] That's the very basics of using u1db: saving data, loading it, and querying for it, same as any database. [18:17] The documentation at http://packages.python.org/u1db/ goes into much, much more detail. [18:18] In particular, the tutorial at http://packages.python.org/u1db/tutorial.html walks through cosas, the example todo list app, and explains how it structures its data and how to sync a U1DB with other places, like Ubuntu One. [18:19] As I said right at the beginning, U1DB is designed to work everywhere. [18:19] This means that there will be, or could be, a U1DB implementation for any choice of platform and language. [18:19] What I've shown above is the Python implementation, which should work on any platform where you have Python (so Ubuntu, other Linuxes, Windows, Mac, N9, etc). [18:20] There is also a C implementation, so if you're writing apps in C or some other C-bound language you can use the C version. [18:20] At U1 we're also building an Android Java version and an iOS Objective-C version, in time. [18:20] There's also a command line client (u1db-client, and u1db-serve to run a simple server). [18:21] Members of the U1DB team are also bringing U1DB to Vala, Go, and in-browser JavaScript. [18:21] So apps using all those languages on all those platforms will be able to sync data: imagine your app on Ubuntu and a mobile version on your Android phone and a web version, all able to sync data between themselves. [18:22] Part of the goal of U1DB is that if you decide that you want to be able to use it in a platform we haven't provided (and you can't or won't bind to the C version) then it ought to be possible to do a full reimplementation of it in your chosen language for your chosen platform [18:23] There's a very comprehensive test suite exactly so that it's possible to test compliance. [18:23] So we're building U1DB for a whole load of different platforms, and it's possible to bring it to others. [18:23] U1DB is in Ubuntu Quantal, so your apps can depend on it. [18:24] You don't actually have to sync data to use U1DB, of course: you can just use it as a database and never sync it at all! [18:24] If you use U1DB in an app now, then you can add syncing later when you choose to. [18:24] This has been a brief tour of U1DB, and I'm sure there are questions, so I'll happily take them now :) [18:24] jsjgruber-l85-p asked: How is this different from couchdb? [18:25] Similar goals, similar-ish implementation, but the big difference is that U1DB runs in your process, and it's built separately for each platform and uses the platform's native data storage capabilities. [18:25] So couchdb was a server; if you wanted to use couchdb on your phone, you (or someone else) had to port the whole couchdb server to that platform, and run it as a server there and connect to it. [18:26] u1db is built separately and independently for each platform [18:26] so it's not making one codebase run everywhere; an implementation can do whatever makes the most sense on the platform it's on: an Android version would likely be written in Java, for example [18:27] and an in-browser JavaScript version would be run entirely by your web app; it's not a separate server that you have to deploy and secure, it's entirely client-side and then you sync it with a u1db server elsewhere. [18:27] using the DB is similar: it's a store of JSON documents, which is similar to couch. [18:28] does that answer the question? :) [18:28] jsjgruber-l85-p asked: How will it avoid the scaling problems couchdb had on the U1 infrastructure? [18:29] we've deliberately built the U1DB server infrastructure to work exactly for this use case [18:30] I'm not the server scaling expert, but I know the people who are :) If you have specific questions, we can answer them [18:31] wan26 asked: What encryption scheme is used for sync? [18:31] Sync to Ubuntu One is always over SSL [18:32] If you run your own server, you are able to run it over SSL or not as you choose [18:33] and sync to U1 is authenticated the same way as all the other Ubuntu One APIs [18:33] again, if you run your own server, you can (and should) define your own authentication policy :) [18:34] The best place to learn about u1db is the documentation, as mentioned, at http://packages.python.org/u1db/ [18:35] There are introductory materials there to get started, and a much deeper dive into the intricacies of querying, syncing, replication, and so on when you want to do more complicated things [18:36] or if you want to do a new implementation of u1db for a different platform [18:36] but the best thing is to use u1db as it exists and build apps on it :) [18:38] one of the apps in the recent Ubuntu App Showdown (tickit) is using u1db for storage precisely to get syncing capabilities [18:38] and we have a bunch of others talking to us as well :) [18:38] you can get hold of the u1db team if you have questions once you start development on irc, here on freenode, on #u1db [18:39] and the mailing list at https://launchpad.net/~u1db-discuss [18:40] and if you fancy helping me work on the JavaScript version, that'd be great :P [18:43] So, that concludes the brief tour: if you have questions, or you're thinking of using u1db, or curious as to how you'd use it, speak now. :) [18:45] Cool, no more questions [18:45] So, thank you for your time: come and tell us about which apps you're using u1db in! }}}