Working on creating items API.
Got a response back from the Billy of the liberator mailing list on the best way to hold onto the passed in item JSON since it can only be read once.
I didn't understand the suggestion for a long while until I started to write him back and realized that he'd been suggesting that liberator decisions can return a vector, the first element of which is their normal response and the 2nd element is an addition to the request context.
With that knowledge, I was able to clean up my less elegant means of holding onto the parsed JSON.
I also updated the project to be using liberator as source (master) rather than a published version to get access to some new features. I used the lein-git-deps plugin to do this, but I'm not entirely happy with it because I still need to include a dependency on the published liberator to get its dependencies. Will need to ask around to see if there is anything better to do.
Working on making creating items API more robust.
Wrote back to Billy on the liberator list thanking him for his guidance on adding to the context from a decision.
Added exception handling and logging to the JSON parsing on item create to make it more robust.
Added check of valid-new-item? for the :processable decision on the create item POST. It checks for bad collections, no name, and slug conflicts.
Worked on getting a bad collection in the create item to return a 404. Got the attempt to return the 404, but since the media type is my own custom one, it's failing because liberator doesn't know how to render-map-generic for that type. Wrote to the liberator mailing list about the problem.
Billy wrote back explaining liberator was trying to coerce the handler's response to the mime-type, and it needs to be wrapped as an explicit ring-response to avoid this and have it handle the raw map as the response. I made this simple fix.
Realized having item and creation listing in the collection controller because they operate on a controller is not right. Refactored them into the items controller.
Working on success response from create items API call.
Added location header and JSON to successful response from created item.
Did a bit of code cleanup and refactoring.
Working on create and delete item API.
Completed full manual tests of all identified create item success and failure scenarios. Found a couple real issues and usability issues and resolved them.
Implemented basic item delete API support.
Found issues around character set in the response and Unicode. Posted question to liberator mailing list.
Identified a screen library for the terminal UI.
Working on item CRD APIs.
Completed full manual tests of all identified delete item success and failure scenarios. Found an issue with bad collection and resolved it.
Completed full manual tests of all identified item retrieval success and failure scenarios. Resolved a few issues.
Fixed a portion of the issue with character set in the response to a create item POST. Followed up to the liberator mailing list with two additional questions about the default charset in that case, and charset for the POSTed content.
Message in body to distinguish between no name provided, conflicting slug provided, and invalid slug provided.
Read about macros.
Work on with-collection macro. Posted question about body's access to something from the macro in IRC and Clojure mailing list.
Upgraded to Clojure 1.5.1.
Added lein ancient plugin.
Got a response from the Clojure mailing list instructing me on creating an anaphoric macro.
I don't quite understand the name, since according to OED anaphora is:
a. Rhetoric. The repetition of the same word or phrase in several successive clauses. b. Grammar. The use of a word which refers to, or is a substitute for, a preceding word or group of words.
Removed the collection retrieval boilerplate from item with the completed with-collection macro.
Fixed up a couple issues in the item model caused by using the new macro.
Got a response in the liberator mailing list about the charset in a successful POST response. Reworked the response to include a Content-Type with a charset. Replied to the responder on the list.
Made some adjustments to the ordering of development work.
Re-organized the project directory structure with better directory names.
Identified the test conditions for updating an item, except for conditional updates.
Added lein spell plugin to catch typos in docs and codestrings. Fixed misspellings.
Fixed some doc bugs in liberator and did a PR.
Worked out how liberator tracing works. Very helpful. Turn on liberator tracing with an environmental switch.
Capture the test cases for item create via PUT.
Start to work out on item update via PUT.
404 for no collection on item update.
Debug issue caused by switching to laptop and not using github master of liberator there.
Work on successful item update.
Work on unit tests for item create API.
Remember how DB views work. Improve DB views. Learn how reduce works in DB views.
Implemented item count for a collection.
Work on item create tests.
Excluded malformed and processable decisions from get requests to item. Emailed the liberator mailing list to see if anyone has ideas on using the built-in by-method function to do it, rather than doing it manually, since it doesn't seem the by-method function supplies the request context.
Wrote a check macro combining assert and is for better cucumber test failure output.
Work on item create API tests.
Finished up REST API DSL out of testing steps for Cucumber. Not entirely satisfied with it as Cucumber ends up creating a very impoverished DSL since it's Gherkin and not real code. I don't Gherkin can avoid repetition. I don't think Gherkin can't maintain state. This means you just have repetition and your state gets pushed down awkwardly into the Clojure step definitions which should be functional.
Work on item create API tests.
Some refactoring and code cleanup of the test code.
API tests for deleting an item.
Created test DSL for dealing with pre-populated collections.
Reworked the test DSL for dealing with the response body.
Fixed bug in malformed? and processable? decisions for delete.
Implemented additional tests for create item fail cases.
Fixed a bug where "JSON" that was just a string wouldn't trigger the malformed decision.
Added validation of the Content-Type of the response to all API tests.
Finish POST create item tests.
Implemented item retrieval tests.
Updated test DSL to allow appropriate a/an usage.
Worked on project planning and shuffled some items into the roadmap.
Refactored some of the resources API, returning true over :OK and simplifying valid-item-update?.
Added initial set of API tests for item update with PUT.
Added item/update-item function which is sort of working.
Implemented portion of the item update API.
Fix item/update-item by retaining the automatic properties of :type, :slug and :collection.
Created the empty features for the rest of the collection tests.
Got item update API call working with manual testing.
Improved the wording on API test features.
Fixed a regression where item delete was incorrectly returning a body.
Fixed some issues with what data item update API was responding with.
Implemented many of the test cases for the item update API.
Changed item create and update to be permissive and assume the right content-type if none was specified.
Worked on project planning and shuffled some items between versions.
Finished the item update implementation and full set of tests.
Did a small refactoring to eliminate redundancy in API input checks.
Implemented item/all-items and collection/all-collections at the Clojure API level in prep. for implementing them at the API level.
Created a high-level externals architecture sketch.
Added some new externals (API wrappers and clients) to the roadmap.
Upgraded to CouchDB 1.3.1 on main dev. machine.
Changed license from MIT to Mozilla Public License v2.0.
Added DB view initiation as a ring :init so it's not skipped when starting the server from lein ring.
Support configurable CouchDB connection, and default to test DB when running tests.
Factor out config into its own namespace.
Move liberator trace config into the config namespace.
Work on slugify implementation and testing.
Finish most of slugify's functionality and tests.
Move slugify functionality into it's own namespace.
Finish slugify unit tests.
Reworked slugify's handling of max length truncation.
Write slugify integration tests for item creation.
Unique slugs generated from item name on item creation.
Added init-db lein aliases and added it to the lein test! alias.
Detect a slug conflict in item/create-item.
Slugify collection name on collection create and don't allow generated slug to conflict.
Detect a slug conflict in collection/create-collection.
Refactor next-slug generation out of item and collection and into common.
Added automatic version numbering on resource create.
Update version number on item update.
Added automatic timestamps on resource create.
Update updated-at timestamp on item update.
Integration tests for initial timestamps and version on item create.
Integration tests for timestamp and version on item retrieval.
Integration tests for updated timestamp and version on item update.
Setup Travis CI for continuous integration testing. It's unclear why the cucumber tests pass fine locally but some fail on Travis' builds.
Worked on bulk delete of items when collections are deleted.
Completed bulk delete of items when collections are deleted.
Reworked naming scheme for CouchDB views.
Factored out dealing with CouchDB views into some common functions.
Updated how the mock data state is held to support the cucumber tests in the hope of making the tests pass when run on Travis CI. Didn't make a difference.
Looked into the first test failure happening on Travis CI. Commented out tests so ONLY one failing test will run to try to eliminate a timing or state related issue.
Figured out the difference between Travis CI and running locally. Travis CI isn't using master from the local git dependency so the version of liberator is different. In particular, the version Travis is using does not support the processable? decision.
Fixed by adding git-deps to the run chain for the test! alias that Travis uses so it has the source version of librerator.
Added the Travis CI badge to the readme. Made an editing pass over the README.
Made the Falkland CMS Trello board public and linked it from the README with a graphic I made.
I'm quite a bit of the pace I originally envisioned as I wanted a full Clojure API and a REST API that covered items, taxonomies and search in July so I could be working on the command-line client by August. I have a Clojure API that covers items and collections and a REST API that covers items. It will be another 2 weeks before I'm working on the commandline client.
Implemented collection/valid-collection-update? and collection/update-collection.
Added a changelog.
Start work on listing items in a collection API and tests.
Added validation of link's content type when validating links in API responses.
Got the item listing API working for the success case (w/o pagination).
Got the test of the item listing API for an empty collection passing.
Full set of integration tests for the item listing API except for pagination.
Still need to verify the content and links of the individual items coming back in the item list. Will require a minor refactoring of the item verification functions.
Refactor item link testing so it can be used in testing item lists.
Test the item links that come back in item list API requests.
Refactor item property testing so it can be used in testing item lists.
Test the properties of items that come back in item list API requests.
v0.1.0 - Tested minimal API is complete. Tagged the version and updated the project to v0.2.0-SNAPSHOT.
Work on README and doc content.
Created a diagram of the key concepts in FCMS based on the initial sketch I did before development started.
Show the diagram in the README and made other content improvements to the README.
Setup Sphinx to build API docs from restructured text.
Setup readthedocs.org to build the docs from Github on each push.
Ported the item API docs from markdown to rst.
Ported the API intro docs from markdown to rst.
Ported the collection API docs from markdown to rst.
Laid out the pros and cons of two different approaches for taxonomies, having them independent of collections and having them contained in collections. Decision hinges on if I expect there to be 1 or very few (but different) collections vs. many collections.
Decided that since there's not a big advantage in having many collections over just 1 collection to optimize for 1 (or fewer collections) and go with the cleaner model of having taxonomies defined in collections. Can always build in sync support of taxonomies later if not being able to reuse them across collections becomes a big issue.
Decided schema can be defined at collection, taxonomy and category level. Attributes defined at the collection level apply to all items in the collection. Attributes defined at the taxonomy level apply to all items categorized in that taxonomy, and attributes defined at the category level apply to all items categorized at that category or below.
Improved docs about the various types of slugs.
Began to work out the JSON format for taxonomies.
Worked out the JSON for an extensive example taxonomy.
Worked on documenting the taxonomy REST API.
Thought about the similarity between items and taxonomies as blobs in the scope of a collection from a data management perspective and about potential for code reuse.
Work on the intro to the taxonomy docs with an explanation and diagrams explaining how the hierarchal taxonomies and item classification works.
Worked on taxonomy REST API docs.
Stubbed out some taxonomy integration tests.
Updated the development plan in Trello to be reordered a bit and to be more realistic in light of the progress to date.
Updated an elastic search API dependency.
Fixed some spelling errors throughout the docs.
Researched Octohipster as a possibly interesting Liberator wrapper. Decided don't need what it offers right now, but it maybe interesting to look at again when I tackle pagination.
Researched Collection+JSON and decided to adopt it for the listing of resources APIs only (it has pretty weak provision in my opinion for creation and deletion too).
I also wasn't overly impressed with the "template" portion of Collection+JSON for helping API users create new items. Researched JSON Schema as a replacement and decided to use it for the "template" section.
Updated the collection item listing API docs to use Collection+JSON and JSON Schema. Neither the API code nor the tests match this yet.
Worked on getting speclj working for the Clojure API unit tests. Couldn't get it to use an alternative directory to /spec for the location of the tests. For FCMS they need to be in /test/spec. The :test-path and :test-paths mentioned in the README don't work.
Started working with expectations a bit before deciding it maybe wasn't ideal for this given its opinionated nature.
Went with plain ole clojure.test instead. Wrote unit test skeleton for all of item Clojure API and implemented it for testing item creation. Found 2 bugs too that had escaped detection in integration testing. Fixed those.
Reworked the lein aliases to remove some repetition.
Worked on taxonomy model implementation and refactoring pieces of item into common to be shared between them.
Did more work on taxonomy model implementation and refactoring pieces of item into common to be shared between them.
Reworked how properties get retained automatically on an update and added name to that set.
Have a minimal (get and create) implementation of taxonomy but tests are broken. Need to fix them next.
Fixed broken tests due to a syntax error in slug validation and a change that the name property is no longer required for updates and will be retained from the prior version if not provided.
Fix the rest of the broken tests.
Added integration tests for conflicting properties during item creation and updates.
Finished the refactor of get, valid-new and create to be shared by item and taxonomy.
Worked on unit tests (mostly) and implementation (a little) for verifying the category tree of new and updated taxonomies.
Did a little research into OrientDB as a potential alternative to CouchDB. Does having a graph/document hybrid buy us something good/interesting for the CMS? For taxonomies? For relations down the road?
Worked on category tree validation and testing. Got it done except detecting duplicate slugs at the same level of the tree. Need to flesh out a few more tests as well.
Finished handling blank category tree initiation and supplied category tree validation on taxonomy create.
Freezing issue when highlighting code in Sublime Text 2 so upgraded to Sublime Text 3 beta and configured.
Refactored delete to be shared by item and taxonomy.
Refactored update and update validation to be shared by item and taxonomy.
Refactored listing all in a collection to be shared by item and taxonomy.
Refactored the validate-new-item and empty-collection-c portions of the unit tests to be reusable by item and taxonomy.
I worked on creating category in an existing taxonomy with the Clojure API and associated tests. Not yet complete.
Fixed an issue with lein profiles that meant repl sessions were getting the test DB.
Changed item and taxonomy update APIs to allow reserved properties to make it easier to round-trip an update from a get.
Updated the REST API tests to account for the update rule change.
Did some minor refactoring in tests that verify the item returned from REST API calls.
Wrote the code that breaks a category path down into its component categories.
More (fruitless) work on the add category API. Not easy to change the data structure as vector of maps with vectors of maps with vectors of maps...
Would be trivial if it was maps of maps of maps, but the order is significant, and maps aren't ordered unless they are sorted. We don't really want sorted, we want ordered. There's a difference.
Need a data structure that's an ordered hash to implement this API.
Looked into ordered hash options to make implementing the add category API sane.
Clojure has an array-map but it auto-switches over to a map at > 8 keys so that's no good.
Found some ports of Ruby's array map to Clojure and picked one.
Did some dependency updates and noticed that lein ancient wasn't warning me of out of date plug-ins. Checked it's page and saw that's an option, and that I wasn't giving lein ancient permission to tell me about qualified releases. Updated the alias to support those things and updated a bunch of dependencies.
While looking at dependencies, I went back to the latest ClojureScript erroring out generating the CouchDB view code. I dropped back 1 release and tried it, still errors. Dropped back 2 releases and the code generates fine. But then (CI FTW!) the tests had massive failures so clearly the generated code is still not equivalent of what I was getting with the few month old cljs generated code. Dropped back to the old release for today to get the tests passing again, but queued up yanking the cljs code for plain js in the CouchDB views. It's so little code that it's not worth the hassle, and it'll be nice to have good Futon support for the views.
Another JSON library swap out. This time to Cheshire which seems to combine the speed and extensibility of the other two into one.
Ported the scant cljs CouchDB view code to JS due to issues with CouchDB running the cljs compiled from the latest versions of cljs and so the view of the views in Futon will be nice and clean.
Work on category tree manipulation.
Tremendous amount of work (mostly reading, thinking, and sketching) about the data structure and API behind hierarchical categorization in CouchDB, taxonomies, categories and categorizing items. Captured notes in notebook and a new doc including all the actions needed and how those breakdown into Clojure and REST APIs.
Some good ideas in the reading about categorizing items and querying, but decided to stick with keeping the category tree all as a JSON block in the taxonomy rather than breaking it up (relational DB style).
Seems that can only optimize for tree manipulation (by keeping category details out of the item doc) at the expense of easy and fast categorized item retrieval, or optimize for fast and easy retrieval (by putting the category details in the item doc) at the expense of easy and fast tree manipulation. Seems clear that with Falkland CMS the right optimization is for easy insertion, deletion and retrieval of items, not tree manipulation.
More work on category tree manipulation.
Nice to be back at it after some time off to focus exclusively on the day job, but frustrating day due to lack of real progress and the complexity of this problem.
Finally got addition of categories to category trees working. Involved transforming the on disk CouchDB data structure, vectors of maps of vectors of maps..., which is the same as the JSON tree returned in the REST API, to a data structure more amenable to manipulation, which is ordered maps of maps of ordered maps of maps... and then back again once the manipulation is complete.
Worked out the remaining tasks to finish up Clojure API and unit testing for taxonomy and categories.
Got all the tests developed and working for creation of categories in taxonomies with the Clojure API.
Developed the category-exists functionality and its associated tests.
Set up the scaffolding for categorize-item and uncategorize-item tests.
Wrote the failure case tests for categorize-item.
Implemented the failure cases for categorize-item.
Thought through what should be in the JSON in CouchDB and REST API and in the data structure for the Clojure API for a categorization of an item. Particularly interesting are the parents of a categorization. Do they appear or are they just implied by the path?
Example:
Categorize an item as: /tax/a/b/c
Do you see:
Or just:
I certainly expect the item to show up when I ask for items of /tax/a/b, but does /tax/a/b need to be in the JSON payload?
It'll need to be emitted in the CouchDB view, so that searches for /tax/a/b will find the items categorized as /tax/a/b/c. That's for certain.
A related question is when I categorize as /tax/a/b/c then I uncategorize. Do I need to uncategorize as /tax/a, or even as /tax? Does uncategorizing as /tax/a/b/c imply it's still a /tax/a/b?
Similarly, if I have an item categorized as /tax/a/b and I then categorize it as /tax/a/b/c is it categorized as both? Or is it no longer categorized as /tax/a/b?
A slightly different question is when I categorize as /tax/a/b/c then suppose uncategorize as /tax/a/b. What's the meaning of that? Should I say, it's not categorized as /tax/a/b so you can't do that? Or instead, do I say it's now categorized as just /tax/a?
Made a decision on these API design questions, then had a discussion with Andrew to ensure his intuitions matched mine for the API design. Decided thusly:
Item will only store the specific category, and will just emit the others where needed in a view. You will only see the specific category that was used when you retrieve the item.
You can only uncategorize a specific category that has been used on the item, not a parent.
You can't categorize a parent of a child that's already been categorized.
Implemented categorize-item with tests.
Refactored some tests and code to eliminate duplication.
Discovered clojure.zip, a natural tree structure. Should this replace my map/vector/map structure? Seems much nicer at the Clojure level, but what about to/from JSON?
Cleanup and finish the item categorization tests.
Test that ClojureScript is setup with a modern setup and working well in the app with an updated browser REPL setup.
Remove some unneeded tests since I decided not to remove a parent categorization when categorizing a child of that parent.
Fix some config stuff to get Travis-CI builds/tests working again.
Updated out of date dependencies.
Implemented all the failure cases, and tests for the failure cases, for improperly trying to uncategorize an item.