Today I Learnt

Feb 19, 2017 - 5:34 pm
URIs don't allow UTF-8
As developers, we increasingly live in a Unicode world. Our code is written in Unicode, we output in Unicode and we - mostly - expect input to be in Unicode.

An interesting bug however alerted to me something that had completely missed me: A HTTP resource identifier with Unicode characters is not a URI.

For example there is an article on Al Jazeera about Lionel Messi with an identifier of: http://sport.aljazeera.net/ligabbva/2017/2/19/هل-يرفض-ميسي-124-مليون-دولار. You can click this and visit the page, and this address will show up in your browser bar - but this is not a URI. This is actually an IRI - International Resource Identifier - and it differs from a URI by the presence of Unicode characters.

An IRI is not something a browser requests. Instead a browser transparently converts the IRI into a URI by percent encoding the Unicode characters.

So when you request http://sport.aljazeera.net/ligabbva/2017/2/19/هل-يرفض-ميسي-124-مليون-دولار your browser is actually requesting http://sport.aljazeera.net/ligabbva/2017/2/19/%D9%87%D9%84-%D9%8A%D8%B1%D9%81%D8%B6-%D9%85%D9%8A%D8%B3%D9%8A-124-%D9%85%D9%84%D9%8A%D9%88%D9%86-%D8%AF%D9%88%D9%84%D8%A7%D8%B1 - not so friendly to humans.

Modern web browsers perform this conversion on your behalf, but this not the case with a tool like curl where you are expected to request a URI rather than an IRI.

How a server will respond to the presence of Unicode in a URI is inconsistent. Al Jazeera responds with a 400, as does Github. However Google, Twitter and Facebook accept them.
```
➜  ~ curl -Is http://sport.aljazeera.net/ligabbva/2017/2/19/%D9%87%D9%84-%D9%8A%D8%B1%D9%81%D8%B6-%D9%85%D9%8A%D8%B3%D9%8A-124-%D9%85%D9%84%D9%8A%D9%88%D9%86-%D8%AF%D9%88%D9%84%D8%A7%D8%B1 | grep HTTP
HTTP/1.1 200 OK
➜  ~ curl -Is http://sport.aljazeera.net/ligabbva/2017/2/19/هل-يرفض-ميسي-124-مليون-دولار | grep HTTP
HTTP/1.1 400 Bad Request
➜  ~ curl -Is https://www.github.com/\?\=🐧 | grep HTTP
HTTP/1.0 400 Bad Request
➜  ~ curl -Is https://www.google.co.uk/\?\=🐧 | grep HTTP
HTTP/1.1 200 OK
➜  ~ curl -Is https://www.facebook.com/\?\=🐧 | grep HTTP
HTTP/1.1 200 OK
➜  ~ curl -Is https://twitter.com/\?\=🐧 | grep HTTP/1.1
HTTP/1.1 200 OK
```
Although the IRI standard has now existed for 12 years it can be difficult to support the conversion from IRI to URI in programming languages. For example Ruby provides support through it’s URI::Escape.escape method, however for JavaScript you’re probably going to need a package.

👻
Permalink | Source
Oct 26, 2016 - 7:06 pm
Interactive Behaviour in HTML e-mail via Radio Buttons

Constraints breed innovation? This seems to be frequently the case in the world of HTML e-mail. Despite a rocky landscape of restrictions people have acheived some incredible things. I was surprised however today to get an interactive email from Dominos where you can click to see different images in a tab like interface.

Digging into view source I could see the usage was achieved using a <label> element with a <input type="radio"> inside it. Once the input is checked (via clicking the label) the adjacent CSS selector(+) is used to change the styling of the image.

This technique isn’t actually new, radio buttons have been a commonly used trick for behaviour without Javascript - known as “checkbox hack”. Famously used for a HTML + CSS version of Minesweeper.

Since forms are available in e-mails and many clients support modern CSS it makes sense this works, however this is the first time I’ve seen it. I found a couple of other examples searching the web.

Nice work Dominos 👏

Permalink | Source
Oct 24, 2016 - 8:00 pm
Rails 5.0.0 bug breaks nullable JSON columns

Getting a string value of "null" in your JSON columns with Rails? That is due to a bug in Rails 5.0.0 that is incorrectly serialising a nil value of the field into JSON.

Good news is that it’s already fixed and should be released with 5.0.1. Until then you can monkey patch a fix in.

Permalink | Source

Oct 9, 2016 - 4:36 pm

PostgreSQL Transaction Isolation Levels

In SQL Standard there are 4 levels of transaction isolation defined: READ UNCOMMITTED, READ COMMITTED, REPEATABLE READ and SERIALIZABLE.

Of these PostgreSQL supports READ COMMITTED, REPEATABLE READ and SERIALIZABLE. Where READ COMMITTED is the default.

Examples of each isolation level

DB Setup

transactions# CREATE TABLE albums (id serial PRIMARY KEY, title varchar NOT NULL, rating DECIMAL(2) );
CREATE TABLE
transactions# INSERT INTO albums (title, rating) values ('Rust in Peace', 10), ('Countdown to Extinction', 9);
INSERT 0 2

READ COMMITTED

The results of queries within this transaction will reflect committed transactions that have happened outside this transaction (thus within this transaction the same query could yield different results) and queries that are executed within this transaction.

transactions# BEGIN;
BEGIN
transactions# -- has no effect as this is the default isolation level
transactions# SET TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET
transactions*# SELECT COUNT(*) FROM ALBUMS;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

other-session# INSERT INTO albums (title, rating) VALUES ('Peace Sells... but Who''s Buying?', 8);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     3 │
└───────┘
(1 row)

transactions*# INSERT INTO albums (title, rating) VALUES ('The World Needs a Hero', 1);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     4 │
└───────┘
(1 row)

transactions*# ROLLBACK;
ROLLBACK
other-session# DELETE FROM albums WHERE title = 'The World Needs a Hero';
DELETE 1

REPEATABLE READ

This isolation level will not see changes that are committed in other transactions while this transaction is open. Transactions on the dataset will be allowed to be committed while this transaction is running.

transactions# BEGIN;
BEGIN
transactions# SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET
transactions*# SELECT COUNT(*) FROM ALBUMS;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

other-session# INSERT INTO albums (title, rating) VALUES ('Peace Sells... but Who''s Buying?', 8);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

transactions*# INSERT INTO albums (title, rating) VALUES ('The World Needs a Hero', 1);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     3 │
└───────┘
(1 row)

transactions*# ROLLBACK;
ROLLBACK
other-session# DELETE FROM albums WHERE title = 'The World Needs a Hero';
DELETE 1

SERIALIZABLE

SERIALIZABLE is very similar to REPEATABLE READ - They were actually synonymous until PostgreSQL 9.1. However the key difference is that it treats transactions as being executed in series and will error on situations where the ordering of concurrent transactions presents different results.

Consider this pair of REPEATABLE READ transactions:

transactions# BEGIN;
BEGIN
transactions*# SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET
transactions*# INSERT INTO albums (title, rating) VALUES ('Peace Sells... but Who''s Buying?', 8);
INSERT 0 1
transactions*# INSERT INTO albums (title, rating) VALUES ('Greatest Hits', (SELECT AVG(rating) FROM albums));
INSERT 0 1
other-session# BEGIN;
BEGIN
other-session*# SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;
SET
other-session*# INSERT INTO albums (title, rating) VALUES ('The World Needs a Hero', 1);
INSERT 0 1
other-session*# INSERT INTO albums (title, rating) VALUES ('Greatest Hits', (SELECT AVG(rating) FROM albums));
INSERT 0 1
other-session*# commit;
COMMIT
transactions*# commit;
COMMIT
transactions# SELECT * FROM albums;
┌────┬──────────────────────────────────┬────────┐
│ id │              title               │ rating │
├────┼──────────────────────────────────┼────────┤
│  1 │ Rust in Peace                    │     10 │
│  2 │ Countdown to Extinction          │      9 │
│  3 │ Peace Sells... but Who's Buying? │      8 │
│  4 │ Greatest Hits                    │      9 │
│  5 │ The World Needs a Hero           │      1 │
│  6 │ Greatest Hits                    │      7 │
└────┴──────────────────────────────────┴────────┘
(6 rows)

💩👎

Whereas with SERIALIZABLE:

transactions# BEGIN;
BEGIN
transactions*# SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SET
transactions*# INSERT INTO albums (title, rating) VALUES ('Peace Sells... but Who''s Buying?', 8);
INSERT 0 1
transactions*# INSERT INTO albums (title, rating) VALUES ('Greatest Hits', (SELECT AVG(rating) FROM albums));
INSERT 0 1
other-session# BEGIN;
BEGIN
other-session*# SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;
SET
other-session*# INSERT INTO albums (title, rating) VALUES ('The World Needs a Hero', 1);
INSERT 0 1
other-session*# INSERT INTO albums (title, rating) VALUES ('Greatest Hits', (SELECT AVG(rating) FROM albums));
INSERT 0 1
transactions*# COMMIT;
COMMIT
other-session*# COMMIT;
ERROR:  40001: could not serialize access due to read/write dependencies among transactions
DETAIL:  Reason code: Canceled on identification as a pivot, during commit attempt.
HINT:  The transaction might succeed if retried.
LOCATION:  PreCommit_CheckForSerializationFailure, predicate.c:4654

READ UNCOMMITTED

This level is unsupported by PostgreSQL and using it is treated as READ COMMITTED. It is provided to allow read access to transactions that haven’t committed. So if we use our imagination:

transactions# BEGIN;
BEGIN
transactions*# -- Let's pretend this works in PostgreSQL
transactions*# SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SET
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

other-session# BEGIN;
BEGIN
other-session*# INSERT INTO albums (title, rating) VALUES ('Hidden Treasures', 6);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     3 │
└───────┘
(1 row)

other-session*# ROLLBACK;
ROLLBACK
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

transactions*# END;
COMMIT

But in reality:

transactions# BEGIN;
BEGIN
transactions*# -- Actually acts the same as READ COMMITTED
transactions*# SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED;
SET
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

other-session# BEGIN;
BEGIN
other-session*# INSERT INTO albums (title, rating) VALUES ('Hidden Treasures', 6);
INSERT 0 1
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     3 │
└───────┘
(1 row)

other-session*# ROLLBACK;
ROLLBACK
transactions*# SELECT COUNT(*) FROM albums;
┌───────┐
│ count │
├───────┤
│     2 │
└───────┘
(1 row)

transactions*# END;
COMMIT

Sep 28, 2016 - 7:57 pm
Useful shortcuts in iTerm2
I’m a relatively new convertee to iTerm2 so I’m still getting to grips with the tricks. Here’s a few I’ve found/used:
- ⌘↩ - Toggle Full screen
- ⌘⇧h - View paste history
- ⌘; - Autocomplete
- ⌘d - Create a terminal splitting current one veritcally
- ⇧⌘d - Create a terminal splitting current one horizontally
- ⌘← and ⌘→ - Navigate tabs
- ⌘[ and ⌘] - Navigate panes
When in search mode (⌘f):
- ↩ - Copy current item
- ⇥ - Expand selection left
- ⇧⇥ - Expand selection right
Make sure to select regular expression search mode 👌

You can set a hot key to switch to and from iTerm from whatever app you are in via Preferences. I’ve gone for ⇧§.

Writing this I also found a great website for getting the unicode characters for Mac keys.
Permalink | Source

Older >

URIs don't allow UTF-8

Interactive Behaviour in HTML e-mail via Radio Buttons

Rails 5.0.0 bug breaks nullable JSON columns