# Shapes (`shapes-1`): Sketch to shape

Rough sketch in, clean shape out.

Shapes turns a hand-drawn sketch into a clean shape. Send the strokes from a pen, finger or mouse and get back what was drawn (line, curve, arrow, rectangle, square, ellipse, circle, triangle, diamond, pentagon, hexagon, octagon, star, heart, cloud or scribble) with tidy geometry: corners, center, size, rotation, and an `outline` you can draw as is.

When a drawing isn't a shape, `shape` is null, so you keep the ink as drawn. Coordinates come back in your own canvas space. It scored 94.6% on Quick, Draw! single-stroke shapes in our tests, at about 2 ms a drawing. You can also send a picture of a sketch instead of strokes.

- **Price:** $0.005 per 1,000 drawings. One unit per drawing sent as strokes; an image of a sketch is 10 units.
- **Languages:** Not applicable
- **Input:** One drawing: an array of strokes, each an array of [x, y] points (or a single stroke, or {"strokes": [...]}), as JSON or a JSON string. For a batch, an array of up to 64 such JSON strings. Or an image of a sketch: an https URL, a data URL or a file in a multipart upload. Up to 200 strokes and 20,000 points per drawing.
- **Batch:** yes, up to 64 inputs per call
- **Status:** live

## Request

```bash
curl https://labs.greatwork.company/v1/run \
  -H "Authorization: Bearer $GWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"shapes-1","input":[[[1242,409],[1197,436],[1152,464],[1108,492],[1065,522],[1022,551],[975,574],[932,548],[902,505],[873,462],[843,418],[816,374],[827,325],[868,293],[911,263],[954,232],[997,202],[1040,172],[1082,142],[1126,112],[1169,82],[1212,52],[1256,24],[1306,21],[1343,57],[1374,100],[1405,142],[1436,184],[1460,230],[1439,275],[1397,306],[1354,336],[1311,366],[1267,395],[1224,424],[1179,452]]]}'
```

## Options

| Name | Type | Required | Default | Description |
|---|---|---|---|---|
| `input` | strokes (JSON), string[] or image | yes | | One drawing: an array of strokes, each an array of [x, y] points (or a single stroke, or {"strokes": [...]}), as JSON or a JSON string. For a batch, an array of up to 64 such JSON strings. Or an image of a sketch: an https URL, a data URL or a file in a multipart upload. Up to 200 strokes and 20,000 points per drawing. |
| `rotation` | string (snapped \| upright \| free) | no | `"snapped"` | `snapped` rounds angles to 15° steps and straightens shapes within 5° of level, `upright` keeps shapes level, `free` keeps the angle as drawn (and never turns a near-square into a square). |
| `kinds` | string[] (line \| curve \| arrow \| rectangle \| square \| ellipse \| circle \| triangle \| diamond \| pentagon \| hexagon \| octagon \| star \| heart \| cloud \| scribble) | no |  | Only recognize these shapes (`square` means rectangle, `circle` means ellipse). |
| `min_confidence` | number | no |  | Keep the ink (shape null) below this confidence. By default each shape has its own tuned cutoff. |

## Output

| Field | Type | Description |
|---|---|---|
| `shape` | string | What was drawn: `line`, `curve`, `arrow`, `rectangle`, `square`, `ellipse`, `circle`, `triangle`, `diamond`, `pentagon`, `hexagon`, `octagon`, `star`, `heart`, `cloud` or `scribble`. Null when it isn't a shape: keep the ink. |
| `kind` | string | The shape family (`rectangle` for squares, `ellipse` for circles). |
| `confidence` | number | 0 to 1. |
| `candidates` | object[] | Top 3 guesses with `shape` and `score`. |
| `closed` | boolean | True for closed outlines. |
| `outline` | number[][] | The clean shape as [x, y] points, ready to draw. |
| `center` | number[] | Rectangles, ellipses, stars and polygons. |
| `width` | number | Rectangles (with `height`, `rotation` in degrees and `corners`). |
| `rx` | number | Ellipses (with `ry`, `rotation`, and `radius` for circles). |
| `corners` | number[][] | Rectangles, triangles, diamonds and polygons. |
| `tips` | number[][] | Stars (with `points`, `outer_radius`, `inner_radius`, `rotation`). |
| `from` | number[] | Lines, curves and arrows, with `to`. Lines and arrows add `length` and `angle`; curves add `controls`. |
| `arrowheads` | object[] | Arrows: each head, and `heads` (`end` or `both`). |
| `image` | object | Image input only: its `width` and `height`; coordinates are in its pixels. |

The response envelope is always `{"id", "object": "run", "model", "created", "output", "usage": {"units", "unit", "cost_usd", "cost_micros"}, "latency_ms"}`. With an array input, `output` is an array in the same order.

## Examples (real outputs from our servers)

### Rectangle

```json
{
  "model": "shapes-1",
  "input": [
    [
      [
        1242,
        409
      ],
      [
        1197,
        436
      ],
      [
        1152,
        464
      ],
      [
        1108,
        492
      ],
      [
        1065,
        522
      ],
      [
        1022,
        551
      ],
      [
        975,
        574
      ],
      [
        932,
        548
      ],
      [
        902,
        505
      ],
      [
        873,
        462
      ],
      [
        843,
        418
      ],
      [
        816,
        374
      ],
      [
        827,
        325
      ],
      [
        868,
        293
      ],
      [
        911,
        263
      ],
      [
        954,
        232
      ],
      [
        997,
        202
      ],
      [
        1040,
        172
      ],
      [
        1082,
        142
      ],
      [
        1126,
        112
      ],
      [
        1169,
        82
      ],
      [
        1212,
        52
      ],
      [
        1256,
        24
      ],
      [
        1306,
        21
      ],
      [
        1343,
        57
      ],
      [
        1374,
        100
      ],
      [
        1405,
        142
      ],
      [
        1436,
        184
      ],
      [
        1460,
        230
      ],
      [
        1439,
        275
      ],
      [
        1397,
        306
      ],
      [
        1354,
        336
      ],
      [
        1311,
        366
      ],
      [
        1267,
        395
      ],
      [
        1224,
        424
      ],
      [
        1179,
        452
      ]
    ]
  ]
}
```

```json
{
  "shape": "rectangle",
  "kind": "rectangle",
  "confidence": 0.942,
  "center": [
    1136.2,
    293.55
  ],
  "width": 605.79,
  "height": 304.83,
  "rotation": -30,
  "corners": [
    [
      797.68,
      313
    ],
    [
      1322.31,
      10.1
    ],
    [
      1474.73,
      274.09
    ],
    [
      950.1,
      576.99
    ]
  ],
  "outline": [
    [
      797.68,
      313
    ],
    [
      1322.31,
      10.1
    ],
    [
      1474.73,
      274.09
    ],
    [
      950.1,
      576.99
    ]
  ],
  "closed": true,
  "candidates": [
    {
      "shape": "rectangle",
      "score": 0.942
    },
    {
      "shape": "ellipse",
      "score": 0.02
    },
    {
      "shape": "hexagon",
      "score": 0.016
    }
  ]
}
```

16 ms, $0.000005.

### Circle

```json
{
  "model": "shapes-1",
  "input": [
    [
      [
        228,
        544
      ],
      [
        237,
        565
      ],
      [
        250,
        584
      ],
      [
        266,
        602
      ],
      [
        286,
        616
      ],
      [
        309,
        626
      ],
      [
        333,
        633
      ],
      [
        356,
        637
      ],
      [
        381,
        637
      ],
      [
        405,
        632
      ],
      [
        428,
        623
      ],
      [
        449,
        612
      ],
      [
        469,
        597
      ],
      [
        484,
        578
      ],
      [
        497,
        557
      ],
      [
        506,
        534
      ],
      [
        511,
        510
      ],
      [
        510,
        485
      ],
      [
        507,
        461
      ],
      [
        500,
        438
      ],
      [
        489,
        416
      ],
      [
        475,
        396
      ],
      [
        457,
        379
      ],
      [
        437,
        366
      ],
      [
        416,
        357
      ],
      [
        392,
        351
      ],
      [
        368,
        350
      ],
      [
        344,
        354
      ],
      [
        319,
        360
      ],
      [
        298,
        371
      ],
      [
        277,
        385
      ],
      [
        259,
        400
      ],
      [
        245,
        417
      ],
      [
        232,
        439
      ],
      [
        223,
        463
      ],
      [
        212,
        463
      ]
    ]
  ]
}
```

```json
{
  "shape": "circle",
  "kind": "ellipse",
  "confidence": 0.93,
  "center": [
    366.34,
    494.37
  ],
  "rx": 144.3,
  "ry": 144.3,
  "rotation": 0,
  "radius": 144.3,
  "outline": [
    [
      510.63,
      494.37
    ],
    [
      509.94,
      508.51
    ],
    [
      507.86,
      522.52
    ],
    [
      504.42,
      536.25
    ],
    [
      499.65,
      549.59
    ],
    [
      493.6,
      562.39
    ],
    [
      486.32,
      574.53
    ],
    [
      477.88,
      585.91
    ],
    [
      468.37,
      596.4
    ],
    [
      457.88,
      605.91
    ],
    [
      446.5,
      614.34
    ],
    [
      434.36,
      621.62
    ],
    [
      421.56,
      627.68
    ],
    [
      408.23,
      632.45
    ],
    [
      394.49,
      635.89
    ],
    [
      380.48,
      637.97
    ],
    [
      366.34,
      638.66
    ],
    [
      352.19,
      637.97
    ],
    [
      338.19,
      635.89
    ],
    [
      324.45,
      632.45
    ],
    [
      311.12,
      627.68
    ],
    [
      298.32,
      621.62
    ],
    [
      286.17,
      614.34
    ],
    [
      274.8,
      605.91
    ],
    [
      264.3,
      596.4
    ],
    [
      254.8,
      585.91
    ],
    [
      246.36,
      574.53
    ],
    [
      239.08,
      562.39
    ],
    [
      233.03,
      549.59
    ],
    [
      228.25,
      536.25
    ],
    [
      224.81,
      522.52
    ],
    [
      222.74,
      508.51
    ],
    [
      222.04,
      494.37
    ],
    [
      222.74,
      480.22
    ],
    [
      224.81,
      466.21
    ],
    [
      228.25,
      452.48
    ],
    [
      233.03,
      439.15
    ],
    [
      239.08,
      426.34
    ],
    [
      246.36,
      414.2
    ],
    [
      254.8,
      402.82
    ],
    [
      264.3,
      392.33
    ],
    [
      274.8,
      382.82
    ],
    [
      286.17,
      374.39
    ],
    [
      298.32,
      367.11
    ],
    [
      311.12,
      361.05
    ],
    [
      324.45,
      356.28
    ],
    [
      338.19,
      352.84
    ],
    [
      352.19,
      350.76
    ],
    [
      366.34,
      350.07
    ],
    [
      380.48,
      350.76
    ],
    [
      394.49,
      352.84
    ],
    [
      408.23,
      356.28
    ],
    [
      421.56,
      361.05
    ],
    [
      434.36,
      367.11
    ],
    [
      446.5,
      374.39
    ],
    [
      457.88,
      382.82
    ],
    [
      468.37,
      392.33
    ],
    [
      477.88,
      402.82
    ],
    [
      486.32,
      414.2
    ],
    [
      493.6,
      426.34
    ],
    [
      499.65,
      439.15
    ],
    [
      504.42,
      452.48
    ],
    [
      507.86,
      466.21
    ],
    [
      509.94,
      480.22
    ]
  ],
  "closed": true,
  "candidates": [
    {
      "shape": "ellipse",
      "score": 0.93
    },
    {
      "shape": "curve",
      "score": 0.01
    },
    {
      "shape": null,
      "score": 0.009
    }
  ]
}
```

2 ms, $0.000005.

### Arrow

```json
{
  "model": "shapes-1",
  "input": [
    [
      [
        114,
        90
      ],
      [
        138,
        100
      ],
      [
        161,
        109
      ],
      [
        185,
        119
      ],
      [
        209,
        129
      ],
      [
        233,
        138
      ],
      [
        256,
        149
      ],
      [
        280,
        158
      ],
      [
        304,
        169
      ],
      [
        328,
        178
      ],
      [
        351,
        189
      ],
      [
        375,
        198
      ],
      [
        399,
        208
      ],
      [
        423,
        218
      ],
      [
        447,
        227
      ],
      [
        471,
        237
      ],
      [
        495,
        246
      ],
      [
        519,
        256
      ],
      [
        542,
        266
      ],
      [
        566,
        276
      ],
      [
        590,
        285
      ],
      [
        614,
        296
      ],
      [
        638,
        306
      ],
      [
        661,
        317
      ],
      [
        685,
        327
      ],
      [
        709,
        336
      ]
    ],
    [
      [
        612,
        383
      ],
      [
        622,
        379
      ],
      [
        632,
        375
      ],
      [
        643,
        371
      ],
      [
        653,
        366
      ],
      [
        663,
        361
      ],
      [
        674,
        357
      ],
      [
        684,
        352
      ],
      [
        694,
        347
      ],
      [
        704,
        342
      ],
      [
        714,
        338
      ],
      [
        709,
        328
      ],
      [
        702,
        319
      ],
      [
        696,
        310
      ],
      [
        689,
        300
      ],
      [
        684,
        291
      ],
      [
        678,
        281
      ],
      [
        673,
        271
      ],
      [
        667,
        261
      ],
      [
        661,
        252
      ],
      [
        654,
        243
      ],
      [
        649,
        233
      ]
    ]
  ]
}
```

```json
{
  "shape": "arrow",
  "kind": "arrow",
  "confidence": 0.971,
  "from": [
    114.08,
    89.81
  ],
  "to": [
    713.58,
    337.46
  ],
  "length": 648.64,
  "angle": 22.44,
  "heads": "end",
  "arrowheads": [
    {
      "tip": [
        713.58,
        337.46
      ],
      "left": [
        642.23,
        244.65
      ],
      "right": [
        597.53,
        352.85
      ],
      "length": 117.07
    }
  ],
  "outline": [
    [
      114.08,
      89.81
    ],
    [
      713.58,
      337.46
    ]
  ],
  "closed": false,
  "candidates": [
    {
      "shape": "arrow",
      "score": 0.971
    },
    {
      "shape": "line",
      "score": 0.004
    },
    {
      "shape": "octagon",
      "score": 0.004
    }
  ]
}
```

4 ms, $0.000005.

### Triangle, free angle

```json
{
  "model": "shapes-1",
  "input": [
    [
      [
        463,
        25
      ],
      [
        466,
        26
      ],
      [
        471,
        30
      ],
      [
        475,
        34
      ],
      [
        481,
        36
      ],
      [
        487,
        40
      ],
      [
        492,
        41
      ],
      [
        494,
        38
      ],
      [
        493,
        32
      ],
      [
        494,
        27
      ],
      [
        494,
        22
      ],
      [
        494,
        16
      ],
      [
        493,
        9
      ],
      [
        492,
        5
      ],
      [
        490,
        -1
      ],
      [
        488,
        -4
      ],
      [
        485,
        -4
      ],
      [
        480,
        -4
      ],
      [
        475,
        -3
      ],
      [
        468,
        -1
      ],
      [
        464,
        0
      ],
      [
        457,
        3
      ],
      [
        452,
        3
      ],
      [
        446,
        5
      ],
      [
        442,
        6
      ],
      [
        437,
        7
      ],
      [
        434,
        7
      ],
      [
        435,
        9
      ],
      [
        441,
        12
      ],
      [
        445,
        14
      ],
      [
        450,
        17
      ],
      [
        453,
        19
      ],
      [
        457,
        23
      ],
      [
        461,
        26
      ],
      [
        466,
        31
      ],
      [
        469,
        31
      ]
    ]
  ],
  "rotation": "free"
}
```

```json
{
  "shape": "triangle",
  "kind": "triangle",
  "confidence": 0.756,
  "center": [
    473.24,
    15.08
  ],
  "corners": [
    [
      434.23,
      7.9
    ],
    [
      491.99,
      -6.83
    ],
    [
      493.52,
      44.18
    ]
  ],
  "rotation": 40.54,
  "outline": [
    [
      434.23,
      7.9
    ],
    [
      491.99,
      -6.83
    ],
    [
      493.52,
      44.18
    ]
  ],
  "closed": true,
  "candidates": [
    {
      "shape": "triangle",
      "score": 0.756
    },
    {
      "shape": null,
      "score": 0.056
    },
    {
      "shape": "rectangle",
      "score": 0.044
    }
  ]
}
```

95 ms, $0.000005.

### Photo of a sketch

```json
{
  "model": "shapes-1",
  "input": "https://labs.greatwork.company/examples/sketch-star.png"
}
```

```json
{
  "shape": "star",
  "kind": "star",
  "confidence": 0.845,
  "center": [
    189.38,
    190.03
  ],
  "points": 6,
  "outer_radius": 179.97,
  "inner_radius": 120.75,
  "rotation": 0,
  "tips": [
    [
      189.38,
      10.06
    ],
    [
      345.24,
      100.04
    ],
    [
      345.24,
      280.01
    ],
    [
      189.38,
      370
    ],
    [
      33.53,
      280.01
    ],
    [
      33.53,
      100.04
    ]
  ],
  "outline": [
    [
      189.38,
      10.06
    ],
    [
      249.76,
      85.45
    ],
    [
      345.24,
      100.04
    ],
    [
      310.14,
      190.03
    ],
    [
      345.24,
      280.01
    ],
    [
      249.76,
      294.6
    ],
    [
      189.38,
      370
    ],
    [
      129.01,
      294.6
    ],
    [
      33.53,
      280.01
    ],
    [
      68.63,
      190.03
    ],
    [
      33.53,
      100.04
    ],
    [
      129.01,
      85.45
    ]
  ],
  "closed": true,
  "candidates": [
    {
      "shape": "star",
      "score": 0.845
    },
    {
      "shape": "octagon",
      "score": 0.041
    },
    {
      "shape": "hexagon",
      "score": 0.037
    }
  ],
  "image": {
    "width": 384,
    "height": 384
  }
}
```

370 ms, $0.00005.

## Good for

- Snap whiteboard sketches to clean shapes
- Draw diagrams and flowcharts with a pen or finger
- Shape recognition in note-taking and drawing apps
- Turn photos of sketches into vector outlines

## Limits and caveats

- About 1 in 10 doodles that aren't shapes still get read as one: raise `min_confidence` or limit `kinds` when that matters.
- Octagons are the weakest shape (80% in our tests).
- One shape per drawing: send each shape on a page as its own drawing.
- Images are traced into strokes first, so dark lines on a plain background work best.
