Skip to content

Rectangle Overlap

Check two boxes without wrapped coordinates, distinguish touching from overlap, and reuse the existing C64 helper.

Two boxes share some space: that is enough to ask whether a player touched an enemy, entered a doorway or reached a collectible. You do not need to compare every visible pixel to answer that first question.

Code status: assembled and emulator-executed. This standalone caller reuses the maintained Collect and Respond helper unchanged. PAL C64, stock recorded ROMs, Asm198x 0.0.72 and Emu198x 0.44.2 are the qualified configuration.

A and B mark two boxes, X marks their shared cells, and the result is 1.
The first digit says overlap. X cells make the shared area visible; these characters are a diagnostic grid, not a sprite engine.

Put the helper in a project

Each rectangle is four edges: left, top, right, bottom. Write those bytes into rect_a and rect_b, then call boxes_overlap. Carry set means overlap; carry clear means apart or empty. A and flags change, X and Y survive. Include the helper once outside a fall-through execution path. It needs no initialisation, allocation or I/O; it owns eight edge bytes and uses the caller’s normal JSR stack. Do not let an interrupt change those edges during a call.

The maintained eight-edge helper

collision.inc
; Half-open boxes [left,top,right,bottom]; all edges unsigned bytes.
; Reject empty/reversed boxes and touching edges. Returns C=1 on overlap.
; Changes A/flags; preserves X/Y. No arithmetic inside the test.
boxes_overlap:
    lda rect_a
    cmp rect_a+2
    bcs boxes_apart
    lda rect_a+1
    cmp rect_a+3
    bcs boxes_apart
    lda rect_b
    cmp rect_b+2
    bcs boxes_apart
    lda rect_b+1
    cmp rect_b+3
    bcs boxes_apart
    lda rect_a
    cmp rect_b+2
    bcs boxes_apart
    lda rect_b
    cmp rect_a+2
    bcs boxes_apart
    lda rect_a+1
    cmp rect_b+3
    bcs boxes_apart
    lda rect_b+1
    cmp rect_a+3
    bcs boxes_apart
    sec
    rts
boxes_apart:
    clc
    rts
rect_a: !byte 0,0,0,0
rect_b: !byte 10,5,11,6

Right and bottom are excluded: [2,2,8,6] is six units wide and four high. Adjacent boxes can share a boundary without sharing area. Zero and reversed extents return false. All edges must fit 0–255. This is a local/grid coordinate helper; a 320-pixel C64 screen needs a wider representation.

There is no addition inside the test. If your objects store position and size, calculate right/bottom in wider arithmetic and validate or clip before storing bytes. Letting 250+16 wrap to 10 changes the geometry before the helper sees it. Negative coordinates need translation or a separately implemented signed/wider routine.

Run and read the caller

From code-samples/commodore-64/patterns/assembly/physics/rectangle-overlap, run make with Asm198x on PATH. Load build/demo.prg and type RUN. Reset to leave. The caller owns ROM banking, interrupts, the display and renderer pointer bytes $FB/$FC; the collision helper owns none of that hardware.

Include order and example geometry

demo.asm
* = $0801
    !byte $0b,$08,$0a,$00,$9e
    !text "2064"
    !byte 0,0,0
* = $0810
!source "cases.inc"
!source "caller.inc"
helper_start:
!source "../../framework/collect-and-respond/collision.inc"
helper_end:
example: !byte 2,2,8,6,6,4,11,7
!source "case-table.inc"

A standalone caller and a separate diagnostic renderer

caller.inc
; Standalone PAL caller. Owns display, ROM banking, interrupts, $fb/$fc.
start:
    sei
    lda #$7f
    sta $dc0d
    sta $dd0d
    lda $dc0d
    lda $dd0d
    lda #$35
    sta $01
    lda #<nmi_return
    sta $fffa
    lda #>nmi_return
    sta $fffb
    lda #0
    sta $dc0e
    sta $dc0f
    sta $d015
    sta $d020
    sta $d021
    lda $dd02
    ora #3
    sta $dd02
    lda $dd00
    ora #3
    sta $dd00
    lda #$14
    sta $d018
    lda #$1b
    sta $d011
    lda #8
    sta $d016
    ldx #0
clear_screen:
    lda #32
    sta $0400,x
    sta $0500,x
    sta $0600,x
    sta $06e8,x
    lda #1
    sta $d800,x
    sta $d900,x
    sta $da00,x
    sta $dae8,x
    inx
    bne clear_screen
; A guest-owned second caller traverses supplied cases; the host only observes.
    lda #<case_table
    sta $fb
    lda #>case_table
    sta $fc
    ldx #0
case_loop:
    ldy #7
copy_case:
    lda ($fb),y
    sta rect_a,y
    dey
    bpl copy_case
    ldy #$5a
    jsr boxes_overlap
    lda #0
    rol
    sta results,x
    tya
    cmp #$5a
    beq preserved_y
    jmp failed
preserved_y:
    clc
    lda $fb
    adc #8
    sta $fb
    bcc next_case
    inc $fc
next_case:
    inx
    cpx #CASE_COUNT
    bne case_loop
    lda #$a5
    sta completed
; Customisation is data: replace B's four edges, leaving the routine unchanged.
    ldx #7
copy_example:
    lda example,x
    sta rect_a,x
    dex
    bpl copy_example
    jsr boxes_overlap
    lda #0
    rol
    sta result
    ora #48
    sta $0400
; Draw a small diagnostic grid. It is deliberately separate from collision.
    lda #<$0428
    sta $fb
    lda #>$0428
    sta $fc
    lda #0
    sta row
row_loop:
    ldy #0
column_loop:
    lda #32
    sta cell
    cpy rect_a
    bcc test_b
    cpy rect_a+2
    bcs test_b
    lda row
    cmp rect_a+1
    bcc test_b
    cmp rect_a+3
    bcs test_b
    lda #1
    sta cell
test_b:
    cpy rect_b
    bcc put_cell
    cpy rect_b+2
    bcs put_cell
    lda row
    cmp rect_b+1
    bcc put_cell
    cmp rect_b+3
    bcs put_cell
    lda cell
    cmp #1
    bne only_b
    lda #24
    bne save_cell
only_b:
    lda #2
save_cell:
    sta cell
put_cell:
    lda cell
    sta ($fb),y
    iny
    cpy #40
    bne column_loop
    clc
    lda $fb
    adc #40
    sta $fb
    bcc next_row
    inc $fc
next_row:
    inc row
    lda row
    cmp #10
    bne row_loop
    lda #$ef
    sta ready
hold:
    jmp hold
failed:
    jmp failed
nmi_return: rti
row: !byte 0
cell: !byte 0
completed: !byte 0
ready: !byte 0
result: !byte 0
results: !fill CASE_COUNT,0

The normal result is 1. The renderer labels A-only cells A, B-only cells B and shared cells X. Its 40×10 viewport is a display choice; it does not define the helper’s numerical range. In your own game, handle the carry result immediately and keep the response—damage, collection or a room change—separate from the geometry.

Customise the shape, keep the contract

The custom caller moves B until its left edge equals A’s right edge. No X cells remain and the result becomes 0. Use smaller player hitboxes for a forgiving game or larger interaction boxes for easy collection. The visible artwork and interaction bounds need not be identical.

Touching boxes: a data-only customisation

custom.asm
* = $0801
    !byte $0b,$08,$0a,$00,$9e
    !text "2064"
    !byte 0,0,0
* = $0810
!source "cases.inc"
!source "caller.inc"
helper_start:
!source "../../framework/collect-and-respond/collision.inc"
helper_end:
example: !byte 2,2,8,6,8,2,13,6
!source "case-table.inc"

A common error: counting an edge as area

Adjacent A and B cells have no X cells, but the faulty caller displays 1.
A useful contradiction: the geometry has no shared cell, yet the damaged comparison reports a collision.

Deliberate fault: accepting equality

wrong-touch.inc
; Deliberate fault: accept equality at separation boundaries.
; Half-open boxes [left,top,right,bottom]; all edges unsigned bytes.
; Reject empty/reversed boxes and touching edges. Returns C=1 on overlap.
; Changes A/flags; preserves X/Y. No arithmetic inside the test.
boxes_overlap:
    lda rect_a
    cmp rect_a+2
    bcs boxes_apart
    lda rect_a+1
    cmp rect_a+3
    bcs boxes_apart
    lda rect_b
    cmp rect_b+2
    bcs boxes_apart
    lda rect_b+1
    cmp rect_b+3
    bcs boxes_apart
    lda rect_a
    cmp rect_b+2
    beq wrong_accept_5
    bcs boxes_apart
wrong_accept_5:
    lda rect_b
    cmp rect_a+2
    beq wrong_accept_6
    bcs boxes_apart
wrong_accept_6:
    lda rect_a+1
    cmp rect_b+3
    beq wrong_accept_7
    bcs boxes_apart
wrong_accept_7:
    lda rect_b+1
    cmp rect_a+3
    beq wrong_accept_8
    bcs boxes_apart
wrong_accept_8:
    sec
    rts
boxes_apart:
    clc
    rts
rect_a: !byte 0,0,0,0
rect_b: !byte 10,5,11,6

The labelled wrong helper accepts equality in the four separation checks. Empty/reversed boxes still fail, so the error is specifically about touching. Restore the original comparisons; repaired has the same geometry and executable payload as custom and returns 0. Keep the faulty build separate.

Costs, limits and checks

The helper is 68 instruction bytes plus eight state bytes. Early rejection means execution cost depends on the boxes. The full working caller is 2,085 payload bytes, including diagnostic drawing and a supplied 184-case test set. Those are different costs. The fault adds eight instruction bytes. These figures do not establish a cycle budget.

The guest traverses 184 independent supplied cases, including argument reversal, zero/reversed extents, containment, separated and touching edges, numerical maxima and deterministic generated inputs. Positive completion guards and X/Y preservation checks must finish. The observer compares outputs with a separate max/min interval oracle, the full imported program before RUN and the native screenshot glyphs. Corrupted results, code, provenance and pixel expectations are rejected by the evidence gate.

Box tests are a useful broad phase before expensive mask/pixel tests on a small set of candidates. That second phase still has its own CPU, memory and data costs. Maths for Games develops intervals and coordinate ranges; the cross-platform collision explanation covers the same geometry independently of an integer width.

Maintained callers and README · Native evidence and configuration. Re-run scripts/check-c64-rectangle-overlap.py and its falsification tests from the samples root. This is named emulator qualification, not a physical C64 or every display configuration.