Hidden vector graphics (3D)

Exactly. There are basically 2 types of techniques here:
(1) "culling" techniques. These filter out triangles that you know for sure you do not want to draw. There are two standard culling techniques used in 3d renderers:
(1.a) "hidden face culling" (or sometimes called "back face culling") (what Sandy Brand was explaining). This is quite straight forward: since a triangle is made out of 3 points (a, b, c), you take the vector a->b, and the vector b->c, and compute the normal of the triangle. Then a simple dot product of that normal with the "camera direction vector" tells you if your triangle is facing the opposite direction (positive dot product) and does not need to be drawn, or is facing the camera (negative dot product), and needs to be drawn. The one thing to be careful in this technique is that you must be careful when defining the a, b, c points of your triangle, so that they are always specified in a "clockwise" order. Otherwise, the normal vector will point in the opposite direction and your dot product will have a flipped sign.
(1.b) "frustum culling": check if the triangle falls completely outside of the frustum volume that will be rendered by the camera, and thus can also be ignored.

(2) "Occlusion" techniques: these are techniques so that triangles that are closer to the camera occlude those that are further behind, and not the other way around. The classic algorithm is the "painter's algorithm" (sorting all the triangles and drawing back to front). This works well for small triangles and for a small number of triangles, but there are situations where it will not work well (specially if triangles are large). The better algorithm is the use of a z-buffer (which is unfortunately unfeasible in small 8-bit machines, as it requires storing one scalar per pixel in the screen, and it is likely we do not have enough RAM for that ).

Anyway, I hope at least the names of the different algorithms help you find what you are looking for! Of course, there are many more culling/occlusion techniques used in modern renderers, but for small 8-bit machines, probably a subset of the above is enough

In my 3D routine I stored the normal vector of each triangle in my data structure. Then I only calculate the Z-component of the 3D rotated normal vector and use the sign of the Z-component to see if I need to actually calculate and draw the other 3 points of the triangle. And then cache those 3 points so that you do not have to recalculate them for adjacent triangles... ;)

You have to be ware that this will not work if you add depth/perspective adjustment. Since in such case the Z-component might indicate a visible triangle while it will become invisible after perspective correction.

I never published that code (for now). I'll probably do so someday but the source is somewhere on an old DD disk that is in storage on an attic far far away, so don't expect anything in the near future.
Of course I'm willing to answer any questions you might have...

Try reading computer graphics books for computer science courses from 1980s and early 1990s, they cover basic 3D rasterization methods.
Also I think there was pc-8801 or msx based book on the subject as well, reading old books about computers is always fun.

Basically I would say that depending on what you want to do it may be fastest to do no checks at all and just paint pixels multiple times. With convex shapes and back face culling there won’t be much overlap anyway. The alternative is to use a z-buffer, but the cost of keeping track of that may be higher than just writing out the pixels, although you would be able to omit the depth sort.

p.s. Check turbor’s reply in this older thread.