https://tedneward.github.io/Research JBake Bootstrap Template en-gb Mon, 13 Apr 2026 14:06:08 +0000 Mon, 13 Apr 2026 14:06:08 +0000 https://tedneward.github.io/Research/writing/writing-tools/index.html Mon, 13 Apr 2026 14:04:56 +0000 writing/writing-tools/index.html <p><em>(by Roy Peter Clark)</em></p> <h3>Part One: Nuts and Bolts</h3> <h4>Tool 1: Begin sentences with subjects and verbs. Make meaning early, then let weaker elements branch to the right.</h4> <ul> <li>Subject and verb are often separated in prose, usually because we want to tell the reader something about the subject before we get to the verb. This delay, even for good reasons, risks confusing the reader.</li> <li>To create suspense, or build tension, or make the reader wait and wonder, or join a journey of discovery, or hold on for dear life, he can save subject and verb of the main clause until later.</li> </ul> <h4>Tool 2: Order words for emphasis. Place strong words at the beginning and at the end.</h4> <ul> <li>The period acts as a stop sign. That slight pause in reading magnifies the final word, an effect intensified at the end of a paragraph, where final words often adjoin white space.</li> <li>Putting strong stuff at the beginning and end helps writers hide weaker stuff in the middle.</li> </ul> <h4>Tool 3: Activate your verbs. Strong verbs create action, save words, and reveal the players.</h4> <ul> <li>Verbs are either active, passive, or a linking verb, which is a form of the verb <em>to be</em>.</li> </ul> <h4>Tool 4: Be passive-aggressive. Use passive verbs to showcase the “victim” of action.</h4> <ul> <li>A strong active verb can add dimension to the cloud created by some uses of the verb to be, such as replacing "there were leaves all over the ground" with "leaves cover the ground."</li> <li>Active verbs move the action and reveal the actors. Passive verbs emphasize the receiver, the victim. The verb <em>to be</em> links word and ideas.</li> </ul> <h4>Tool 5: Watch those adverbs. Use them to change the meaning of the verb.</h4> <ul> <li>At their best, adverbs spice up a verb or adjective. At their worst, they express a meaning already contained in it.</li> <li>A bad adverb is repetitive: "She smiled happily." A good adverb can change the meaning: "She smiled sadly."</li> <li>Look for weak verb-adverb combinations that you can revise with stronger verbs.</li> </ul> <h4>Tool 6: Take it easy on the -ings. Prefer the simple present or past.</h4> <ul> <li>Add -ing to a verb, and it takes on a progressive sense, or a happening.</li> <li>"Wish and hope and think and pray" is stronger than "Wishing and hoping and thinking and praying."</li> </ul> <h4>Tool 7: Fear not the long sentence. Take the reader on a journey of language and meaning.</h4> <ul> <li>While length makes a bad sentence worse, it can make a good sentence better.</li> <li>Some strategies to achieve mastery of the long sentence: It helps if subject and verb of the main clause come early in the sentence. Use the long sentence to describe something long. Let form follow function. It helps if the long sentence is written in chronological order. Use the long sentence in variation with sentences of short and medium length.</li> </ul> <h4>Tool 8: Establish a pattern, then give it a twist. Build parallel constructions, but cut across the grain.</h4> <ul> <li>If two or more ideas are parallel, they are easier to grasp when expressed in parallel grammatical form. Single words should be balanced with single words, phrases with phrases, clauses with clauses.</li> <li>A pure parallel construction would be "Boom, boom, boom." Parallelism with a twist gives us "Boom, boom, bang," like "hither, thither, and yon," "Peter, Paul, and Mary," or "sex, drugs, and rock n roll."</li> </ul> <h4>Tool 9: Let punctuation control pace and space. Learn the rules, but realize you have more options than you think.</h4> <ul> <li>A well-placed comma points to where the writer would pause if he read the passage aloud.</li> <li>More muscular than the comma, the semicolon is most useful for dividing and organizing big chunks of information.</li> <li>The dash has become the default mark for writers who never mastered the formal rules. But the dash has two brilliant uses: a pair of dashes can set off an idea contained within a sentence, and a dash near the end can deliver a punch line.</li> </ul> <h4>Tool 10: Cut big, then small. Prune the big limbs, then shake out the dead leaves.</h4> <ul> <li>Brevity comes from selection, not compression, a lesson that requires lifting blocks from the work.</li> <li>Cut any passage that does not support your focus. Don't invite others to cut. You know the work better. Mark optional trims. Then decide whether they should become actual cuts.</li> <li>Targets for cuts include: Adverbs that intensify rather than modify, prepositional phrases that repeat the obvious, phrases that grow on verbs, abstract nouns that hide active verbs, and restatements.</li> </ul> <h3>Part Two: Special Effects</h3> <h4>Tool 11: Prefer the simple over the technical. Use shorter words, sentences, and paragraphs at points of complexity.</h4> <ul> <li>Familiarization takes the strange or opaque or complex and, through the power of explanation, makes it comprehensible, even familiar.</li> <li>Readers benefit from shorter words and phrases, and simpler sentences, at the points of greatest complexity.</li> <li>Clear prose is not just a product of sentence length and word choice. It derives first from a sense of purpose -- a determination to inform.</li> </ul> <h4>Tool 12: Give key words their space. Do not repeat a distinctive word unless you intend a specific effect.</h4> <ul> <li>Repeat words or phrases for emphasis or rhythm.</li> <li>Some words act as building blocks and can be repeated to good effect, while distinctive words deserve their own space.</li> <li>Leave <em>said</em> alone. Don't be tempted by the muse of variation to permit characters to opine, elaborate, cajole, or chortle.</li> </ul> <h4>Tool 13: Play with words, even in serious stories. Choose words the average writer avoids but the average reader understands.</h4> <ul> <li>Too often, writers suppress their vocabularies in a misguided attempt to lower the level of language for a general audience.</li> <li>A rich writing vocabulary does not require big or fancy words.</li> <li>All of us possess a reading vocabulary as big as a lake but draw from a writing vocabulary as small as a pond.</li> </ul> <h4>Tool 14: Get the name of the dog. Dig for the concrete and specific, details that appeal to the senses.</h4> <ul> <li>When details of character and setting appeal to the senses, they create an experience for the reader that leads to understanding.</li> <li>The good writer uses telling details not only to inform, but to persuade.</li> </ul> <h4>Tool 15: Pay attention to names. Interesting names attract the writer— and the reader.</h4> <ul> <li>The best reporters recognize and take advantage of coincidence between name and circumstance.</li> </ul> <h4>Tool 16: Seek original images. Reject clichés and first-level creativity.</h4> <ul> <li>Using metaphors, similes, or other figures of speech which you are used to seeing in print is a substitute for thinking, a form of automatic writing.</li> <li>If brianstorming does not lead you to an inspired image, write it straight.</li> <li>Worse than clichés of language are "clichés of vision," or the narrow frames through which writers learn to see the world. E.g. victims are innocent, politicians are corrupt, and it's lonely at the top.</li> </ul> <h4>Tool 17: Riff on the creative language of others. Make word lists, free-associate, be surprised by language.</h4> <ul> <li>Writers collect sharp phrases and colorful metaphors, sometimes for their conversations and sometimes for their prose, but this risks plagiarism.</li> <li>The notion that new knowledge derives from old wisdom should liberate the writer from a scrupulous fear of snatching the words of others.</li> <li>Always take credit for good writing you did not intend because you'll be getting plenty of criticism for bad writing you did not mean either.</li> </ul> <h4>Tool 18: Set the pace with sentence length. Vary sentences to influence the reader's speed.</h4> <ul> <li>Long sentences create a flow that carries the reader down a stream of understanding, creating an effect called "steady advance." Short ones slam on the brakes.</li> <li>Writers slow the pace of a story for three reasons: To simplify the complex, to create suspense, and to focus on the emotional truth.</li> <li>Radical clarity eases the reader into a story with short sentences and paragraphs, where the stopping points give the reader time and space to comprehend, but there is enough variation to imitate the patterns of normal conversation.</li> <li>Short sentences can achieve not only clarity, but also suspense and emotional power. Some call this the "Jesus wept" effect.</li> </ul> <h4>Tool 19: Vary the lengths of paragraphs. Go short or long— or make a turn— to match your intent.</h4> <ul> <li>When the big parts fit, we call that good feeling coherence; when sentences connect, we call it cohesion.</li> <li>The paragraph is essentially a unit of thought, not of length. All sentences in a paragraph should be about the same thing and move in a sequence.</li> <li>While a writer can break a long paragraph into parts, they should not paste together paragraphs that are short and disconnected.</li> <li>A short paragraph, especially after a long one, to bring the reader to a sudden, dramatic stop.</li> <li>A solid, unified paragraph can take a turn in the middle, or demonstrate the logic of cause and effect.</li> </ul> <h4>Tool 20: Choose the number of elements with a purpose in mind. One, two, three, or four: each sends a secret message to the reader.</h4> <ul> <li>If a writer wants the reader to think something the absolute truth, the writer should render it in the shortest possible sentence.</li> <li>Telling two characteristics invites the reader to balance them, weigh them against each other, compare and contrast them.</li> <li>Three characteristics allows us to triangulate further characteristics, and provides a sense of the whole, such as "beginning, middle, and end."</li> <li>Listing four characteristics can list and expand, but can also twist the wholeness of three: "That girl is smart, sweet, determined, and neurotic."</li> </ul> <h4>Tool 21: Know when to back off and when to show off. When the topic is most serious, understate; when least serious, exaggerate.</h4> <ul> <li>If the subject is serious or dramatic, let the story tell itself. If the topic is playful or inconsequential, then the writer can show off.</li> </ul> <h4>Tool 22: Climb up and down the ladder of abstraction. Learn when to show, when to tell, and when to do both.</h4> <ul> <li>At the top of the ladder is general and abstract language; at the bottom is concrete evidence. Beware of the middle rungs.</li> <li>Metaphors and similes help us understand abstractions through comparison with concrete things, working both ends of the ladder.</li> <li>"Can you give me an example?" will drive the speaker down the ladder. But "What does that mean?" will carry him aloft.</li> </ul> <h4>Tool 23: Tune your voice. Read stories aloud.</h4> <ul> <li>Voice is the sum of all the strategies used by the author to create the illusion that the writer is speaking directly to the reader from the page. No effect is more important or elusive.</li> <li>Voice is influenced by: The level of language, the "person" or point of view, the range and source of allusions, the frequency of metaphors and figures of speech, sentence length and structure, degree of neutrality, and framing of material.</li> <li>To test your writing voice, read your story aloud to hear if it sounds like you. Do this to test what the deletion of an unnecessary phrase does to the rhythm of the sentence.</li> </ul> <h3>Part Three: Blueprints</h3> <h4>Tool 24: Work from a plan. Index the big parts of your work.</h4> <ul> <li>Good work has parts, and the reader who sees the big parts is more likely to remember the whole story.</li> <li>If you cannot write the outline from a story, then you cannot discern the parts from the whole, and this is a problem of organization.</li> <li>An outline, or even subtitles, reveal a movement of theme, logic, and chronology that readers can perceive and remember.</li> </ul> <h4>Tool 25: Learn the difference between reports and stories. Use one to render information, the other to render experience.</h4> <ul> <li>Readers read for two reasons: information and experience. Reports convey information, stories create experience. Reports transfer knowledge, stories transport the reader. Reports point us there, stories put us there.</li> <li>Transforming information to a narrative, who becomes character, what becomes action, where becomes setting, when becomes chronology, why becomes cause or motive, and how becomes process.</li> <li>By combining story and report, the writer can speak to both our hearts and our heads, creating sympathy and understanding.</li> </ul> <h4>Tool 26: Use dialogue as a form of action. Dialogue advances narrative; quotes delay it.</h4> <ul> <li>In many ways dialogue defines a story because its power drags us to the scene and sets our ears to the action.</li> <li>A good quote introduces a human voice, explains something important, frames a problem or dilemma, adds information, reveals the character or personality of the speaker, and introduces what is next to come.</li> <li>Most quotes are displaced because they are spoken above or outside the action, and not in the action. But dialogue is not just heard, but overheard.</li> </ul> <h4>Tool 27: Reveal traits of character. Show character-istics through scenes, details, and dialogue.</h4> <ul> <li>Too often, writers turn abstractions into adjectives to define character. The reader who encounters character adjectives screams silently for examples, for evidence.</li> <li>The best writers create moving pictures of people, images that reveal their characteristics and aspirations, their hopes and fears.</li> </ul> <h4>Tool 28: Put odd and interesting things next to each other. Help the reader learn from contrast.</h4> <ul> <li>Ironic juxtaposition is the fancy term for what happens when two disparate things are placed side by side, each commenting on the other.</li> <li>Examples include valley girl meets demons (Buffy the Vampire Slayer), Hitler meets musical (The Producers), and murder meets Christmas (Holidays on Ice).</li> </ul> <h4>Tool 29: Foreshadow dramatic events and powerful conclusions. Plant important clues early.</h4> <ul> <li>Playwright Anton Chekhov wrote "One must not put a loaded rifle on the stage if no one is thinking of firing it," a foreshadowing device called <em>Chekhov's Gun</em>.</li> </ul> <h4>Tool 30: To generate suspense, use internal cliffhangers. To propel readers, make them wait.</h4> <ul> <li>A cliffhanger creates suspsense, a word derived from the Latin suspendere, "to hang under." Suspense leaves the reader, and sometimes a character, hanging.</li> </ul> <h4>Tool 31: Build your work around a key question. Stories need an engine, a question that the action answers for the reader.</h4> <ul> <li>The engine is the question the story answers for the reader. If the internal cliffhanger drives the reader from one section to the next, the engine moves the reader across the arc from beginning to end.</li> <li>Create a cast of characters for your stories by asking the question: Who has something at stake here? The answer can lead to the creation of a story engine.</li> <li>The premise of Othello is, "Jealousy destroys itself and the object of its love." Its engine could be, "Will Othello's jealousy destroy him and the woman he loves?"</li> <li>The engine and its themes are distinct. "What is Rosebud?" is the engine of Citizen Kane, but its themes explore politics, democracy, and America.</li> <li>Some stories are driven not by what questions, but by how. For example, we know that Bond will conquer the villians and get the girl, but we are driven to know how.</li> </ul> <h4>Tool 32: Place gold coins along the path. Reward the reader with high points, especially in the middle.</h4> <ul> <li>A good start is its own reward, and crafty writers know enough to put something shiny at the end, a final reward. But put gold coins in the middle to reward the reader, and for motivation.</li> <li>A gold coin can appear as a small scene or anecdote, a startling fact, or a telling quote.</li> </ul> <h4>Tool 33: Repeat, repeat, and repeat. Purposeful repetition links the parts.</h4> <ul> <li>Repeating key words, phrases, and story elements creates a rhythm, a pace, a structure, a wavelength that reinforces the central theme of the work.</li> <li>If you're worried about too much repetition, delete all the repetition and read the passage aloud without it. Repeat the key element once. Repeat it again. Your voice and ear will let you know when you've gone too far.</li> </ul> <h4>Tool 34: Write from different cinematic angles. Turn your notebook into a camera.</h4> <ul> <li>An establishing shot captures the setting in which action takes place, describing the world that the reader is about to enter, sometimes creating a mood.</li> <li>A middle distance shot is closer to the action, close enough to see the key players and their interaction.</li> <li>A close-up shot gets in the face of the subject, close enough to detect anger, fear, dread, sorrow, irony, the full range of emotions.</li> <li>An extreme close-up shot focuses on an important detail that would be invisible from a distance.</li> </ul> <h4>Tool 35: Report and write for scenes. Then align them in a meaningful sequence.</h4> <ul> <li>Tom Wolfe argues that realism is built on "scene-by-scene construction, telling the story by moving from scene to scene and resorting as little as possible to sheer historical narrative."</li> <li>Scenes can be witnessed or, in fiction, invented, but they can also be remembered.</li> <li>Scenes can be arranged in space as well as in time, can be used to balance parallel narrative lines, shifting from the perspective of the criminal to the cop, and can flash back in time, or look ahead.</li> </ul> <h4>Tool 36: Mix narrative modes. Combine story forms using the broken line.</h4> <ul> <li>A movement between narrative and analysis both instructs and delights the reader.</li> <li>Any story that begins without the news requires a phrase, a sentence, a paragraph, a zone that answers the question "So what?"</li> <li>With the broken line form, the writer can begin with narrative and move to explanation, or begin with straight information and then illustrate the facts with an anecdote.</li> </ul> <h4>Tool 37: In short works, don’t waste a syllable. Shape short writing with wit and polish.</h4> <ul> <li>The author of a great big novel will waste a syllable, but in an ocean of words the reader will not notice. But the shorter the story form, the more precious is each word.</li> <li>Brevity gives short works a focused power; it creates opportunity for wit; and it inspires the writer to polish, to reveal the luster of the language.</li> <li>Longer works can contain powerful, witty, and polished shorter elements: anecdotes, scenes, descriptions, vignettes, set pieces that can be lifted out of the work for inspection and delight.</li> </ul> <h4>Tool 38: Prefer archetypes to stereotypes. Use subtle symbols, not crashing cymbals.</h4> <ul> <li>Good writers can be original by standing on a foundation of narrative archetypes, a set of story expectations that can be manipulated, frustrated, or fulfilled in novel ways on behalf of the reader.</li> <li>Examples include: the journey there and back, winning the prize, winning or losing the loved one, loss and restoration, the blessing becomes the curse, overcoming obstacles, the wasteland restored, rising from the ashes, the ugly duckling, the emperor has no clothes, and descent into the underworld.</li> <li>When it comes to powerful writing, a symbol need not be a cymbal. Subtlety is a writer’s virtue.</li> </ul> <h4>Tool 39: Write toward an ending. Help readers close the circle of meaning.</h4> <ul> <li>In written compositions, the author can build to a cresendo, fade out, stop short, echo the opening, close the circle, tieback to the body, tell what happens last in time or space, reveal a secret or solve a mystery, show an epilogue, offer solutions to a problem, use an apt quote, look to the future, or mobilize the reader.</li> <li>Remember that other parts of your story, like sentences and paragraphs, need endings too. Each of these mini-endings anticipates your finale.</li> <li>Don’t bury your ending. Put your hand over the last paragraph, and ask yourself "What would happen if this ended here?" Move it up another paragraph and repeat the question until you find the natural stopping place.</li> </ul> <h3>Part Four: Useful Habits</h3> <h4>Tool 40: Draft a mission statement for your work. To sharpen your learning, write about your writing.</h4> <ul> <li>Good writers turn stories into workshops, intense moments of learning in which they advance their craft.</li> <li>Writing down your mission, covering both content and format, turns your vague hopes into language. By writing about your writing, you learn what you need to learn.</li> <li>Such mission statements are an expression of purpose, which show up in introductions or epilogues.</li> </ul> <h4>Tool 41: Turn procrastination into rehearsal. Plan and write it first in your head.</h4> <ul> <li>To write with your hands, you must write in your head. Treat each act of procrastination as a time of mental planning and preparation.</li> <li>Eliminate writers block by lowering your standards until there is no felt threshold to go over in writing. Nothing is more liberating than no standards.</li> <li>Other strategies for eliminating procrastination: Let your fingers do the writing and not your brain, adopt a daily routine, build in rewards, write earlier to discover the information you need instead of over-researching, discount nothing, rewrite, eliminate negative words from your vocabulary, dispose of distracting tasks, find someone who praises productivity and effort, and keep a daybook for fleeting ideas.</li> </ul> <h4>Tool 42: Do your homework well in advance. Prepare yourself for the expected— and unexpected.</h4> <ul> <li>Good writers fill a reservoir of knowledge they can drain at a moment’s notice. As Hamlet said, "the readiness is all."</li> <li>Homework should answer what the point is, why this story is being told, and what it says about life, about the world, about the times we live in.</li> </ul> <h4>Tool 43: Read for both form and content. Examine the machinery beneath the text.</h4> <ul> <li>Reading other works for content and form informs you of what you're trying to build, and what tools you need to build it.</li> <li>When you can't put a story down, put it down and think about what magic propels you from paragraph to paragraph, page to page, chapter to chapter.</li> <li>Pay attention to the voice of the writer, undeveloped story ideas, and new storytelling forms. Read bits of books for taste, books directed by your writing compass, books on topics outside your discipline, and read with a pen nearby.</li> </ul> <h4>Tool 44: Save string. For big projects, save scraps others would toss.</h4> <ul> <li>Use a simple file box to save items, anecdotes, and statistics about a topic. It will fill up without effort, creating a literary life cycle: planting, cultivation, and harvesting.</li> <li>Grow several crops at a time, fertilizing one even as you harvest another.</li> </ul> <h4>Tool 45: Break long projects into parts. Then assemble the pieces into something whole.</h4> <ul> <li>At the front end, book projects seem impossible to get your arms around. Break long projects into parts, long stories into chapters, long chapters into episodes to make the task of writing a book tractable.</li> </ul> <h4>Tool 46: Take an interest in all crafts that support your work. To do your best, help others do their best.</h4> <ul> <li>If your story is well edited, accompanied by a powerful photograph, on a page that is well designed, it will look more important and more people will read it.</li> <li>Take an interest in all of the associated literary crafts, like copyediting, photography, and design, and make nice with people who come forth to help you.</li> <li>Think of copyeditors as champions of standards, invaluable test readers, your last line of defense.</li> <li>Learn to meet your deadlines to give others time to do their jobs. Even if you lack the authority to convene conversations, encourage early planning that includes all key players.</li> </ul> <h4>Tool 47: Recruit your own support group. Create a corps of helpers for feedback.</h4> <ul> <li>Work on developing the support system you need and deserve, including a cheerleader, an disinterested reader willing to answer your questions, expert helper who matches your topic, and a coach who helps you figure out what needs work.</li> </ul> <h4>Tool 48: Limit self-criticism in early drafts. Turn it loose during revision.</h4> <ul> <li>To become a fluent writer, one must silence the internal critic early until enough work has been done to warrant evaluation and revision.</li> <li>Pretend to write for a friend pleading, "Tell me more. Tell me all you can. I want to understand more about everything you feel and know and all the changes inside and out of you. Let more come out."</li> <li>But during revision, the self-conscious application of all writing advice will turn you to stone if you try to do it too early, or if you misapply it as orthodoxy.</li> </ul> <h4>Tool 49: Learn from your critics. Tolerate even unreasonable criticism.</h4> <ul> <li>Don't fall into the trap of arguing about matters of taste or defending your work against negative criticism. Explain to your critic what you were trying to do, thereby transforming arguments into conversations.</li> <li>Even when an attack is personal, in your mind deflect it back onto the work.</li> </ul> <h4>Tool 50: Own the tools of your craft. Build a writing workbench to store your tools.</h4> <ul> <li>Sniff around for news, then explore ideas, collect evidence, find a focus, select the best material, recognize an order, write a draft, and finally revise and clarify.</li> <li>The focus of a story can be expressed in a title, a first sentence, a summary paragraph, a theme statement, a thesis, a question the story will answer for the reader, one perfect word.</li> <li>Don't dump all your research into a story or essay. Use a sharp focus to cut tempting material that does not contribute to the central meaning of the work.</li> <li>Such a blueprint for writing not only demystifies the process, but will help you diagnose problems, account for your strengths and weaknesses, and build a critical vocabulary for talking about your craft.</li> </ul> https://tedneward.github.io/Research/writing/the-book-on-writing/index.html Mon, 13 Apr 2026 14:04:56 +0000 writing/the-book-on-writing/index.html <p><em>(by Paula LaRocque)</em></p> <h3>Section One: Writing Mechanics</h3> <ul> <li>Accuracy aside, simplicity, clarity, and brevity are the most important criteria for all writing. They form the bedrock of all good communication, and have for time immemorial.</li> </ul> <h4>Chapter 1: Keep Sentences Short, Varied, and to One Main Idea</h4> <ul> <li>Vary sentence length to avoid tedium, but a safe average is probably around 20 words, with longish sentences reaching around 25.</li> <li>Precede and follow a long or "difficult" sentence with short and crisp sentences. This gives the reader rest and adds variety like in natural speech.</li> <li>The average sentence length, grade level, and Flesch Reading Ease score of your writing could be the difference between readability and unreadability.</li> <li>Clarifying a subject-verb-object relationship is one way to rewrite a badly written passage.</li> <li>For clear and readable lists, get the subject and verb out of the way early, and keep list items parallel, where the first word of each list item begins with the same part of speech.</li> <li>A list fails when it begins a paragraph, hanging in a vacuum. Preface it with a sentence or fragment that sets it up.</li> <li>Keep to one idea per sentence. You can use simple main idea-subordinate-idea sentences, as long as both ideas together don't result in an unmanageable sentence.</li> <li>In technical writing, there's probably no reason to stretch sentence length beyond what bulleted lists can handle.</li> <li>Some long sentences that work well introduce a single idea first and then only expand upon it.</li> </ul> <h4>Chapter 2: Avoid Pretensions, Gobbledygook, and Euphemisms</h4> <ul> <li>Most educated Americans prefer to read at or below the 10th grade level, and an appropriate Flesch Reading Ease Index score for most writing is 60 or 70.</li> <li>There's nothing intelligent about pretentious and abstract writing. One hallmark of intellect is the ability to make the complex easy to understand; anyone can be unclear.</li> <li>The wasted time and effort as well as the cost of mistakes and misunderstanding make fuzzy writing an expensive habit wherever it flourishes.</li> <li>Euphamisms at best amuse and at worst alienate. They don't soften language, but make the reader suspicious. Simple words seem more sincere and soften best.</li> <li>A long word is the right word if it's the best word.</li> <li>Say what something is not is misleading; say what it is.</li> <li>Stop trying to impress and try instead to communicate, thereby using concrete and not abstract phrasing, and disabusing ourselves of the notion that big words sound more intelligent, more professional, and more serious.</li> </ul> <h4>Chapter 3: Change Long and Difficult Words to Short and Simple Words</h4> <ul> <li>Many of our most ancient words are one-syllable utterances. Such words tend to be concrete and emotive.</li> <li>The clearest, most stirring writers and speakers depend upon plain, short words. Such words usually don't go to war with each other; they have a nice mesh.</li> <li>The clarity that comes with honest, basic expression benefits all writing. Utter simplicity offers the bedrock of authenticity.</li> <li>The more words we know, the surer and freer we are to choose the plainest, simplest, and right words.</li> <li>Use complex words when they're the best choice for the context. But whenever we have a choice, we choose the shortest words.</li> </ul> <h4>Chapter 4: Be Wary of Jargon, Fad, and Cliché</h4> <ul> <li>Good jargon is a specialized term for a similarly specialized audience. The challenge for the communicator is translating jargon into plain English for a lay audience.</li> <li>Good jargon sets forth the complex in economical, albeit specialized, language. Bad jargon sets forth the simple in bloated language.</li> <li>Wrapped in jargon, inanities can actually sound smart, which is always hurtful to good communication and a sham.</li> <li>The problem with language fad and cliché is such expression grows flat, predictable, and dull over time. Mimicry is the antithesis of freshness and originality.</li> </ul> <h4>Chapter 5: Use the Right Word</h4> <ul> <li>How we use words matters if only because we may be judged ignorant (by some) for what they consider a misuse.</li> <li>Avoid <em>alright</em> and <em>alot</em>, and use <em>all right</em> and <em>a lot</em> instead. <em>Awhile</em> is an adverb, while <em>a while</em> is a noun.</li> <li>"The whole <em>comprises</em> the parts." Avoid <em>comprise of</em>; instead use <em>consist of</em>, like "The whole <em>consists of</em> the parts." But "The parts <em>constitute</em> the whole."</li> <li><em>Forthcoming</em> means available or ready when needed, while <em>forthright</em> means candid, direct, and straightforward.</li> <li><em>Prerequisite</em> is a requirement or precondition, while <em>perquisite</em> is a benefit beyond one's salary, or perk.</li> <li><em>Prescribe</em> means to order, direct, or mandate, while <em>proscribe</em> means to prohibit.</li> <li><em>Refute</em> means to disprove with conclusive evidence, while <em>rebut</em> means to deny or dispute, but doesn't imply any evidence.</li> </ul> <h4>Chapter 6: Avoid Beginning With Long Dependent Phrases</h4> <ul> <li>When a sentence begins with a preposition, verb, verbal, or with certain conjunctions and adverbs, it begins with a dependent phrase rather than the subject, thereby delaying its point.</li> <li>Such sentences aren't caught because no one reads the "backing-in" phrases; our eyes skip ahead to the subject.</li> <li>Backing into the first sentence of a piece is not attractive, but doing so later for transition and variety in sentence structure is sensible as long as the resulting sentence is clear.</li> <li>To fix a backed-into beginning, simply start the sentence with the subject.</li> <li>Especially avoid beginning a structure with a long, unanchored list. Instead, place the list after the subject, clarifying their relationship.</li> <li>The worst fault of backing-in sentences is that they hide the subject. Readers learn some corollary thing about the subject before they even know what that subject is.</li> <li>If a sentence backs-in to something or someone unknown, the reader is bewildered instead of interested.</li> <li>Only back-in when we've considered the alternatives, we know what we're doing and why, we have specific stylistic purposes that the backing-in structure satisfies.</li> </ul> <h4>Chapter 7: Prefer Active Verbs and the Active Voice</h4> <ul> <li>Just because a sentence has an auxiliary verb or a "be" verb does not mean that it is passive.</li> <li>The passive voice may be more effective than the active voice when the actor is irrelevant or gets in the way of the main point.</li> <li>Sometimes replacing a handful of weaker words with a single, strong verb leads to the active voice.</li> <li>Writing in active voice is clearer, more vigorous, and shorter than writing in the passive voice.</li> </ul> <h4>Chapter 8: Cut Wordiness</h4> <ul> <li>While a conversational style contributes to good writing, it is also wordy.</li> <li>Cutting prepositional phrases tightens writing. For example, replace <em>in regard to</em> with <em>about</em>, <em>in the event that</em> with <em>if</em>, <em>were in agreement</em> with <em>agreed</em>, and so on.</li> <li>Sentences that begin with <em>it</em> and <em>there</em> constructions are often wordy, and these constructions can be omitted.</li> <li>Imprecise or unnecessary adverbs and qualifiers also can lead to wordiness, such as using <em>moved quickly</em> instead of <em>rushed</em> or <em>hurried</em>.</li> <li>Phrases like "many," "most," and "what many described as" do not add specificity and can be omitted. Likewise, omit words that specify the obvious.</li> <li>Pointless verb strings and unnecessary passives, or any handful of words instead of one active verb, can also lead to wordiness.</li> </ul> <h4>Chapter 9: Avoid Vague Qualifiers</h4> <ul> <li>The word <em>very</em> and its siblings like <em>extremely</em>, <em>totally</em>, <em>completely</em>, <em>entirely</em>, <em>really</em>, <em>quite</em>, <em>rather</em>, <em>somewhat</em>, and <em>fairly</em> help us get closer to our meaning when we can't think of the perfect word.</li> <li>If a word with a vague qualifier is really just fine standing alone, then the qualifier should be removed. The word left standing alone becomes stronger and more assertive.</li> <li>Words like <em>dead</em>, <em>alive</em>, and <em>unique</em> have no degrees of meaning, like <em>pregnant</em>, and should not be qualified.</li> <li>Sometimes intensifiers such as very are simply inflationary, or a way of exclaiming rather than explaining. Prefer the latter, adding vital and concrete information.</li> <li>A good test for whether we have the right word is whether it can stand alone. If it can't, then find a replacement.</li> </ul> <h4>Chapter 10: Prune Prepositions</h4> <ul> <li>A sentence generally can bear three prepositional phrases, but it breaks down rapidly if more are added after that, yielding an annoying singsong.</li> <li>Some prepositional phrases can be rendered adjective-plus-noun. For example, <em>members of the faculty</em> becomes <em>faculty members</em>.</li> <li>If trimming prepositions sacrifices incidental information, it can easily be placed in subsequent sentences.</li> <li>Even when excessive prepositional phrases don’t damage clarity, they damage flow.</li> </ul> <h4>Chapter 11: Limit Number and Symbol</h4> <ul> <li>Limit to three the numbers in a sentence if the reader must compare, contrast, or calculate with those numbers.</li> <li>Numbers are even more confusing when they have different forms like percentages, fractions, written out, numerals, etc.</li> <li>Symbols like dollar signs, decimals, percentage symbols, acronyms, or abbreviations are also visually uninviting.</li> <li>Graphic explanation like bulleted lists, tabulated material, charts, and white space is both clearer and more attractive than prose with numbers and symbols.</li> <li>Two exceptions to the three-number guideline include dates, and when each number in a consecutive run has the same form and identifies the same thing.</li> </ul> <h4>Chapter 12: Get Right to the Point. And Stay There.</h4> <ul> <li>If a story is inherently dramatic, tell it simply and swiftly. A florid style or rhetorical devices such as repetition can seem melodramatic, and may even trivialize the tragic or poignant.</li> <li>An opening anecdote should be short and pertinent. It should reveal something important about the story's theme and thus open a way in to the story, instead of getting in its way.</li> <li>Be wary of frustrating the reader with unanswered questions, instead of creating in him or her a healthy sense of curiosity.</li> <li>Failure to get right to the point and stay there is a chief cause of reader annoyance, whether from overwriting and wordiness, an ill-advised anecdote, or a dithering, vague style.</li> </ul> <h3>Section Two: Storytelling Devices</h3> <h4>Chapter 20: Write Fast, Edit Slow</h4> <ul> <li>We cannot forgive slow, because fast is interesting and slow is dull. Slow makes us wait for the <em>first</em> shoe to drop.</li> <li>Reading aloud forces us to listen, helping us locate the slow parts that interfere with the reader's progress, or stifle a speedy and seamless flow of writing.</li> <li>Such speedbumps include errors in content or form, distractions like awkward phrasing, the wrong word, and dense, wordy, fuzzy, repetitive, tentative, or extraneous passages.</li> <li>Proper research, finding a focus or theme, dividing your work into pieces, devising a beginning, middle, and end, and making an outline can help you write fast-moving prose.</li> <li>Most serious pace problems come from lack of focus and direction. Write a sentence that captures the essence of a section or chapter, and use it as a roadmap.</li> <li>One of the best way to "remove all things that are not the story" is to keep them out of the story in the first place. Fast writing helps keep extraneous material out.</li> <li>Don't edit as you write, or you'll lose momentum. You can edit slowly, because writing problems won't disappear as inspiration and spontaneity do.</li> </ul> <h4>Chapter 21: Speedbumps</h4> <ul> <li>Mistakes, failing to get to the point or present material logically, excessive length, obtrusive or tangential material, bad diction, and packing too much into sentences can lead to slow reading.</li> <li>The worst place to find speedbumps is at the beginning of the trip. Slow starts lose many readers, and clumsily executed anecdotal beginnings are among the worst offenders.</li> <li>Multiple anecdotes at the beginning is confusing because the readers is left guessing what the story is about. Get them over with quickly or have them follow the main point.</li> <li>Also mind not having a point, or failing to express that point briefly and clearly, or interrupting the work on the way to the point.</li> <li>Subjects and verbs should be close together, as should verbs and objects.</li> <li>Who, what, when, where, and why can wait if they’re going to get in the way of the message. Such information is not as important as writing well.</li> <li>Writing entails organizing into families of related thought, but one-sentence-per-paragraph writing orphans every sentence. And each has the same weight, so the thesis is not clear.</li> <li>Parenthetical or bracketed material is a major intrusion when striving for a conversational, story-telling style.</li> </ul> <h4>Chapter 22: Logic and Speedy Reading</h4> <ul> <li>The writer and reader must proceed in lockstep. If the writer is slower than the reader, the reader gets impatient. If the writer gets ahead, readers may get lost.</li> <li>Before embroidery comes the fabric, and the fabric of good writing is the tightly woven stuff of accuracy, clarity, brevity, precision, and logic.</li> <li>Illogic and loose connections can drive readers away and damage the writer's credibility.</li> <li>The essence of illogic is the non sequitur, which is an inference or conclusion that does not follow from the premise, such as in an if-then sentence.</li> <li>Non sequiturs also arise from a comment following an unrelated comment, and even conjoining the two with <em>but</em>.</li> <li>Trouble also arises from illogical word pairs, mishandled figurative expressions, and using arguably to introduce sweeping, illogical, and insupportable claims.</li> </ul> <h3>Section Three: A Handbook</h3> <h4>Chapter 23: A Brief (But Not Necessarily Easy) Quiz</h4> <ul> <li>Use adjectives rather than adverbs with sense verbs or with linking verbs such as forms of the verb <em>to be</em>, <em>seem</em>, <em>appear</em>, <em>become</em>, etc. For example, <em>the food smells bad</em>, but <em>she swam badly</em>. But <em>bad</em> describes condition or passive states, such as <em>he felt bad</em>, where <em>bad</em> modifies the subject <em>he</em>.</li> <li>Only place a comma between adjectives if it could be replaced by <em>and</em>. For example, <em>a beautiful baby girl</em>, and <em>a large, ugly dog</em> are both correct.</li> <li>Use possessive pronouns before gerunds, which are words that end in "ing" but act as nouns.</li> <li><em>Like</em> is not a conjunction; use it as a conjunction to compare nouns and pronouns. Use <em>as</em>, <em>as if</em>, or <em>as though</em> to introduce clauses, or a group of words containing both subject and verb.</li> <li>When using the phrase "one of those [nouns] who," the subject of the following verb is <em>who</em> and not <em>one</em>, and so the verb should adopt the plural and not the singular form.</li> <li>The word <em>as</em> should not follow <em>equally</em>, or verbs such as <em>named</em>, <em>called</em>, <em>elected</em>, etc.</li> <li>Subjective pronouns are <em>I</em>, <em>he</em>, <em>she</em>, <em>we</em>, <em>they</em>, or <em>who</em>. Objective pronouns are <em>me</em>, <em>him</em>, <em>her</em>, <em>us</em>, <em>them</em>, or <em>whom</em>.</li> <li><em>Myself</em> is a "self" pronoun that differs because it is neither a subject nor object, but a reflexive like <em>I hurt myself</em>, or an intensifier like <em>I myself am staying at home</em>.</li> <li>Removing other people from the sentence and letting the pronoun stand alone, or replacing the pronoun with another one, quickly reveals whether it is a subject or object.</li> <li>If a pronoun is preceded by <em>than</em>, and placing a verb after the pronoun makes grammatical sense, then choose the subjective form.</li> </ul> <h4>Chapter 24: Dispelling the Myths</h4> <ul> <li>The infinitive is <em>to</em> plus a verb. The split infinitive separates the two, typically with an adverb, such as <em>to boldly go</em>. They are often unattractive, but they are not wrong.</li> <li>Ending sentences with a preposition is fine, but omit the preposition if it is a gross redundancy, such as with <em>Where is it at?</em></li> <li>Avoid ambiguous contractions like <em>he'd</em>, <em>we'd</em>, and <em>I'd</em>. The first can mean either <em>he would</em>, or <em>he had</em>. Avoiding contractions also makes text more emphatic or stately.</li> <li>Omitting the serial comma may cause ambiguities, whereas including it never will.</li> <li>Regarding <em>none</em>, if it clearly means <em>no one</em> or <em>not one</em>, it's singular. If the stress is on <em>not any</em>, or is on more than one, or if the following noun cannot be construed as singular, treat <em>none</em> as plural.</li> <li>If each person in a couple is acting individually, then <em>couple</em> is plural. But if two people are acting as one, then it is singular.</li> <li>Collective nouns like <em>company</em>, <em>team</em>, and <em>committee</em> refer to an <em>it</em>, not a <em>they</em>, which can be awkward. Adding a plural noun to the collective one, like <em>faculty members</em>, can remedy this.</li> <li>Retain <em>that</em> with sentences containing both attribution and time, and following words such as <em>announce</em>, <em>believe</em>, <em>thought</em>, <em>reveal</em>, <em>declare</em>, <em>understand</em>, <em>assert</em>, <em>assume</em>, <em>allege</em>, and so on.</li> <li>Use <em>a</em> before a consonant sound and <em>an</em> before a vowel sound. And so <em>an NCAA policy</em>, <em>a eulogy</em>, <em>an herb</em>, and <em>a herbicide</em> are all correct.</li> </ul> <h4>Chapter 25: Style Guide</h4> <ul> <li>Use <em>a.m.</em> and <em>p.m.</em> instead of <em>AM</em> and <em>PM</em>.</li> <li><em>Anno</em> and <em>annus</em> mean <em>year</em>, so use <em>first anniversary</em> instead of <em>one-year anniversary</em> and <em>25th anniversary</em> instead of <em>25-year anniversary</em>. <em>Three-week anniversary</em> and <em>one-month anniversary</em> are incorrect.</li> <li>It's <em>backward</em>, not <em>backwards</em>. Likewise, it's <em>toward</em>, not <em>towards</em>.</li> <li>It's the <em>better</em> of two, and the <em>best</em> of three or more.</li> <li>Countries have <em>citizens</em>, while cities and states have <em>residents</em>.</li> <li>Avoid the colon after a <em>be</em> verb, like <em>The three objectives of the new plan are:</em></li> <li>Place a comma before a conjunction midsentence if what follows the conjunction could stand as a complete sentence.</li> <li>If a city and state, or a month, day, and year appear midsentence, then place a comma both before and after the state or the year. But no comma is necessary between month and year when the day is absent.</li> <li><em>et al</em>, <em>e.g.</em>, and <em>i.e.</em> should be lowercase and italicized.</li> <li>Use commas both before and after <em>etc.</em> when it appears midsentence.</li> <li>Use <em>hang on to</em> or <em>hold on to</em>, but not <em>hang onto</em> or <em>hold onto</em>.</li> <li><em>Holdup</em> is a noun, while <em>hold up</em> is a verb.</li> <li><em>Likely</em> is not an adverb and should not be used as a substitute for <em>probably</em>. It's an adjective, parallel to <em>probable</em> rather than <em>probably</em>.</li> <li><em>Makeup</em> is a noun, <em>make up</em> is a verb, and <em>make-up</em> is an adjective.</li> <li><em>Mini</em> generally has no hyphen. <em>mid</em> generally has no hyphen unless followed by a capital letter or numeral. <em>Non</em> is generally not hyphenated except before a proper noun or to avoid awkward constructions, like <em>nonnuclear</em>.</li> <li>Write out the numerals one through nine, while use numerals for 10 or more. Avoid numerals at sentence beginnings.</li> <li><em>One of the only</em> is an illiteracy; it should be <em>one of the few</em>.</li> <li>Use <em>percent</em> in text, and the symbol <em>%</em> for charts, graphs, tabulated material, lists, and the like.</li> <li>Write out state names in text, and reserve postal abbreviations for lists, charts, or graphic material.</li> <li>Don't mark deletions at sentence beginning; just start the quotation there. An ellipsis at sentence end is four spaced dots, the last being the period.</li> <li>Avoid the constructions <em>the reason is because</em> and <em>the reason why</em>. The text within or following them can stand alone.</li> <li>Hyphenate <em>re-create</em> when meaning to create again rather than fun or leisure activity.</li> <li><em>Round trip</em> is a noun, while <em>round-trip</em> is an adjective.</li> <li>If you are including <em>sic.</em> or many brackets in a quote, it should be paraphrased or made a partial quote.</li> <li>Drop <em>single</em> from <em>single most</em>, <em>single best</em>, and <em>single biggest</em>.</li> <li>Replace <em>the past decade</em> with <em>the last decade</em>.</li> <li><em>These</em> and <em>this</em> point forward, while <em>those</em> and <em>that</em> point back.</li> <li>Don't use a hyphen between times; instead, use <em>from/to</em>, or <em>between/and</em>.</li> <li>Capitalize the first and last words of a title and all the principal words between. Lowercase articles, <em>to</em>, coordinating conjunctions, and prepositions fewer than four letters. Capitalize longer prepositions like <em>between</em>, <em>toward</em>, <em>beyond</em>, <em>among</em>, <em>with</em>, and <em>from</em>.</li> <li>The initial article <em>A</em>, <em>An</em>, or <em>The</em> in titles can be omitted if the article follows a possessive noun or pronoun, or if the article follows an adjective or another article.</li> <li>Avoid using <em>verbal agreement</em> for <em>oral agreement</em>, as the preferred meaning of <em>verbal</em> is words, both written and spoken.</li> <li><em>Wait for</em>, and not <em>wait on</em>, unless one is waiting on tables.</li> </ul> https://tedneward.github.io/Research/vms/wasm/composable-runtime/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/composable-runtime/index.html <p><a href="https://crates.io/crates/composable-runtime">Website</a></p> https://tedneward.github.io/Research/vms/wasm/happy-new-moon-with-report/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/happy-new-moon-with-report/index.html <p><a href="https://github.com/fishjd/HappyNewMoonWithReport">Source</a></p> <p>Implemented in Java. Interpreted. Java/JVM interop.</p> <p><strong>Non-MVP features supported</strong>: Sign Extensions</p> https://tedneward.github.io/Research/vms/wasm/motor/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/motor/index.html <p><a href="https://github.com/penberg/motor">Source</a></p> <p>Implemented in Rust using Dynasm. JIT compiled.</p> https://tedneward.github.io/Research/vms/wasm/twvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/twvm/index.html <p><a href="https://github.com/Becavalier/TWVM">Source</a></p> <p>Implemented in C++. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wakit/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wakit/index.html <p><a href="https://github.com/akkyie/WAKit">Source</a></p> <p>Implemented in Swift. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wasmboy/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmboy/index.html <p><a href="https://wasmboy.app/">Website</a> | <a href="https://github.com/torch2424/wasmBoy">Source</a></p> https://tedneward.github.io/Research/vms/wasm/wasmo/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmo/index.html <ul> <li> <p><strong>Languages written in</strong></p> <table> <tbody> <tr> <td>Rust</td> </tr> </tbody> </table> </li> <li> <p><strong>Compiler framework</strong></p> <table> <tbody> <tr> <td>LLVM</td> </tr> </tbody> </table> </li> <li> <p><strong>Compilation / Execution modes</strong></p> <table> <tbody> <tr> <td>JIT</td> </tr> </tbody> </table> </li> <li> <p><strong>Interoperability with other languages</strong></p> <ul> <li><code>N/A</code></li> </ul> </li> <li> <p><strong>Non-MVP features supported</strong></p> <ul> <li><code>N/A</code></li> </ul> </li> <li> <p><strong>Host APIs supported</strong></p> <ul> <li><code>N/A</code></li> </ul> </li> <li> <p><strong>Non-web standards</strong></p> <ul> <li><code>N/A</code></li> </ul> </li> <li> <p><strong>Used by</strong></p> <ul> <li><code>N/A</code></li> </ul> </li> <li> <p><strong>Platforms supported</strong></p> <table> <tbody> <tr> <td>macOS</td> </tr> </tbody> </table> </li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmvm2/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmvm2/index.html <p><a href="https://github.com/kogai/wasvm">Source</a></p> <p>Implemented in Rust using Life. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wizard/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wizard/index.html <p><a href="https://github.com/titzer/wizard-engine">Source</a> |</p> <p>The Wizard Engine is a lightweight WebAssembly virtual machine designed for teaching and research. Its implementation is designed to be flexible and easy to grasp, ideal for experimentation and modification. Built with the future in mind, it is written in a fast and lightweight safe, garbage-collected programming language, Virgil.</p> <p>Implemented in Virgil. Interpreted and JIT compiled.</p> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Bulk Memory Operations</li> <li>Import/Export of Mutable Globals</li> <li>Sign-extension operators</li> <li>Multi-Value Returns</li> <li>Name Section</li> <li>Non-trapping float-to-int conversions</li> <li>Reference Types</li> <li>SIMD</li> <li>Tail-call</li> <li>GC</li> <li>Exception Handling</li> </ul> <p><strong>Host APIs supported</strong>: Wave, WASI</p> <p><strong>Non-web standards</strong>: WASI</p> https://tedneward.github.io/Research/vms/zetavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/zetavm/index.html <p><a href="https://pointersgonewild.com/category/zeta/">Website</a> | <a href="https://github.com/zetavm/zetavm">Source</a></p> <p>(Claims under development, but no commits for 3 years.)</p> <p>ZetaVM is a Virtual machine and JIT compiler for dynamic programming languages. It implements a basic core runtime environment on top of which programming dynamic languages can be implemented with relatively little effort.</p> <p>Features of the VM include:</p> <ul> <li>Built-in support for dynamic typing</li> <li>Garbage collection</li> <li>JIT compilation</li> <li>Dynamically growable objects (JS-like)</li> <li>Dynamically-typed arrays (JS/Python-like)</li> <li>Integer and floating-point arithmetic</li> <li>Immutable UTF-8 strings</li> <li>Text-based code and data storage format (JSON-like)</li> <li>First-class stack-based bytecode (code is data)</li> <li>Built-in graphical and audio libraries</li> <li>Coming soon: built-in package manager</li> </ul> <p>Zeta image files (.zim) are JSON-like, human-readable text files containing objects, data and bytecodes to be executed by ZetaVM. They are intended to serve as a compilation target, and may contain executable programs, or libraries/packages.</p> https://tedneward.github.io/Research/vms/wasm/extism/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/extism/index.html <p><a href="https://github.com/extism/extism">Source</a></p> <p>Implemented in Rust. JIT compiled.</p> <p>SDKs to execute WebAssembly in:<br> - <a href="https://extism.org/docs/integrate-into-your-codebase/browser-runtime-sdk/"><code>Browser</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/c-host-sdk/"><code>C</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/cpp-host-sdk/"><code>C++</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/dotnet-host-sdk"><code>.NET</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/elixir-host-sdk/"><code>Elixir</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/go-host-sdk/"><code>Go</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/haskell-host-sdk/"><code>Haskell</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/java-host-sdk"><code>Java</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/node-host-sdk/"><code>Node</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/ocaml-host-sdk/"><code>OCaml</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/php-host-sdk/"><code>PHP</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/python-host-sdk/"><code>Python</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/ruby-host-sdk/"><code>Ruby</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/rust-host-sdk/"><code>Rust</code></a><br> - <a href="https://extism.org/docs/integrate-into-your-codebase/zig-host-sdk"><code>Zig</code></a></p> <p>PDKs to create WebAssembly plug-ins in:<br> - <a href="https://github.com/extism/rust-pdk"><code>Rust</code></a><br> - <a href="https://extism.org/docs/write-a-plugin/js-pdk"><code>JavaScript</code></a><br> - <a href="https://github.com/extism/go-pdk"><code>Go</code></a><br> - <a href="https://extism.org/docs/write-a-plugin/haskell-pdk"><code>Haskell</code></a><br> - <a href="https://github.com/extism/assemblyscript-pdk"><code>AssemblyScript</code></a><br> - <a href="https://github.com/extism/c-pdk"><code>C</code></a><br> - <a href="https://github.com/extism/c-pdk"><code>C++</code></a><br> - <a href="https://extism.org/docs/write-a-plugin/zig-pdk"><code>Zig</code></a></p> <p><strong>Host APIs supported</strong><br> - module loading &amp; linking via <a href="https://extism.org/docs/concepts/manifest"><code>Manifest</code></a><br> - WASI<br> - non-WASI <code>http_request</code></p> https://tedneward.github.io/Research/vms/wasm/innative/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/innative/index.html <p><a href="https://github.com/innative-sdk/innative">Source</a></p> <p>Implemented in C++ using LLVM. AOT and JIT compiled.</p> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Threads</li> <li>Multiple Result and Block Parameters</li> <li>Name Section</li> <li>Bulk Memory Operations</li> <li>Sign Extension Instructions</li> <li>Non-trapping Conversion Instructions</li> </ul> <p><strong>Host APIs supported</strong>: Custom</p> https://tedneward.github.io/Research/vms/wasm/pywasm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/pywasm/index.html <p><a href="https://github.com/ethereum/py-wasm">Source</a></p> <p>Implemented in Python. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/vmir/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/vmir/index.html <p><a href="https://github.com/andoma/vmir">Source</a></p> <p>Implemented in C using LLVM. Interpreted and JIT compiled.</p> https://tedneward.github.io/Research/vms/wasm/wamr/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wamr/index.html <p><a href="https://github.com/bytecodealliance/wasm-micro-runtime">Source</a></p> <p>Implemented in C. Interpreted, AOT, and JIT compiled.</p> <p>It includes a few parts as below:</p> <ul> <li> <p>The "iwasm" VM core to run WASM applications, supporting interpreter mode, AOT mode (Ahead-of-Time compilation) and JIT modes (Just-in-Time compilation, LLVM JIT and Fast JIT are supported)</p> </li> <li> <p>The "wamrc" AOT compiler to compile WASM file into AOT file for best performance and smaller runtime footprint, which is run by "iwasm" VM Core</p> </li> <li> <p>The application framework and the supporting APIs for the WASM applications</p> </li> <li> <p>The dynamic management of the WASM applications</p> </li> </ul> <p><strong>Non-MVP features supported</strong></p> <ul> <li><a href="https://github.com/WebAssembly/nontrapping-float-to-int-conversions">Non-trapping float-to-int conversions</a></li> <li><a href="https://github.com/WebAssembly/sign-extension-ops">Sign-extension operators</a></li> <li><a href="https://github.com/WebAssembly/bulk-memory-operations">Bulk memory operations</a></li> <li><a href="https://github.com/WebAssembly/threads/blob/main/proposals/threads/Overview.md#shared-linear-memory">Shared memory</a></li> <li><a href="https://github.com/WebAssembly/multi-value">Multi-value</a></li> <li><a href="https://github.com/WebAssembly/tail-call">Tail-call</a></li> </ul> <p><strong>Host APIs supported</strong></p> <ul> <li><a href="https://github.com/WebAssembly/wasm-c-api">wasm-c-api</a></li> </ul> <p><strong>Non-web standards</strong></p> <ul> <li>[x] WASI</li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmedge/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmedge/index.html <p><a href="https://wasmedge.org/">Website</a> | <a href="https://github.com/WasmEdge/WasmEdge">Source</a></p> <p>Implemented in C++ using LLVM. Interpreted and AOT compiled.</p> <p><strong>Interoperability with other languages</strong>:</p> <ul> <li>Solidity</li> <li>Rust</li> <li>C/C++</li> <li>Go/TinyGo</li> <li>JavaScript</li> <li>Python</li> <li>Grain</li> <li>Swift</li> <li>Zig</li> <li>Ruby</li> </ul> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Bulk Memory Operations</li> <li>SIMD</li> <li>Multi-Memory</li> <li>Multi-Value</li> <li>Reference Types</li> <li>Sign Extension Instructions</li> <li>Non-Trapping Float-to-Int Conversions</li> <li>Mutable Global</li> </ul> <p><strong>Host APIs supported</strong>:</p> <ul> <li>WASI</li> <li>Networking Socket</li> <li>TensorFlow</li> <li>Command Interface</li> <li>Image Processing</li> </ul> <p><strong>Non-web standards</strong>:</p> <ul> <li>WASI</li> <li>Gas Metering</li> <li>Ethereum Environment Interface</li> <li>Oasis</li> <li>Substrate</li> </ul> <p><strong>Used by</strong>:</p> <ul> <li><a href="https://blog.suborbital.dev/suborbital-wasmedge">Suborbital</a></li> <li><a href="https://github.com/containers/crun/pull/774">crun</a></li> <li><a href="https://github.com/superedge/superedge/pull/335">SuperEdge</a></li> <li><a href="https://github.com/openyurtio/openyurt.io/pull/85">OpenYurt</a></li> <li><a href="https://www.infoq.com/articles/webassembly-dapr-wasmedge/">Dapr</a></li> <li><a href="https://github.com/yomorun/yomo-wasmedge-tensorflow">Yomo</a></li> <li><a href="https://medium.com/oasis-protocol-project/the-oasis-eth-paratime-is-live-on-mainnet-33d8713ec870">Oasis ETH ParaTime</a></li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmrt/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmrt/index.html <p><a href="https://github.com/rhitchcock/wasmrt">Source</a></p> <p>Implemented in C++. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wavm/index.html <p><a href="https://github.com/WAVM/WAVM">Github</a> | <a href="https://wavm.github.io/">Website</a></p> <p>Implemented in C++ and Python using LLVM. JIT compiled.</p> <p><strong>Host APIs supported</strong>: WASI, Emscripten, WAVIX</p> <p><strong>Non-web standards</strong>: WASI</p> <p>From the Github page:</p> <p>WAVM fully supports WebAssembly 1.0, plus many proposed extensions to it:</p> <ul> <li>WASI</li> <li>128-bit SIMD</li> <li>Threads</li> <li>Reference types</li> <li>Multiple results and block parameters</li> <li>Bulk memory operations</li> <li>Non-trapping float-to-int conversions</li> <li>Sign-extension instructions</li> <li>Exception handling</li> <li>Extended name section</li> <li>Multiple memories</li> </ul> <p>WAVM uses LLVM to compile WebAssembly code to machine code with close to native performance. It can even beat native performance in some cases, thanks to the ability to generate machine code tuned for the exact CPU that is running the code. WAVM also leverages virtual memory and signal handlers to execute WebAssembly's bounds-checked memory accesses at the same cost as a native, unchecked memory access.</p> https://tedneward.github.io/Research/vms/webvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/webvm/index.html <p><a href="https://webvm.io/">Website</a> | <a href="https://github.com/leaningtech/webvm">Source</a></p> <ul> <li><a href="https://labs.leaningtech.com/blog/webvm-server-less-x86-virtual-machines-in-the-browser">https://labs.leaningtech.com/blog/webvm-server-less-x86-virtual-machines-in-the-browser</a></li> <li><a href="https://labs.leaningtech.com/blog/mini-webvm-your-linux-box-from-dockerfile-via-wasm">https://labs.leaningtech.com/blog/mini-webvm-your-linux-box-from-dockerfile-via-wasm</a></li> <li><a href="https://labs.leaningtech.com/blog/webvm-20">https://labs.leaningtech.com/blog/webvm-20</a></li> </ul> https://tedneward.github.io/Research/vms/zmachine/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/zmachine/index.html <p><a href="http://www.ifwiki.org/index.php/Z-machine_versions">Z-machine versions</a> | <a href="http://inform-fiction.org/zmachine/standards/">Specification</a> <a href="http://inform-fiction.org/zmachine/standards/index.html">Inform Standards</a> | <a href="https://en.wikipedia.org/wiki/Z-machine">Wikipedia</a></p> <p><a href="http://curiousdannii.github.io/if/zspec12.html">Z-Machine Standard 1.2 (draft)</a> | <a href="https://inform-fiction.org/zmachine/standards/z1point1/index.html">Z-Machine Standards Document v1.1</a> | <a href="http://www.wolldingwacht.de/if/z-spec.html">if-resources</a>: Collection of Z-Machine standard document, consisting of Z-Machine Standard Document 1.0, Z-machine Common Save-File Format Standard (Quetzal) v1.3b, IF Resource Collection Format (Blorb) v1.1, and proposal for a Z-Machine Standard v1.1, in one PDF</p> <p><a href="https://web.archive.org/web/20100807003406/http://xlisp.org/zil.pdf">Learning ZIL</a></p> <p><a href="http://xlisp.org/zip.pdf">Description of ZIP</a> (the Z-Language Interpreter Program, as per an Infocom internal document)</p> <p>Interpreters/VMs:<br> * Glulx – Similar to the Z-machine, but relieves several legacy limitations<br> * <a href="../languages/inform6">Inform</a> – A computer language that can produce Z-machine programs<br> * <a href="../gamedev/scummvm">SCUMM</a> – Script Creation Utility for Maniac Mansion by LucasArts, a graphical system similar to Z-machine<br> * <a href="../tads">TADS</a> – Like Glulx, made to address some of its limitations<br> * Motorola 68000 series – The base architecture used for virtual machines in Magnetic Scrolls' adventure games</p> <p><a href="https://foss.heptapod.net/zilf/zilf">ZILF</a>: A set of tools for working with ZIL, including a compiler, assembler, disassembler, and game library. | <a href="https://github.com/heasm66/ZILF-Reference-Guide">ZILF Reference Guide</a></p> <p><a href="https://www.facebook.com/groups/ZILcom/">Zork Implementation Language Facebook group</a></p> <hr> <h1><a href="https://www.linusakesson.net/software/zeugma/index.php">Zeugma</a></h1> <p>A modern, highly optimised, open-source Z-machine implementation for the 6502 processor family. It is designed around a virtual memory model, where pages are retrieved on demand from an external storage resource, 512 KB in size, which must be preloaded with the game file.</p> https://tedneward.github.io/Research/vms/wasm/chicory/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/chicory/index.html <p><a href="https://github.com/dylibso/chicory">Source</a></p> <p>Chicory is a JVM native WebAssembly runtime. It allows you to run WebAssembly programs with zero native dependencies or JNI. Chicory can run Wasm anywhere that the JVM can go. It is designed with simplicity and safety in mind.</p> https://tedneward.github.io/Research/vms/wasm/fdvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/fdvm/index.html <p><a href="https://github.com/funcdef/fdvm">Source</a></p> <p>Implemented in Javascript using NodeJS/V8. JIT compiled.</p> https://tedneward.github.io/Research/vms/wasm/life/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/life/index.html <p><a href="https://github.com/perlin-network/life">Source</a></p> <p>Implemented in Go. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/swam/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/swam/index.html <p><a href="https://github.com/satabin/swam">Source</a></p> <p>Implemented in Scala. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wac/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wac/index.html <p><a href="https://github.com/kanaka/wac">Source</a></p> <p>Implemented in C. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/warpy/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/warpy/index.html <p><a href="https://github.com/kanaka/warpy">Source</a></p> <p>Implemented in RPython. JIT-compiled.</p> https://tedneward.github.io/Research/vms/wasm/wasmer/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmer/index.html <p><a href="https://wasmer.io/">Website</a> | <a href="https://github.com/wasmerio">Source</a></p> <p>Implemented in C++ and Rust using Singlepass, Cranelift, and LLVM. JIT and AOT compiled.</p> <p><strong>Interoperability with other languages</strong>:</p> <ul> <li><a href="https://github.com/wasmerio/wasmer">Rust</a></li> <li>C</li> <li>C++</li> <li><a href="https://github.com/wasmerio/wasmer-python">Python</a></li> <li><a href="https://github.com/wasmerio/wasmer-go">Go</a></li> <li><a href="https://github.com/wasmerio/wasmer-php">PHP</a></li> <li>Java</li> <li><a href="https://github.com/wasmerio/wasmer-ruby">Ruby</a></li> <li>Postgres</li> <li>C#/.Net</li> <li>Elixir</li> <li>R</li> <li>D</li> <li>Swift</li> </ul> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Threads</li> <li>SIMD</li> <li>Multi Value returns</li> <li>Name Section</li> <li>Bulk Memory Operations</li> <li>Sign Extension Instructions</li> </ul> <p><strong>Host APIs supported</strong>:</p> <ul> <li>Emscripten</li> <li>WASI</li> </ul> <p><strong>Non-web standards</strong>:</p> <ul> <li>WASI</li> <li>wasm-c-api</li> </ul> <p><strong>Used by</strong></p> <ul> <li><a href="https://github.com/spacemeshos/svm">Spacemesh Virtual Machine</a> - A Spacemesh smart contracts vm</li> <li><a href="https://github.com/CosmWasm/cosmwasm">CosmWasm</a> - A Cosmos-compatible library for running wasm smart contracts</li> <li><a href="https://github.com/near/nearcore">NEAR Protocol</a> - The reference client for NEAR protocol</li> <li><a href="https://github.com/dprint/dprint">Dprint</a> - Pluggable and configurable code formatting platform</li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmtime/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmtime/index.html <p><a href="https://wasmtime.dev/">Website</a> | <a href="https://github.com/bytecodealliance/wasmtime">Source</a></p> <p>Implemented in Rust using Cranelift. JIT compiled. Interops with Python.</p> <p><strong>Non-MVP features supported</strong>: Interface types</p> <p><strong>Host APIs supported</strong>: WASI</p> <p><strong>Non-web standards</strong>: WASI, wasm-c-api</p> <p><strong>Used by</strong></p> <ul> <li><a href="https://github.com/paritytech/wasmi">Wasmi</a>: an efficient WebAssembly interpreter that is used by <a href="https://github.com/paritytech/substrate">Substrate</a>, a next-generation framework for blockchain innovation.</li> </ul> https://tedneward.github.io/Research/vms/wasm/wazero/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wazero/index.html <p><a href="https://wazero.io">Source</a></p> <p>Implemented in Go. Interpreted and JIT compiled.</p> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Bulk Memory Operations</li> <li>Import/Export of Mutable Globals</li> <li>Sign-extension operators</li> <li>Multi-Value Returns</li> <li>Name Section</li> <li>Non-trapping float-to-int conversions</li> <li>Reference Types</li> <li>SIMD</li> </ul> <p><strong>Host APIs supported</strong>:</p> <ul> <li>AssemblyScript</li> <li>WASI</li> </ul> <p><strong>Non-web standards</strong>:</p> <ul> <li>WASI</li> </ul> <p><strong>Used by</strong></p> <ul> <li><a href="https://github.com/dapr/dapr">dapr</a> - APIs that simplify microservice connectivity</li> <li><a href="https://github.com/aquasecurity/trivy">trivy</a> - vulnerability/misconfiguration/secret scanner for containers and other artifacts</li> <li><a href="https://github.com/wapc/wapc-go">wapc-go</a> - WebAssembly Host Runtime for waPC-compliant modules</li> </ul> https://tedneward.github.io/Research/vms/xvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/xvm/index.html <p><a href="https://github.com/xtclang/xvm">Source</a></p> https://tedneward.github.io/Research/writing/22-rules-of-storytelling/index.html Mon, 13 Apr 2026 14:04:56 +0000 writing/22-rules-of-storytelling/index.html <p><a href="https://42courses.medium.com/pixars-22-rules-of-storytelling-dc24162fe8ce">Website</a></p> <ol> <li> <p>You admire a character for trying more than for their successes.</p> </li> <li> <p>Keep in mind what’s interesting to you as an audience, not what’s fun as a writer. They can be very different.</p> </li> <li> <p>Trying for theme is important, however you won’t see what the story is about until you’re at the end of that story. Got it? Now rewrite.</p> </li> <li> <p>Once upon a time there was ____. Every day, ___. One day ____. Because of that, ______. Until finally ____.</p> </li> <li> <p>Simplify. Focus. Combine characters. Hop over detours. You’ll feel like you’re losing valuable stuff but it sets you free.</p> </li> <li> <p>What is your character good at or comfortable with? Throw the polar opposite at him. Challenge him. How does he deal with it?</p> </li> <li> <p>Come up with your ending before you figure out your middle. Seriously. Endings are hard. Get yours working up front.</p> </li> <li> <p>Finish your story. Let go even if it’s not perfect. In an ideal world, you have both, but move on. Do better next time.</p> </li> <li> <p>When you’re stuck, make a list of what wouldn’t happen next. More often than not, the material that gets you unstuck appears.</p> </li> <li> <p>Pull apart the stories you like. What you like in them is part of you. Recognise it before you use it.</p> </li> <li> <p>Why must you tell this story in particular? What’s the belief burning within you that your story feeds off? That’s the heart of it.</p> </li> <li> <p>Discount the 1st thing that comes to mind. And the 2nd, 3rd, 4th, 5th — get the obvious out of the way. Surprise yourself.</p> </li> <li> <p>Give your characters opinions. A character being passive or malleable is easy for you as a writer, but it’s poison to your audience.</p> </li> <li> <p>What’s the essence of your story? What’s the most economical way of telling of it? If you know that, you can build out there.</p> </li> <li> <p>If you were your character, in this situation, how would you feel? Honesty lends credibility to unbelievable situations.</p> </li> <li> <p>What are the stakes? Give us reason to root for the character. What happens if he doesn’t succeed? Stack the odds against him.</p> </li> <li> <p>No work is ever wasted. And if it’s not working, let go and move on — if it’s useful, it’ll show up again.</p> </li> <li> <p>You have to know yourself, and know the difference between doing your best &amp; being fussy. Story is testing, not refining.</p> </li> <li> <p>Coincidences that get characters into trouble are great. Coincidences that get them out of it is cheating.</p> </li> <li> <p>Exercise. Take the building blocks of a movie you dislike. How would you rearrange them into what you DO like?</p> </li> <li> <p>Identify with your situation/characters. Don’t write "cool." What would make YOU act that way?</p> </li> <li> <p>Putting it on paper only allows you to start fixing it. If a perfect idea stays in your head, you’ll never share it with anyone.</p> </li> </ol> https://tedneward.github.io/Research/vms/wasm/cmm-of-wasm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/cmm-of-wasm/index.html <p><a href="https://github.com/SimonJF/cmm_of_wasm">Source</a></p> <p>Implemented in OCaml. AOT compilation.</p> https://tedneward.github.io/Research/vms/wasm/fizzy/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/fizzy/index.html <p><a href="https://github.com/wasmx/fizzy">Source</a></p> <p>Implemented in C++. Interpreted.</p> <p>Supports WASI.</p> https://tedneward.github.io/Research/vms/wasm/lucet/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/lucet/index.html <p><a href="https://github.com/fastly/lucet">Source</a></p> <blockquote> <p>NOTE: Lucet is now in maintenance mode. Work has been moved to <a href="https://github.com/bytecodealliance/wasmtime/">wasmtime</a>.</p> </blockquote> <p>Implemented in Rust using Cranelift. AOT compiled.</p> <p><strong>Host APIs supported</strong>: WASI</p> <p><strong>Non-web standards</strong>: WASI</p> https://tedneward.github.io/Research/vms/wasm/teavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/teavm/index.html <p><a href="https://teavm.org/">Website</a> | <a href="https://github.com/konsoletyper/teavm">Source</a></p> <p>WebAssembly support is in experimental status. It may lack major features available in JavaScript backend. There's no documentation yet, and you should do many things by hands (like embedding generated wasm file into your page, importing JavaScript objects, etc).</p> <h2>Purpose</h2> <p>TeaVM is primarily a web development tool. It's not for taking your large existing codebase in Java or Kotlin and producing JavaScript. Unfortunately, Java was not designed to run efficiently in the browser. There are Java APIs that are impossible to implement without generating inefficient JavaScript. Some of these APIs are: reflection, resources, class loaders, and JNI. TeaVM restricts usage of these APIs. Generally, you'll have to manually rewrite your code to fit into TeaVM constraints.</p> <p>TeaVM is for you, if:</p> <ul> <li>You are a Java developer and you are going to write a web front-end from scratch.</li> <li>You already have a Java-based backend and want to integrate front-end code tightly into your existing development infrastructure.</li> <li>You have some Java back-end code you want to reuse in the front end.</li> <li>You are ready to rewrite your code to work with TeaVM.</li> </ul> <p>If you have tightly-coupled applications that use Swing, you want to run these applications in web, and you don't care about download size, start-up time and performance, you should probably look elsewhere; there are more appropriate tools for you, like CheerpJ.</p> <h2>Strong parts</h2> <ul> <li>TeaVM tries to reconstruct original structure of a method, so in most cases it produces JavaScript that you would write manually. No bloated while/switch statements, as naive compilers often do.</li> <li>TeaVM has a very sophisticated optimizer, which knows a lot about your code. Some examples are: <ul> <li>Dead code elimination produces very small JavaScript.</li> <li>Devirtualization turns virtual calls into static function calls, which makes code faster.</li> <li>TeaVM can reuse one local variables to store several local variables.</li> <li>TeaVM renames methods to as short forms as possible; UglifyJS usually can't perform such optimization.</li> </ul> </li> <li> <p>TeaVM supports threads. JavaScript does not provide APIs to create threads (WebWorkers are not threads, since they don't allow to share state between workers). TeaVM is capable of transforming methods to continuation-passing style. This makes possible to emulate multiple logical threads in one physical thread. TeaVM threads are, in fact, green threads.</p> </li> <li>TeaVM is very fast, you don't need to wait for minutes until your application gets recompiled.</li> <li>TeaVM produces source maps; TeaVM IDEA plugin allows you to debug code right from the IDE.</li> <li>TeaVM has a nice JavaScript interop API.</li> </ul> <h3>Integrating with Gradle</h3> <p>Here's a minimal <code>build.gradle</code>:</p> <pre><code>plugins { id "java" id "war" id "org.teavm" version "0.12.3" } repositories { mavenCentral() } // This is optional, but for real-world applications you need to interact with browser. // This dependency provides useful wrappers. dependencies { implementation teavm.libs.jsoApis } teavm { all { mainClass = "example.MainClass" } js { addedToWebApp = true // this is also optional, default value is &lt;project name&gt;.js targetFileName = "example.js" } wasmGC { addedToWebApp = true // this is also optional, default value is &lt;project name&gt;.wasm targetFileName = "example.wasm" } } </code></pre> <p>If you don't need either JS or WebAssembly GC, you can remove corresponding <code>addedToWebApp</code> setting, or ever entire configuration section related to particular backend.</p> <p>where <code>MainClass</code> could do something simple like writing "Hello, world" string in the console. A bit more complex example of MainClass could be following:</p> <pre><code>package example; import org.teavm.jso.dom.html.HTMLDocument; public class MainClass { public static void main(String[] args) { var document = HTMLDocument.current(); var div = document.createElement("div"); div.appendChild(document.createTextNode("TeaVM generated element")); document.getBody().appendChild(div); } } </code></pre> <p>Finally, you need to add to your webapp resources index.html page, which includes examples.js and runs main method, like this:</p> <pre><code>&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;TeaVM example&lt;/title&gt; &lt;meta http-equiv="Content-Type" content="text/html;charset=utf-8"&gt; &lt;script type="text/javascript" charset="utf-8" src="js/example.js"&gt;&lt;/script&gt; &lt;/head&gt; &lt;body onload="main()"&gt;&lt;/body&gt; &lt;/html&gt; </code></pre> <p>or if you need WebAssembly, HTML file like this:</p> <pre><code>&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;TeaVM WebAssembly example&lt;/title&gt; &lt;meta http-equiv="Content-Type" content="text/html;charset=utf-8"&gt; &lt;script type="text/javascript" charset="utf-8" src="wasm-gc/example.wasm-runtime.js"&gt;&lt;/script&gt; &lt;script&gt; async function main() { let teavm = await TeaVM.wasmGC.load("wasm-gc/example.wasm"); teavm.exports.main([]); } &lt;/script&gt; &lt;/head&gt; &lt;body onload="main()"&gt;&lt;/body&gt; &lt;/html&gt; </code></pre> <p>now you can run <code>gradle build</code> or, if you are using Gradle wrapper, <code>./gradlew build</code> / <code>gradlew.bat build</code>. Finally, take <code>.war</code> file from <code>build/libs</code> directory and deploy it to any compatible container or simply unzip and open <code>index.html</code>.</p> <p>Note that during development you can use Gretty plugin to serve you .war file via HTTP.</p> https://tedneward.github.io/Research/vms/wasm/wagon/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wagon/index.html <p><a href="https://github.com/go-interpreter/wagon">Source</a></p> <p>Implemented in Go. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/wasm3/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasm3/index.html <p><a href="https://github.com/wasm3/wasm3">Source</a></p> <p>Implemented in C. Interpreted.</p> <ul> <li><a href="https://wapm.io/package/vshymanskyy/wasm3">Wasm3 on WAPM</a> - WAPM package.</li> </ul> <p><strong>Why use a "slow interpreter" versus a "fast JIT"?</strong></p> <p>In many situations, speed is not the main concern. Runtime executable size, memory usage, startup latency can be improved with the interpreter approach. Portability and security are much easier to achieve and maintain. Additionally, development impedance is much lower. A simple library like Wasm3 is easy to compile and integrate into an existing project. (Wasm3 builds in a just few seconds). Finally, on some platforms (i.e. iOS and WebAssembly itself) you can't generate executable code pages in runtime, so JIT is unavailable.</p> <p><strong>Why would you want to run WASM on embedded devices?</strong></p> <p>Wasm3 started as a research project and remains so by any means. Evaluating the engine in different environments is part of the research. Given that we have <code>Lua</code>, <code>JS</code>, <code>Python</code>, <code>Lisp</code>, <code>...</code> running on MCUs, <code>WebAssembly</code> is a promising alternative. It provides toolchain decoupling as well as a completely sandboxed, well-defined, predictable environment. Among practical use cases we can list <code>edge computing</code>, <code>scripting</code>, <code>plugin systems</code>, running <code>IoT rules</code>, <code>smart contracts</code>, etc.</p> <p><strong>Interoperability with other languages</strong>:</p> <ul> <li>Python</li> <li>C/C++</li> <li>Rust</li> <li>Go</li> <li>Zig</li> <li>Swift</li> <li>C#/.Net</li> </ul> <p><strong>Non-MVP features supported</strong>: Multi-Value, Bulk Memory Operations, Sign-extension operators, Non-trapping conversions, Name Section</p> <p><strong>Host APIs supported</strong>: WASI, Custom</p> <p><strong>Non-web standards</strong>: WASI, Gas Metering</p> <p><strong>Used by</strong></p> <ul> <li><a href="https://wasmcloud.dev/">wasmcloud</a> - A platform for writing portable business logic</li> <li><a href="https://shareup.app/">Shareup</a> - Fast, private sharing for everyone</li> <li><a href="https://wowcube.com/">WowCube</a> - An nnovative console and the gaming platform</li> <li><a href="https://github.com/saghul/txiki.js">txiki.js</a> -A small and powerful JavaScript runtime</li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmi/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmi/index.html <p><a href="https://github.com/paritytech/wasmi">Source</a></p> <p>Implemented in Rust. Interpreted.</p> <p><strong>Non-MVP features supported</strong>:</p> <ul> <li>Mutable Globals</li> <li>Sign-extension operators</li> <li>Non-trapping conversions</li> <li>Multi-Value</li> <li>Bulk Memory Operations</li> <li>Reference Types</li> <li>Tail Calls</li> <li>Extended Const</li> </ul> <p><strong>Non-web standards</strong></p> <ul> <li>WASI</li> </ul> <p><strong>Used by</strong></p> <ul> <li>Substrate</li> <li>smoldot</li> <li>Ayaka Game Engine</li> </ul> https://tedneward.github.io/Research/vms/wasm/wasmvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/wasmvm/index.html <p><a href="https://github.com/WasmVM/WasmVM">Source</a></p> <p>Implemented in C++. Interpreted.</p> https://tedneward.github.io/Research/vms/wasm/webasm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/webasm/index.html <p><a href="https://github.com/dcodeIO/webassembly">Source</a></p> <p>Implemented in Javascript using NodeJS/V8. JIT compiled.</p> https://tedneward.github.io/Research/vms/yarv/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/yarv/index.html <p><a href="https://kddnewton.com/yarv/">Instruction docs</a> | <a href="http://www.atdot.net/yarv/yarvarch.ja.html">Design Document</a></p> <h2>Articles</h2> <ul> <li>Compiling Ruby <ul> <li><a href="https://lowlevelbits.org/compiling-ruby-part-0/">Part 0</a></li> <li><a href="https://lowlevelbits.org/compiling-ruby-part-1/">Part 1</a></li> </ul> </li> <li><a href="https://www.hexdevs.com/posts/ruby-just-in-time-compiler-tenderjit/">A Tender Introduction to Ruby Internals with TenderJIT</a></li> <li><a href="https://sitaramshelke.medium.com/inside-rubyvm-967b25e234db">Inside RubyVM</a></li> <li><a href="https://i.loveruby.net/ja/rhg/book/">Ruby source code complete explanation</a> (Jul 2004) - Japanese text</li> <li><a href="https://magazine.rubyist.net/articles/0006/0006-YarvManiacs.html">YARV maniacs</a> - (May 2005) Japanese text</li> <li><a href="http://graysoftinc.com/the-ruby-vm-interview/the-ruby-vm-serial-interview">The Ruby VM Serial Interview</a> - (Feb 2007)</li> <li><a href="https://lifegoo.pluskid.org/upload/doc/yarv/yarv_iset.html">YARV instruction set</a> - (Apr 2008)</li> <li><a href="https://patshaughnessy.net/2012/6/29/how-ruby-executes-your-code">How Ruby Executes Your Code (excerpt from Ruby Under a Microscope)</a> - (Jun 2012)</li> <li><a href="http://www.atdot.net/~ko1/diary/201212.html#d1">ko1's 2012 Ruby VM advent calendar</a> - (Dec 2012) Japanese text</li> <li><a href="https://qiita.com/sisshiki1969/items/3d25aa81a376eee2e7c2">Ruby made with Rust</a> - (Dec 2019)</li> <li><a href="https://iliabylich.github.io/2020/01/25/evaluating-ruby-in-ruby.html">Evaluating Ruby in Ruby</a> - (Jan 2020)</li> </ul> <h2><a href="https://lifegoo.pluskid.org/upload/doc/yarv/yarv_iset.html">Bytecode set</a></h2> <ul> <li><code>adjuststack</code>: accepts a single integer argument and removes that many elements from the top of the stack.</li> <li><code>anytostring</code>: ensures that the value on top of the stack is a string.</li> <li><code>branchif</code>: If the value popped off the stack is <code>true</code>, branchif jumps to the jump index and continues executing there.</li> <li><code>branchnil</code>: If the value popped off the stack is <code>nil</code>, branchnil jumps to the jump index and continues executing there.</li> <li><code>branchunless</code>: If the value popped off the stack is <code>false</code> or <code>nil</code>, branchunless jumps to the jump index and continues executing there.</li> <li><code>concatarray</code>: concatenates the two Arrays on top of the stack.</li> <li><code>concatstrings</code>: just pops a number of strings from the stack joins them together into a single string and pushes that string back on the stack.</li> <li><code>defined</code>: checks if the top value of the stack is defined. If it is, it pushes its value onto the stack. Otherwise it pushes <code>nil</code>.</li> <li><code>definemethod</code>: defines a method on the class of the current value of <code>self</code>.</li> <li><code>dup</code>: copies the top value of the stack and pushes it onto the stack.</li> <li><code>dup_hash</code>: pushes a hash onto the stack.</li> <li><code>duparray</code>: copies a literal Array and pushes it onto the stack.</li> <li><code>dupn</code>: duplicates the top n stack elements.</li> <li><code>expandarray</code>: looks at the top of the stack, and if the value is an array it replaces it on the stack with num elements of the array, or nil if the elements are missing.</li> <li><code>getconstant</code>: performs a constant lookup and pushes the value of the constant onto the stack.</li> <li><code>getglobal</code>: pushes the value of a global variables onto the stack.</li> <li><code>getlocal</code>: fetches the value of a local variable from a frame determined by the level and index arguments.</li> <li><code>getlocal_wc_0</code>: is a specialized version of the getlocal instruction. It fetches the value of a local variable from the current frame determined by the index given as its only argument.</li> <li><code>getlocal_wc_1</code>: is a specialized version of the getlocal instruction. It fetches the value of a local variable from the parent frame determined by the index given as its only argument.</li> <li><code>intern</code>: converts top element from stack to a symbol.</li> <li><code>invokeblock</code>: invokes the block passed to a method during a yield.</li> <li><code>jump</code>: set PC to (PC + dst).</li> <li><code>leave</code>: exits the current frame.</li> <li><code>newarray</code>: puts a new array initialized with size values from the stack.</li> <li><code>newhash</code>: puts a new hash onto the stack, using num elements from the stack. num needs to be even.</li> <li><code>newrange</code>: creates a Range.</li> <li><code>nop</code>: is a no-operation instruction.</li> <li><code>objtostring</code>: pops a value from the stack, calls to_s on that value and then pushes the result back to the stack.</li> <li><code>opt_and</code>: a specialization of the opt_send_without_block instruction that occurs when the &amp; operator is used.</li> <li><code>opt_aref</code>: a specialization of the opt_send_without_block instruction that occurs when the [] operator is used.</li> <li><code>opt_aref_with</code>: a specialization of the opt_aref instruction that occurs when the [] operator is used with a string argument known at compile time.</li> <li><code>opt_aset</code>: an instruction for setting the hash value by the key in recv[obj] = set format.</li> <li><code>opt_aset_with</code>: an instruction for setting the hash value by the known string key in the recv[obj] = set format.</li> <li><code>opt_case_dispatch</code>: a branch instruction that moves the control flow for case statements.</li> <li><code>opt_div</code>: a specialization of the opt_send_without_block instruction that occurs when the / operator is used.</li> <li><code>opt_empty_p</code>: an optimization applied when the method empty? is called on a String, Array or a Hash.</li> <li><code>opt_eq</code>: a specialization of the opt_send_without_block instruction that occurs when the == operator is used.</li> <li><code>opt_ge</code>: a specialization of the opt_send_without_block instruction that occurs when the &gt;= operator is used.</li> <li><code>opt_getinlinecache</code>: a wrapper around a series of getconstant instructions that allows skipping past them if the inline cache is currently set.</li> <li><code>opt_gt</code>: a specialization of the opt_send_without_block instruction that occurs when the &gt; operator is used.</li> <li><code>opt_le</code>: a specialization of the opt_send_without_block instruction that occurs when the &lt;= operator is used.</li> <li><code>opt_length</code>: a specialization of opt_send_without_block, when the length method is called on a Ruby type with a known size.</li> <li><code>opt_lt</code>: a specialization of the opt_send_without_block instruction that occurs when the &lt; operator is used.</li> <li><code>opt_ltlt</code>: a specialization of the opt_send_without_block instruction that occurs when the &lt;&lt; operator is used.</li> <li><code>opt_minus</code>: a specialization of the opt_send_without_block instruction that occurs when the - operator is used.</li> <li><code>opt_mod</code>: a specialization of the opt_send_without_block instruction that occurs when the % operator is used.</li> <li><code>opt_mult</code>: a specialization of the opt_send_without_block instruction that occurs when the * operator is used.</li> <li><code>opt_neq</code>: an optimisation that tests whether two values at the top of the stack are not equal by testing their equality and performing a logical NOT on the result.</li> <li><code>opt_newarray_max</code>: an instruction that represents calling max on an array literal. It is used to optimize quick comparisons of array elements.</li> <li><code>opt_newarray_min</code>: an instruction that represents calling min on an array literal.</li> <li><code>opt_nil_p</code>: an optimization applied when the method nil? is called.</li> <li><code>opt_not</code>: negates the value on top of the stack.</li> <li><code>opt_or</code>: a specialization of the opt_send_without_block instruction that occurs when the | operator is used.</li> <li><code>opt_plus</code>: a specialization of the opt_send_without_block instruction that occurs when the + operator is used.</li> <li><code>opt_regexpmatch2</code>: a specialization of the opt_send_without_block instruction that occurs when the =~ operator is used.</li> <li><code>opt_send_without_block</code>: a specialization of the send instruction that occurs when a method is being called without a block.</li> <li><code>opt_setinlinecache</code>: the final instruction after a series of getconstant instructions that populates the inline cache associated with an opt_getinlinecache instruction.</li> <li><code>opt_size</code>: a specialization of opt_send_without_block, when the size method is called on a Ruby type with a known size.</li> <li><code>opt_str_freeze</code>: pushes a frozen known string value with no interpolation onto the stack.</li> <li><code>opt_str_uminus</code>: pushes a frozen known string value with no interpolation onto the stack.</li> <li><code>opt_succ</code>: a specialization of the opt_send_without_block instruction when the method being called is succ.</li> <li><code>pop</code>: pops the top value off the stack.</li> <li><code>putnil</code>: pushes a global nil object onto the stack.</li> <li><code>putobject</code>: pushes a known value onto the stack.</li> <li><code>putobject_int2fix_0</code>: pushes 0 on the stack. It is a specialized instruction resulting from the operand unification optimization. It is the equivalent to putobject 0.</li> <li><code>putobject_int2fix_1</code>: pushes 1 on the stack. It is a specialized instruction resulting from the operand unification optimization. It is the equivalent to putobject 1.</li> <li><code>putself</code>: pushes the current value of self onto the stack.</li> <li><code>putstring</code>: pushes a string literal onto the stack.</li> <li><code>send</code>: invokes a method with a block.</li> <li><code>setglobal</code>: sets the value of a global variable.</li> <li><code>setlocal</code>: sets the value of a local variable on a frame determined by the level and index arguments. The level is the number of frames back to look and the index is the index in the local table. It pops the value it is setting off the stack.</li> <li><code>setlocal_wc_0</code>: a specialized version of the setlocal instruction. It sets the value of a local variable on the current frame to the value at the top of the stack as determined by the index given as its only argument.</li> <li><code>setlocal_wc_1</code>: a specialized version of the setlocal instruction. It sets the value of a local variable on the parent frame to the value at the top of the stack as determined by the index given as its only argument.</li> <li><code>setn</code>: an instruction for set Nth stack entry to stack top.</li> <li><code>splatarray</code>: calls to_a on an array to splat.</li> <li><code>swap</code>: swaps the top two elements in the stack.</li> <li><code>topn</code>: has one argument: n. It gets the nth element from the top of the stack and pushes it on the stack.</li> <li><code>toregexp</code>: is generated when string interpolation is used inside a regex literal //. It pops a defined number of values from the stack, combines them into a single string and coerces that string into a Regexp object, which it pushes back onto the stack.</li> </ul> <h2>Tools</h2> <ul> <li><a href="https://github.com/kddnewton/yarv">YARV emulator written in Ruby</a></li> </ul> https://tedneward.github.io/Research/vms/seax/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/seax/index.html <p><a href="https://github.com/hawkw/seax">Source</a> | <a href="https://github.com/hawkw/seax_svm">Source</a></p> https://tedneward.github.io/Research/vms/spim/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/spim/index.html <p><a href="https://sourceforge.net/projects/spimsimulator/">Source</a> | <a href="https://github.com/Tipoca/spim">Source</a> (Fork)</p> https://tedneward.github.io/Research/vms/sushivm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/sushivm/index.html <p><a href="https://github.com/eqela/sushivm">Source</a></p> <p>Sushi is a fast and lightweight JIT-enabled virtual machine and dynamic language interpreter. It embeds and utilizes the LuaJIT interpreter and JIT engine and implements a fully custom-built cross-platform built-in API for Sushi applications. The entire virtual machine runs from a single executable and requires no particular installation. Executable Sushi programs can be compressed and appended to the executable, easily creating single-file executables that contain both the interpreter and application code. The executable size is about 400kb when dynamically linked and around 4MB when linked statically.</p> <p>Sushi uses the Lua syntax as its native code format. While it is possible to write and execute hand-written Lua code on Sushi as well, the primary purpose of Sushi is to provide an optimal runtime environment for the Sling Programming Language.</p> https://tedneward.github.io/Research/vms/vm86/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/vm86/index.html <p><a href="https://github.com/tboox/vm86">Source</a></p> https://tedneward.github.io/Research/vms/sabervm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/sabervm/index.html <p><a href="https://github.com/SaberVM/SaberVM">Source</a></p> <blockquote> <p>It will eventually have a number of implementations on different systems, and it will hopefully be easy enough to implement your own on your system/platform of choice.</p> </blockquote> <h2>From the README</h2> <h3>What is SaberVM?</h3> <p>SaberVM uses a typed stack-based bytecode language with compile-time and run-time instructions. Each instruction is one byte, though some instructions eat the next 1-4 bytes as a compile-time parameter. This all leads to a very expressive compile-time language for annotations that SaberVM uses for safety guarantees and, eventually, optimization.</p> <p>By design, SaberVM will be able to be AOT-compiled with a number of optimizations, run on a JIT VM, or run on a simple VM for hobby projects or resource-constrained environments. Currently, a naive VM implementation is fairly fleshed-out, and being used as a prototype for rapid design iteration.</p> <p>SaberVM has some of the resilience guarantees of the BEAM, and uses a concurrent semantics for being run in parallel or sequentially. It can also be compiled without memory safety runtime checks/tagging for a more LLVM-like experience. SaberVM is built on the research of <a href="https://scholar.google.com/citations?user=Dswus94AAAAJ&amp;hl=en">Dr. Greg Morrisett</a>, some of which (see Cyclone in <a href="https://doc.rust-lang.org/reference/influences.html">here</a>) inspired Rust's memory safety model.</p> <p>In a traditional CPS-based compiler pipeline, SaberVM is targeted after closure-conversion and hoisting, though like Erlang it will eventually have higher-level languages (namely Saber) to target and even use for projects.</p> <h3>Why target SaberVM?</h3> <p>A common choice for compiler backends is LLVM. However, for CPS-based compilers of functional languages, LLVM is not really an option. CPS is a good backend language for functional languages, for reasons that are too low-level to be expressed in LLVM's IR. LLVM is too tied to having C-style calling conventions and a return-address stack. In addition, functional languages (with pervasive immutability and recursion) can have their own set of good optimizations and heuristics that LLVM won't do. SaberVM aims to fill the vacuum left by LLVM, by being a CPS-based compiler backend oriented towards functional languages. However, it doesn't expect to be the only player in that field, and therefore makes opinionated tradeoffs. Namely, SaberVM tries to optimize for safety, portability, and expressivity.</p> <p>SaberVM bytecode can guarantee memory safety and deadlock freedom, without a loss of expressivity. This is achieved in a way similar to <a href="https://vale.dev/">Vale</a>, with a combination of borrow-checker-like compiler analysis and generational-reference-like runtime checks. These runtime checks don't crash the program when a dangling pointer is dereferenced, they instead put the program in a built-in panic mode so the broken parts of the program can be restarted, as in Erlang (but more low-level and manual).</p> <p>This all combines to no loss in expressivity for safe programs, such that you can even write garbage collectors for your language without borrow-checker-induced nightmares. Supporting user-written garbage collectors is, of course, a hard requirement when we're trying to support functional languages. SaberVM's memory model finds a way to do this without compromising the soundness of the memory safety guarantees.</p> <p>However, a compiler that cares more about performance than safety can turn off the runtime checks and tagging without any change to the bytecode; SaberVM's bytecode language is designed to be runtime-check-agnostic: proper resource handling is checked, not inferred.</p> <p>SaberVM can also be used in a JVM-style, where the compiler simply outputs SaberVM bytecode and ships it to users, who have their own SaberVM or AOT-compiler. Thus they can run your program in a safe mode if they don't trust you (like websites), or a fast mode if they do (like games).</p> <h3>How does it work?</h3> <p>SaberVM has two main systems at play: regions and exceptions.</p> <h4>Regions</h4> <p>A region can be thought of as an arena with a malloc/free-style internal memory management system. To read or write from the heap, you need a region. Regions have statically-checked lifetimes and a borrow-checker-like system for checking that values within that region are only read or written to during the lifetime of the region. When a region's lifetime ends, it is freed like an arena. Doing borrow-checking on the scale of regions instead of values means that values can be freely mutated and aliased however you see fit; the system ensures that pointers into the region are never aliased after the region is freed, even if they continue to be passed around.</p> <p>This structure of memory is important because in a safe compilation the values in the heap are tagged with information about which inhabitant is at that place in memory, so pointers can check that they're pointing at the thing they think they are when they're dereferenced. This introduces a memory fragmentation issue: "slots" in memory can then later be used only by values that are the same size or smaller; they have an unchangeable "max size." To prove this, consider two values <strong>adjacent</strong> in memory, <code>A</code> and <code>B</code>, and their pointers, <code>&amp;A</code> and <code>&amp;B</code>. Now say we free <strong>both</strong>, and allocate <code>C</code> at the same address where <code>A</code> was (that is, <code>&amp;C == &amp;A</code>). If <code>C</code> is bigger than <code>A</code> was, then it has arbitrary control of the bytes used to tag <code>B</code>! <code>B</code> has been freed, but now that <code>C</code> controls those bytes, it could potentially make the bytes where <code>B</code>'s tag was appear to still show <code>B</code> as unfreed. That means a nefarious program could potentially cause <code>&amp;B</code> to think that <code>B</code> is still there, leading dereferences of <code>&amp;B</code> to read memory controlled by <code>C</code>, instead of panicking.</p> <p>The solution to this is to require that <code>C</code> is no larger than the <code>A</code> that was there before it. If allocating something in memory fixes a certain max-size for that bit of memory for the rest of the program's lifetime, that can cause issues with poor use of memory. Therefore, SaberVM puts its values in <strong>regions</strong> so there are certain points in the program's lifecycle where it's statically known that nothing will dereference some set of pointers ever again, so their referent memory can be <em>really</em> freed, with no restriction on its future use. As a language writer, if you find your output programs have significant fragmentation issues, you can do some light region inference to fix it. In addition, since regions are freed like arenas (reading uninitialized memory is statically prevented), regions offer a way to deallocate a bunch of memory instantly, and improve cache locality.</p> <h4>Exceptions</h4> <p>NOTE: this is currently unimplemented in the VM prototype, simply because I'm just one person. You can <a href="CONTRIBUTING.md">contribute</a> these things yourself (super appreciated!) or financially support me so I can work fewer hours at my day job, if you want to see these features sooner.</p> <p>SaberVM's other interesting system is exceptions. Exceptions in SaberVM are not like normal exceptions, though there's nothing stopping a compiler writer from building a normal exception system on top of SaberVM. Instead, SaberVM exceptions <strong>don't take arguments</strong>. Every function must have a catch-all exception case, and only that. Why? Having this built-in to SaberVM means that instructions that fail at runtime don't crash your program, they just jump to the exception handler. The language targeting SaberVM is then expected to produce exception handlers that do at least one of four things:</p> <ol> <li>crash the program (with an explicit <code>halt</code> instruction)</li> <li>rethrow the exception (that is, jump to the <em>caller's</em> catch-all exception handler)</li> <li>restart the crashed function (in a microreboot or Erlang style, without information about what caused the exception)</li> <li>release held resources (currently SaberVM doesn't have locks, only CAS, but this is likely to change)</li> </ol> <p><strong>Note that SaberVM exceptions are not expected to be how your own language handles its exceptions!</strong></p> <p>For example, if you prefer a <code>Result</code>-style exception handling, you can write functions that attempt single fallible instructions with an exception handler that produces the corresponding <code>Err</code> value.</p> <h3>The progress so far</h3> <p>Currently work is underway on an MVP, that is, a simple non-JITing VM in Rust. Like Wasm, SaberVM bytecode must be verified before it is run. The project so far can parse an array of bytes, typecheck it, and execute it, but only supports a subset of the full SaberVM design. Thankfully, SaberVM has been designed to be easy to implement, so that languages targeting it can easily extend the number of platforms they support. Indeed, development has gone extremely swiftly so far.</p> <p>The type checker uses a two-phase system: functions are forward-declared, and then their definitions are checked. This can be done in parallel trivially, though it isn't at the moment.</p> <p>There are 27 instructions implemented right now, chosen to implement a simple "duplicate" function in naive CPS.</p> <pre><code>foo x = (x, x) </code></pre> <p>In SaberVM this has a fairly complex type:</p> <pre><code>Forall r: Rgn, t: Type: 8-byte. (t, Exists u: 8-byte. ((u, (t, t)@r)-&gt;0, u)@r, handle(r))-&gt;0 </code></pre> <p>To pick this apart a little, the function is quantified over a region, and a type that must be 8 bytes in size. <code>(a..)-&gt;0</code> is the syntax for a function type that takes the arguments <code>a..</code>. The <code>-&gt;0</code> notation is to suggest that it's a CPS function, and therefore doesn't return. With the default region-quantification, you can imagine that <code>r</code> is borrowed, if you're familiar with Rust. <code>Exists x: repr. t</code> is an existential type, and is mostly used for typing closure environments. In this case, the continuation is existentially quantified so that it could have any closure environment struct. That is, a closure is typed at <code>Exists x. ((x, a..)-&gt;0, x)@R</code>. The <code>(a..)@R</code> notation is product types (tuples) that are allocated in region <code>R</code>. Finally, <code>handle(R)</code> is the singleton type for the runtime information of a region <code>R</code>, holding for example the pointer to the physical region in memory.</p> <p>In summary, that type says that the function takes a value of type <code>t</code>, a region <code>r</code>, and a continuation that expects a value of type <code>(t, t)</code> allocated in region <code>r</code>. It also requires that <code>r</code> hasn't been freed. That should seem like a reasonable type for the duplicate function. The richness of the information present in the type should hopefully justify how many compile-time instructions are used to construct it! Resorting too much to type inference would hurt the performance of the verifier asymptotically, so in the tradeoff between binary size and verifier performance I chose a faster verifier. Very small instructions helps the binary size a lot already.</p> <p>I'm also writing a little lambda-calculus <a href="https://github.com/RyanBrewer317/SaberLC">compiler</a> to get much better tests of SaberVM.</p> <p>This is very WIP; the purpose of this MVP is not to ship it per se, but mostly to play with the system and figure out what breaking changes need to be made. I've already gone through several major rewrites and redesigns!</p> <p>Polymorphism is currently just done in a simple Kinds-Are-Calling-Conventions-inspired approach, where the number of bytes of acceptible are specified by the quantifier. This is like given SaberVM a second, much simpler, C-like type system that has no polymorphism and only cares about numbers of bytes.</p> https://tedneward.github.io/Research/vms/secd/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/secd/index.html <p><a href="https://en.wikipedia.org/wiki/SECD_machine">Wikipedia</a> | <a href="https://web.archive.org/web/20100620004154/http://fsl.cs.uiuc.edu/images/e/ef/P157-landin.pdf">"The next 700 programming languages"</a> | <a href="https://pqnelson.github.io/org-notes/comp-sci/abstract-machines/secd.html">SECD Machine</a></p> <p>The letters stand for Stack, Environment, Control, Dump--the internal registers of the machine. The registers Stack, Control, and Dump point to (some realizations of) stacks, and Environment points to (some realization of) an associative array.</p> <h3>Informal description</h3> <p>When evaluation of an expression begins, the expression is loaded as the only element of control C. The environment E, stack S and dump D begin empty.</p> <p>During evaluation of C it is converted to reverse Polish notation (RPN) with ap (for apply) being the only operator. For example, the expression F (G X) (a single list element) is changed to the list X:G:ap:F:ap.</p> <p>Evaluation of C proceeds similarly to other RPN expressions. If the first item in C is a value, it is pushed onto the stack S. More exactly, if the item is an identifier, the value pushed onto the stack will be the binding for that identifier in the current environment E. If the item is an abstraction, a closure is constructed to preserve the bindings of its free variables (which are in E), and it is this closure which is pushed onto the stack.</p> <p>If the item is ap, two values are popped off the stack and the application done (first applied to second). If the result of the application is a value, it is pushed onto the stack.</p> <p>If the application is of an abstraction to a value, however, it will result in a lambda calculus expression that may itself be an application (rather than a value), and so cannot be pushed onto the stack. In this case, the current contents of S, E, and C are pushed onto the dump D (which is a stack of these triples), S is reinitialized to empty, and C is reinitialized to the application result with E containing the environment for the free variables of this expression, augmented with the binding that resulted from the application. Evaluation then proceeds as above.</p> <p>Completed evaluation is indicated by C being empty, in which case the result will be on the stack S. The last saved evaluation state on D is then popped, and the result of the completed evaluation is pushed onto the stack contents restored from D. Evaluation of the restored state then continues as above.</p> <p>If C and D are both empty, overall evaluation has completed with the result on the stack S.</p> <h3>Registers and memory</h3> <p>The SECD machine is stack-based. Functions take their arguments from the stack. The arguments to built-in instructions are encoded immediately after them in the instruction stream.</p> <p>Like all internal data-structures, the stack is a list, with the S register pointing at the list's head or beginning. Due to the list structure, the stack need not be a continuous block of memory, so stack space is available as long as there is a single free memory cell. Even when all cells have been used, garbage collection may yield additional free memory. Obviously, specific implementations of the SECD structure can implement the stack as a canonical stack structure, thus improving the overall efficiency of the virtual machine, provided that a strict bound be put on the dimension of the stack.</p> <p>The C register points at the head of the code or instruction list that will be evaluated. Once the instruction there has been executed, the C is pointed at the next instruction in the list—it is similar to an instruction pointer (or program counter) in conventional machines, except that subsequent instructions are always specified during execution and are not by default contained in subsequent memory locations, as is the case with the conventional machines.</p> <p>The current variable environment is managed by the E register, which points at a list of lists. Each individual list represents one environment level: the parameters of the current function are in the head of the list, variables that are free in the current function, but bound by a surrounding function, are in other elements of E.</p> <p>The dump, at whose head the D register points, is used as temporary storage for values of the other registers, for example during function calls. It can be likened to the return stack of other machines.</p> <p>The memory organization of the SECD machine is similar to the model used by most functional language interpreters: a number of memory cells, each of which can hold either an atom (a simple value, for example 13), or represent an empty or non-empty list. In the latter case, the cell holds two pointers to other cells, one representing the first element, the other representing the list except for the first element. The two pointers are traditionally named car and cdr respectively—but the more modern terms head and tail are often used instead. The different types of values that a cell can hold are distinguished by a tag. Often different types of atoms (integers, strings, etc.) are distinguished as well.</p> <p>So, a list holding the numbers 1, 2, and 3, usually written as (1 2 3), might be represented as follows:</p> <pre><code>Address Tag Content (value for integers, car &amp; cdr for lists) 9 [ integer | 2 ] 8 [ integer | 3 ] 7 [ list | 8 | 0 ] 6 [ list | 9 | 7 ] ... 2 [ list | 1 | 6 ] 1 [ integer | 1 ] 0 [ nil ] </code></pre> <p>The memory cells 3 to 5 do not belong to our list, the cells of which can be distributed randomly over the memory. Cell 2 is the head of the list, it points to cell 1, which holds the first element's value, and the list containing only 2 and 3 (beginning at cell 6). Cell 6 points at a cell holding 2 and at cell 7, which represents the list containing only 3. It does so by pointing at cell 8 containing the value 3, and pointing at an empty list (nil) as cdr. In the SECD machine, cell 0 always implicitly represents the empty list, so no special tag value is needed to signal an empty list (everything needing that can simply point to cell 0).</p> <p>The principle that the cdr in a list cell must point at another list is just a convention. If both car and cdr point at atoms, that will yield a pair, usually written like (1 . 2)</p> <h3>Instructions</h3> <ul> <li><code>nil</code> pushes a nil pointer onto the stack</li> <li><code>ldc</code> pushes a constant argument onto the stack</li> <li><code>ld</code> pushes the value of a variable onto the stack. The variable is indicated by the argument, a pair. The pair's car specifies the level, the cdr the position. So (1 . 3) gives the current function's (level 1) third parameter.</li> <li><code>sel</code> expects two list arguments, and pops a value from the stack. The first list is executed if the popped value was non-nil, the second list otherwise. Before one of these list pointers is made the new C, a pointer to the instruction following <code>sel</code> is saved on the dump.</li> <li><code>join</code> pops a list reference from the dump and makes this the new value of C. This instruction occurs at the end of both alternatives of a sel.</li> <li><code>ldf</code> takes one list argument representing a function. It constructs a closure (a pair containing the function and the current environment) and pushes that onto the stack.</li> <li><code>ap</code> pops a closure and a list of parameter values from the stack. The closure is applied to the parameters by installing its environment as the current one, pushing the parameter list in front of that, clearing the stack, and setting C to the closure's function pointer. The previous values of S, E, and the next value of C are saved on the dump.</li> <li><code>ret</code> pops one return value from the stack, restores S, E, and C from the dump, and pushes the return value onto the now-current stack.</li> <li><code>dum</code> pushes a "dummy", an empty list, in front of the environment list.</li> <li><code>rap</code> works like <code>ap</code>, only that it replaces an occurrence of a dummy environment with the current one, thus making recursive functions possible</li> </ul> <p>A number of additional instructions for basic functions like car, cdr, list construction, integer addition, I/O, etc. exist. They all take any necessary parameters from the stack.</p> https://tedneward.github.io/Research/vms/squeakvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/squeakvm/index.html <p><a href="http://wiki.squeak.org/squeak/1447">Website</a></p> <p><a href="https://github.com/victorr/jsqueak">JSqueak</a>: JVM-based port of Squeak, originally at <a href="http://research.sun.com/projects/JSqueak/">Sun Research</a> but since closed down.</p> https://tedneward.github.io/Research/vms/topazruby/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/topazruby/index.html <p><a href="http://topazruby.com/">Website</a> | <a href="https://github.com/topazproject/topaz">Source</a></p> https://tedneward.github.io/Research/vms/wasm/awsm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/wasm/awsm/index.html <p><a href="https://github.com/gwsystems/aWsm">Source</a></p> <p>Implemented in Rust, C using LLVM</p> <p>AOT compilation</p> <p>Interop with C</p> https://tedneward.github.io/Research/vms/salt/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/salt/index.html <p><a href="https://github.com/discus-lang/salt">Source</a></p> <p>Salt is a System-F variant intended as an intermediate language between higher level languages and an abstract assembly language (LLVM). Hand written code can also be used to implement runtime systems. The Disco Discus Compiler uses an earlier version of Salt (v1), and its runtime system is written in it. This current repo contains a newer version of Salt (v2) which is being split out into its own project. Salt v1 has a working LLVM backend, but the one for v2 in this repo is still a work in progress.</p> <p>This project is a V2; see the Github for more notes and links to V1.</p> https://tedneward.github.io/Research/vms/smallvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/smallvm/index.html <p><a href="https://github.com/shanhuio/smlvm">Source</a></p> https://tedneward.github.io/Research/vms/stable/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/stable/index.html <p><a href="https://w3group.de/stable.html">Website</a> | Source on site</p> <p>Stable (STAck Bytecode Language &amp; Engine) is an extreme minimal but useful stack oriented virtual machine, currently written in C (200 lines of extensively commented C code. The original comes with 47 lines of C code). The machine language consists only of printable letters which make it simple hackable with an editor. The goal of this engine is to be human hackable, simple and fast. So there are no complicated address calculations (for branch and jump addresses) and an easy to use instruction set. (43 instructions, 26 registers, 26 user defineable functions)</p> <p>To avoid addresses to variables (registers) and functions, names are predefined. So instead of given the address for a function there is an ASCII opcode for that. Functions are upper case letters from A to Z which leads to 26 user defineable functions, registers are named from a to z (lower letter) which leads to 26 registers.</p> <h3>Instructions</h3> <p>From <a href="https://w3group.de/stable_glossar.html">here</a>:</p> <pre><code>arithmetic + ( a b--a+b) addition - ( a b--a-b) subtraction * ( a b--a*b) multiply / ( a b--a/b) division % ( a b--a%b) modulo (division reminder) _ ( n-- -n) negate bit manipulation &amp; ( a b--a&amp;b) 32 bits and | ( a b--a|b) 32 bits or ~ ( n -- n') not, all bits inversed (0=&gt;1 1=&gt;0) stack # ( a--a a) duplicate top of stack \ ( a b--a) drop top of stack $ ( a b--b a) swap top of stack @ ( a b--a b a) (over) copy next of stack on top register x ( --) select register x (x: a..z) ; ( --value) fetch from selected register : ( value--) store into selected register ? ( --value) selected register contains an address. Fetch value from there ! ( value--) selected register contains an address. Store value there. + ( --) immediately after register, increment content by 1 - ( --) immediately after register, decrement content by 1 functions {X ( --) start function definition for function X (A..Z) } ( --) end of function definition X ( --) call function X (A..Z) I/O . ( value--) display value as decimal number on stdout , ( value--) send value to terminal, 27 is ESC, 10 is newline, etc. ^ ( --key) read key from terminal, don't wait for newline. " ( --) read string until next " put it on stdout 0..9 ( --value) scan decimal number until non digit. to push two values on stack separete those by space (4711 3333) to enter negative numbers call _ (negate) after the number 0..9.0 ( --value) to enter float numbers digits must contain one . (only available if float module is active, see 0`) conditions &lt; ( a b--f) true (-1) if b is &lt; a, false (0) otherwise &gt; ( a b--f) true (-1) if b is &gt; a, false (0) otherwise = ( a b--f) true (-1) if a is queal to b, flase (0) otherwise ( ( f--) if f is true, execute content until ), if false skip code until ) [ ( f--f) begin while loop only if f is true. keep flag on stack if f is false, skip code until ] ] ( f--) repeat the loop if f is true. If f is false, continue with code after ] extensions (expermiental) ` ( n--) call extension function n 0 ... switch to floating point mode + - * / . _ &lt; &gt; 1 ... switch back to interger mode 2 ... dbg, function dbg() to set breakpoint for debugging 3 ... switch traceing on (IP, TOKEN, SP, STACK) (not in stable_fast) 4 ... switch traceing off. Tracing int file trace 5 ... &lt; = &gt; without dropping their 2nd operand (NOS). This is the behavior of Santors original virtual engine. 6 ... mstime, time in milliseconds, for timing 7 ... ( n--) edit block number n 8 ... ( n--) load block number n. Data segment remains. So this is a kind of shared memory. Registers could be used as arguments. After leaving the application and 0 is on TOS, STABLE will be terminated. A value not equal 0 on TOS will load this block. If the block is not valid, the command block will be loaded Use block 0 as an index for all your block numbers 9 ... ( n 9--) copy block n (1000 cells) into memory begining of 1000. write back the old content before. At exit write back current data block. STABLE is starting with block 0 loaded. 10 ... trace only current state (ip, rp, sp, ..) on stdout 11 ... quit VM ( n--) n is exit code to os 12 ... ( --n) push current data block number on stack 13 ... ( --) copy 1000 cells from address 1000 to 2000 14 ... ( --) copy 1000 cells from address 2000 to 1000 </code></pre> <h3>Interpreter</h3> <p>Source in a single C file; there is a <a href="https://w3group.de/stable_fast.c">fast version</a> and a <a href="https://w3group.de/stable_debug.c">debugging</a> version, supping single-step monitoring:</p> <pre><code>/* stable.c interpreter: * * Idea and_ implementation Sandor Schneider * Refactoring and extended Andreas Klimas klimas@w3group.de * * Tracing and Floating point as extension (see stable_glossar.html) * IDE, source and code blocks * * */ #include &lt;stdio.h&gt; #include &lt;stdlib.h&gt; #include &lt;setjmp.h&gt; #include &lt;string.h&gt; #include &lt;ctype.h&gt; #include &lt;unistd.h&gt; #include &lt;termios.h&gt; #include &lt;signal.h&gt; #include &lt;sys/time.h&gt; #include &lt;sys/stat.h&gt; extern void (*words[])(void); static FILE *trace_fp; static int cur_reg,ip; // selected register, instruction pointer static int data[4000];// data segment (register and memory) static char sp=-1,rp=-1,token,k,code[4096];// data stack pointer, return stack pointer, current executed token, and code segment static int stack[256]; // data stack static int rstack[256]; // return stack static short fentry[256];// startaddress of functions A-Z static int is_trace, is_flt; // trace mode, floating point mode static int cur_code_block; // current code block; static int cur_data_blk; // current data block static FILE *data_fp; // database file, permanent open static struct termios oldterm; // for later use onexit() static jmp_buf do_halt; static sigjmp_buf do_sigint; static void trace(void) ; static void noop(void){} // 01..32 static void grow(int delta) { int empty[4000]; memset(empty, 0, sizeof(empty)); fseek(data_fp, 0, SEEK_END); int i; for(i=0;i&lt;delta;i++) { fwrite(empty, 1, sizeof(empty), data_fp); } } static void select_block(int blknr) { cur_data_blk=blknr; struct stat sb; fstat(fileno(data_fp), &amp;sb); unsigned blks=sb.st_size/4000; if((cur_data_blk+1)&gt;blks) grow(blks-cur_data_blk+10); fseek(data_fp, cur_data_blk*sizeof(int)*1000, SEEK_SET); fread(data+1000, sizeof(int), 1000, data_fp); } static void blocknr(void) {stack[++sp]=cur_data_blk;} static void save_block(void) { tcsetattr(0, 0, &amp;oldterm); fseek(data_fp, cur_data_blk*sizeof(int)*1000, SEEK_SET); fwrite(data+1000, sizeof(int), 1000, data_fp); } static void block(void) { save_block(); // write back current block select_block(stack[sp--]); } static int one_key(void) { struct termios t; fflush(stdout); if (!isatty(0)) return 0; tcgetattr(0, &amp;oldterm); tcgetattr(0, &amp;t); t.c_iflag &amp;= ~(ICRNL | ICRNL | IXON|IXANY|IXOFF); t.c_iflag |= BRKINT; t.c_oflag &amp;= ~OPOST; t.c_lflag &amp;= ~(ICANON | ECHO); t.c_lflag |= ISIG; t.c_cc[4] = 1; tcsetattr(0, 0, &amp;t); int c=0; read(0, &amp;c, 1); tcsetattr(0, 0, &amp;oldterm); return c; } static void halt(void) {longjmp(do_halt,1);} // 0-Byte static void store(void){data[data[cur_reg]]=stack[sp];sp--;}// ! static void print(void){char c; while((c=code[ip++])!='\"') putc(c,stdout);} // " static void character(void){stack[++sp]=code[ip++];} static void system_(void){ // ` system char buffer[256], *p=buffer, *end=buffer+sizeof(buffer)-1; char c; while((c=code[ip++])!='`') if(p&lt;end) *p++=c; *p=0; system(buffer); } static void mstime(void) { struct timeval tv[1]; gettimeofday(tv, 0); stack[++sp]=1000*tv-&gt;tv_sec+tv-&gt;tv_usec/1000; } static void dup_(void){sp++; stack[sp]=stack[sp-1]; } // # static void modulo(void){stack[sp-1]%=stack[sp];sp--;} // % static void and_(void){stack[sp-1]&amp;=stack[sp];sp--;} // &amp; bitwise and_ static void if_(void){if(!stack[sp--]) while(code[ip++]!=')');} // (..) static void mul(void){stack[sp-1]*=stack[sp]; sp--;} // * static void emit(void){printf("%c",stack[sp--]);} // , static void dot(void){printf("%d",stack[sp--]);} // . print digit static void div_(void){stack[sp-1]/=stack[sp]; sp--; } // / division static void reg_get(void){data[cur_reg]=stack[sp--]; k=0; } // : static void reg_set(void){stack[++sp]=data[cur_reg]; k=0; } // ; static void lessv1(void){stack[sp]=stack[sp]&gt;stack[sp-1]?-1:0; } // &lt; static void equalv1(void){stack[sp]=stack[sp]==stack[sp-1]?-1:0; } // = static void greaterv1(void){stack[sp]=stack[sp]&lt;stack[sp-1]?-1:0; } // &gt; static void less(void){stack[sp-1]=stack[sp]&gt;stack[sp-1]?-1:0; sp--;} // &lt; static void equal(void){stack[sp-1]=stack[sp]==stack[sp-1]?-1:0; sp--;} // = static void greater(void){stack[sp-1]=stack[sp]&lt;stack[sp-1]?-1:0; sp--;} // &gt; static void fetch(void){stack[++sp]=data[data[cur_reg]];} // ? static void drop(void){sp--;} /* \ */ static void key(void){int c=one_key();stack[++sp]=c;}// ^ read char, push on stack static void negate(void){stack[sp]=-stack[sp]; } // _ static void reg(void){k=1; cur_reg=token-97; } // a-z static void enddef(void){ip=rstack[rp--];} // } end of definition, return from subroutine static void or_(void){stack[sp-1]|=stack[sp]; sp--; } // | static void not_(void){ stack[sp]=~stack[sp]; } // ~ bitwise inverse (not_) static void add(void){ // + or_ increment register if(k) {data[cur_reg]++; k=0;} else {stack[sp-1]+=stack[sp]; sp--;} } static void sub(void){ // - sub or_ decrement register if(k) {data[cur_reg]--; k=0;} else {stack[sp-1]-=stack[sp]; sp--;} } static void digit(void){ /* build a number */ int i=0; do { i=i*10+token-'0'; } while((token=code[ip++])&amp;&amp;(isdigit(token))) ; stack[++sp]=i; ip--; } static void swap(void){ // $ int i=stack[sp]; stack[sp]=stack[sp-1]; stack[sp-1]=i; } static void over(void){ // @ int i=stack[sp-1]; stack[++sp]=i; } static void call(void){ // A..Z Call word rstack[++rp]=ip; // push current instruction pointer on return stack ip=fentry[token-65]; // jump to word in token } static void while_(void){ // [. top of stack will be droped on end of loop rstack[++rp]=ip; // push current instruction pointer, for looping // top of stack is false (0), so terminate loop char c; if(!stack[sp]) while((c=code[ip++]) &amp;&amp; (c!=']')) ; } static void endwhile(void){ // ] (code 93) if(stack[sp]) ip=rstack[rp]; // jump back to [ if top of stack is not_ zeor else rp--; // clean up return and_ data stack sp--; // drop datastack in any case } static void ext(void) { token=stack[sp--]; words[token+130](); } static void def(void){ // { define word char i=code[ip++]-65; fentry[i]=ip; // point to code where definition already remains char c; while((c=code[ip++]) &amp;&amp; (c!='}')); // skip forward to } (end of definition) } static void flt_dot(void) { /*.*/ printf("%f",*(float*)&amp;stack[sp--]); } static void flt_less(void) { int i=stack[sp--]; stack[sp]=(*(float*)&amp;i)&gt;*(float*)&amp;stack[sp]?-1:0; } static void flt_greater(void) { int i=stack[sp--]; stack[sp]=(*(float*)&amp;i)&lt;(*(float*)&amp;stack[sp])?-1:0; } static void flt_negate(void) { float f=(*(float*)&amp;stack[sp])*-1.0; stack[sp]=*(int*)&amp;f; } static void flt_mul(void) { float i=*(float*)&amp;stack[sp--]; *(float*)&amp;stack[sp]=(*(float*)&amp;stack[sp])*i; } static void flt_div(void) { float i=*(float*)&amp;stack[sp--]; *(float*)&amp;stack[sp]=(*(float*)&amp;stack[sp])/i; } static void flt_add(void) { // + or_ increment register if(k) {data[cur_reg]++; k=0;} else { float i=*(float*)&amp;stack[sp--]; *(float*)&amp;stack[sp]=(*(float*)&amp;stack[sp])+i; } } static void flt_sub(void) { // + or_ increment register if(k) {data[cur_reg]--;k=0;} else { float i=*(float*)&amp;stack[sp--]; *(float*)&amp;stack[sp]=(*(float*)&amp;stack[sp])-i; } } static void flt_digit(void) { /* build a number */ float val; char buffer[32], *p=buffer; int has_dot=0; int tos=0; do { if(token=='.') has_dot=1; *p++=token; } while((token=code[ip++])&amp;&amp;(isdigit(token)||(!has_dot&amp;&amp;(token=='.')))) ; *p=0; if(has_dot) { val=strtof(buffer, 0); tos=*(int*)&amp;val; } else { tos=strtol(buffer, 0, 0); } stack[++sp]=tos; ip--; } static void flt(void) { /*000*/ is_flt=1; words[46]=flt_dot; words[60]=flt_less; words[62]=flt_greater; words[95]=flt_negate; words[42]=flt_mul; words[47]=flt_div; words[43]=flt_add; words[45]=flt_sub; words[48]=flt_digit; words[49]=flt_digit; words[50]=flt_digit; words[51]=flt_digit; words[52]=flt_digit; words[53]=flt_digit; words[54]=flt_digit; words[55]=flt_digit; words[56]=flt_digit; words[57]=flt_digit; } static void unflt(void) {/*001*/ is_flt=0; words[46]=dot; words[60]=less; words[62]=greater; words[95]=negate; words[42]=mul; words[47]=div_; words[43]=add; words[45]=sub; words[48]=digit; words[49]=digit; words[50]=digit; words[51]=digit; words[52]=digit; words[53]=digit; words[54]=digit; words[55]=digit; words[56]=digit; words[57]=digit; } static void traceon(void) {is_trace=1;} static void traceoff(void) {is_trace=0;} static void edit(void) { char buffer[32]; snprintf(buffer, sizeof(buffer), "vi %d", stack[sp--]); system(buffer); } static void load(void) { char buffer[8]; cur_code_block=stack[sp--]; if(!cur_code_block) cur_code_block=99; snprintf(buffer, sizeof(buffer), "%d", cur_code_block); FILE *fp=fopen(buffer, "r"); if(!fp) { cur_code_block=99; fp=fopen("99", "r"); // 99 is edit block } if(fp) { code[fread(code, 1, sizeof(code)-1, fp)]=0; fclose(fp); memset(stack, 0, sizeof(stack)); memset(rstack, 0, sizeof(rstack)); memset(fentry, 0, sizeof(fentry)); sp=rp=-1; cur_reg=0; ip=0; traceoff(); unflt(); longjmp(do_halt,9); } else { printf("no block %s\n", buffer); } } static void version1(void) { /* &lt; 60 */ words['&lt;']=lessv1; /* = 61 */ words['=']=equalv1; /* &gt; 62 */ words['&gt;']=greaterv1; /* ` 96 */ words['`']=system_; } static void quit(void) {exit(stack[sp--]);} static void btod(void) {memcpy(data+2000, data+1000, sizeof(int)*1000);} static void dtob(void) {memcpy(data+1000, data+2000, sizeof(int)*1000);} static void dbg(void) { } void (*words[])(void)={ /* 0 1 2 3 4 5 6 7 8 9 */ /* 0*/ halt, noop, noop, noop, noop, noop, noop, noop, noop, noop, /* 10*/ noop, noop, noop, noop, noop, noop, noop, noop, noop, noop, /* 20*/ noop, noop, noop, noop, noop, noop, noop, noop, noop, noop, /* 30*/ noop, noop, noop, store, print, dup_, swap, modulo,and_, ext, /* 40*/ if_, noop, mul, add, emit, sub, dot, div_, digit, digit, /* 50*/ digit, digit, digit, digit, digit, digit, digit, digit, reg_get,reg_set, /* 60*/ less, equal, greater,fetch, over, call, call, call, call, call, /* 70*/ call, call, call, call, call, call, call, call, call, call, /* 80*/ call, call, call, call, call, call, call, call, call, call, /* 90*/ call, while_,drop, endwhile,key, negate, character,reg, reg, reg, /*100*/ reg, reg, reg, reg, reg, reg, reg, reg, reg, reg, /*110*/ reg, reg, reg, reg, reg, reg, reg, reg, reg, reg, /*120*/ reg, reg, reg, def, or_, enddef, not_, noop, noop, noop, /*130*/ flt, unflt, dbg, traceon, traceoff,version1,mstime, edit, load, block, /*140*/ trace, quit, blocknr,btod, dtob, noop, noop, noop, noop, noop, /*150*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, /*160*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, /*170*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, /*180*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, /*190*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, /*200*/ noop , noop, noop ,noop, noop, noop, noop, noop, noop, noop, }; static void trace_out_program(void) { if(!trace_fp) trace_fp=fopen("trace", "a"); int i=0; fprintf(trace_fp, "blk:%d\n", cur_code_block); fprintf(trace_fp, "code 0123456789 123456789 123456789 123456789"); char *p=code, c; while(c=*p++) { if(i--==0) { fprintf(trace_fp, "\n %4ld: ", p-code-1); i=39; } fputc(isprint(c)?c:' ', trace_fp); } fprintf(trace_fp, "\n\n"); } static char *tr_header=" ip:code:reg:value:mem :rp: rp0 rp1 rp2:sp: TOS sp1 sp2 sp3\n"; static void trace_header(void) { trace_out_program(); fputs(tr_header, trace_fp); } static void trace_start_data(void) { static int header; if(!header++) trace_header(); fprintf(trace_fp, "%3d:", ip-1); if(isprint(token)) { fprintf(trace_fp, " %c:", token); } else { fprintf(trace_fp, "\\%03d:",token); } } static void trace_data(void) { static int lines; int i=data[cur_reg]; fprintf(trace_fp, " %c:%5d:", cur_reg+'a', data[cur_reg]); if(i&gt;=0 &amp;&amp; i&lt;sizeof(data)/sizeof(int)) { fprintf(trace_fp, "%5d", data[i]); } fprintf(trace_fp, ":%2d:", (char)(rp+1)); fprintf(trace_fp, "%4d", rstack[rp]); fprintf(trace_fp, "%4d", rstack[rp-1]); fprintf(trace_fp, "%4d", rstack[rp-2]); fprintf(trace_fp, ":%2d:", (char)(sp+1)); char s=sp; if(is_flt) { fprintf(trace_fp, "%5d(%f)", stack[s], *(float*)&amp;stack[s]); s--; fprintf(trace_fp, "%5d(%f)", stack[s], *(float*)&amp;stack[s]); s--; fprintf(trace_fp, "%5d(%f)", stack[s], *(float*)&amp;stack[s]); s--; fprintf(trace_fp, "%5d(%f)", stack[s], *(float*)&amp;stack[s]); } else { fprintf(trace_fp, "%5d", stack[s]); s--; fprintf(trace_fp, "%5d", stack[s]); s--; fprintf(trace_fp, "%5d", stack[s]); s--; fprintf(trace_fp, "%5d", stack[s]); } fprintf(trace_fp,"\n"); if(lines++==20) { lines=0; fputc('\n', trace_fp); trace_header(); } } static void sigint_handler(int sig) { siglongjmp(do_sigint, 1); } static void trace(void) { printf("blk=%d:ip=%d:reg=%c:value=%d:mem=", cur_code_block,ip, cur_reg+'a',data[cur_reg]); int i=data[cur_reg]; if(i&gt;=0 &amp;&amp; i&lt;sizeof(data)/sizeof(int)) { printf("%d", data[i]); } printf(":rp=%d:%d %d %d:sp=%d:%d %d %d %d:\n", rp, rstack[rp],rstack[rp-1],rstack[rp-2], sp, stack[sp],stack[sp-1],stack[sp-2],stack[sp-3]); } int main(int argc,char *argv[]){ FILE *be=fopen(argv[1], "r"); if (!be) be=fopen("99", "r"); // command line if (!be) {printf("usage stable filename [args]\n"); return 1;} fread(code, 1, sizeof(code), be); // read program into memory fclose(be); int i; for(i=2;i&lt;argc;i++) data[i-2]=atoi(argv[i]); // set arguments into registers data_fp=fopen("stable.db", "r+"); if(!data_fp) data_fp=fopen("stable.db", "w+"); select_block(0); atexit(save_block); signal(SIGINT, sigint_handler); if(sigsetjmp(do_sigint, SIGINT)) { printf("\n\n** SIGINT **\n\n"); trace(); stack[sp]=99; load(); } if(setjmp(do_halt)==1) load(); for(;;) { // start the VM, until do_halt if(is_trace) { token=code[ip++]; trace_start_data(); words[token](); trace_data(); } else { token=code[ip++]; words[token](); } } return 0; } </code></pre> https://tedneward.github.io/Research/vms/typhon/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/typhon/index.html <p><a href="https://github.com/monte-language/typhon">Source</a></p> https://tedneward.github.io/Research/vms/scratch-vm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/scratch-vm/index.html <p><a href="https://github.com/LLK/scratch-vm">Source</a> | <a href="https://scratch.mit.edu/">Scratch</a></p> https://tedneward.github.io/Research/vms/som/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/som/index.html <p><a href="http://som-st.github.io/">Website</a> | <a href="https://github.com/SOM-st/">Github</a></p> <p>Nine implementations:<br> * <a href="https://github.com/SOM-st/som-java">SOM (Java)</a><br> * <a href="https://github.com/SOM-st/AweSOM">AweSOM (Smalltalk)</a><br> * <a href="https://github.com/SOM-st/CSOM">CSOM (C)</a><br> * <a href="https://github.com/SOM-st/SOMpp">SOM++ (C++)</a><br> * <a href="https://github.com/SOM-st/PySOM">PySOM (Python)</a><br> * <a href="https://github.com/SOM-st/RPySOM">RPySOM (Python)</a><br> * <a href="https://github.com/SOM-st/TruffleSOM">TruffleSOM (Java/Graal)</a><br> * <a href="https://github.com/SOM-st/RTruffleSOM">RTruffleSOM (Tree-based)</a><br> * <a href="https://github.com/SOM-st/JsSOM">JsSOM (JavaScript)</a></p> https://tedneward.github.io/Research/vms/strongtalk/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/strongtalk/index.html <p><a href="http://strongtalk.org/">Website</a></p> https://tedneward.github.io/Research/vms/uxn/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/uxn/index.html <p><a href="https://wiki.xxiivv.com/site/uxn.html">Website</a></p> <p>To get started, equip yourself with an assembler to convert tal source files into rom binary files, and an emulator to evaluate the rom files. The minimal system below includes the Console device:</p> <pre><code># Build the emulator cc uxnmin.c -o uxnmin # Build the assembler xxd -r -p uxnasm.rom.txt uxnasm.rom # Assemble a tal file cat opctest.tal | uxnmin uxnasm.rom &gt; opctest.rom # Run a rom file uxnmin opctest.rom </code></pre> https://tedneward.github.io/Research/vms/s4/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/s4/index.html <p><a href="https://github.com/CCurl/S4">Source</a></p> <h4>README</h4> <p>I think of S4 as a stack-based, RPN, Forth-like, virtual CPU/VM that can have as many registers, functions, and amount of user ram as the system supports.</p> <p>A register is identified by up to 3 UPPER-case characters, so there is a maximum or (26x26x26) = 17576 registers available. I tend to think of registers as built-in variables. Reading, setting, incrementing or decrementing a register is a single operation.</p> <p>A function is identified by any number of UPPER-case characters. The maximum number of functions is set in the config.h file.</p> <p>The number of registers, function vectors, and user memory can be scaled as desired to fit into a system of any size. For example, on an ESP-8266 board, a typical configuration might be 576 (26<em>26) registers, 5000 functions, and 24K of user ram. In such a system, register names would be in the range of [aa.zz]. For a RPI Pico, I use 576 registers, 1000 functions, and 128K USER RAM. On a Arduino Leonardo, you might configure the system to have 26 registers and functions, and 1K USER. On a Windows or Linux system, I use 17576 registers (26</em>26*26), 5000 functions, and 1MB USER.</p> <ul> <li>Example 1: "Hello World!" - the typical "hello world" program.</li> <li>Example 2: 1sA 2sB 3sC rA rB rC ++ . - would print 6.</li> <li>Example 4: 32 126[13,10,rI#."-",] - would print the ASCII table</li> <li>Example 3: 1000sS 13xPO 1{1 13 xPWD rS xW 0 13 xPWD rS xW} - the typical Arduino "blink" program.</li> </ul> <p>There were multiple reasons why to do this:</p> <ol> <li> <p>Many interpreted environments use tokens and a large SWITCH statement in a loop to execute the user's program. In these systems, the "machine code" (i.e. - byte-code ... the cases in the SWITCH statement) are often arbitrarily assigned and are not human-readable, so they have no meaning to the programmer when looking at the code that is actually being executed. Additionally there is a compiler and/or interpreter, often something similar to Forth, that is used to create the programs in that environment. For these enviromnents, there is a steep learning curve ... the programmer needs to learn the user environment and the hundreds or thousands of user functions in the libraries (or "words" in Forth). I wanted to avoid as much as that as possible, and have only one thing to learn: the machine code.</p> </li> <li> <p>I wanted to be free of the need for a multiple gigabyte tool chain and the edit/compile/run paradigm for developing everyday programs.</p> </li> <li> <p>I wanted a simple, minimal, and interactive programming environment that I could modify easily.</p> </li> <li> <p>I wanted an environment that could be easily configured for and deployed to many different types of development boards via the Arduino IDE.</p> </li> <li> <p>I wanted to be able to use the same environment on my personal computer as well as development boards.</p> </li> <li> <p>I wanted short commands so there was not a lot of typing needed.</p> </li> </ol> <p>S4 is the result of my work towards those goals.</p> <p>The implementation of S4</p> <ul> <li>The entire system is implemented in a few files. The engine is in S4.cpp.</li> <li>There are a few additional files to support optional functionality (e.g - WiFi and File access).</li> <li>The same code runs on Windows, Linux, and multiple development boards (via the Arduino IDE).</li> <li>See the file "config.h" for system configuration settings.</li> </ul> <h2>S4 Reference</h2> <h3>ARITHMETIC</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> + </td> <td align="left"> (a b--n) </td> <td align="left"> n: a+b - addition</td> </tr> <tr> <td align="left"> - </td> <td align="left"> (a b--n) </td> <td align="left"> n: a-b - subtraction</td> </tr> <tr> <td align="left"> * </td> <td align="left"> (a b--n) </td> <td align="left"> n: a*b - multiplication</td> </tr> <tr> <td align="left"> / </td> <td align="left"> (a b--q) </td> <td align="left"> q: a/b - division</td> </tr> <tr> <td align="left"> ^ </td> <td align="left"> (a b--r) </td> <td align="left"> r: MODULO(a, b)</td> </tr> <tr> <td align="left"> &amp; </td> <td align="left"> (a b--q r) </td> <td align="left"> q: DIV(a,b), r: MODULO(a,b) - /MOD</td> </tr> </tbody> </table> <h3>BIT MANIPULATION</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> b&amp; </td> <td align="left"> (a b--n) </td> <td align="left"> n: a AND b</td> </tr> <tr> <td align="left"> b| </td> <td align="left"> (a b--n) </td> <td align="left"> n: a OR b</td> </tr> <tr> <td align="left"> b^ </td> <td align="left"> (a b--n) </td> <td align="left"> n: a XOR b</td> </tr> <tr> <td align="left"> b~ </td> <td align="left"> (a--b) </td> <td align="left"> b: NOT a (ones-complement, e.g - 101011 =&gt; 010100)</td> </tr> </tbody> </table> <h3>STACK</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> # </td> <td align="left"> (a--a a) </td> <td align="left"> Duplicate TOS (DUP)</td> </tr> <tr> <td align="left"> \ </td> <td align="left"> (a b--a) </td> <td align="left"> Drop TOS (DROP)</td> </tr> <tr> <td align="left"> $ </td> <td align="left"> (a b--b a) </td> <td align="left"> Swap top 2 stack items (SWAP)</td> </tr> <tr> <td align="left"> % </td> <td align="left"> (a b--a b a) </td> <td align="left"> Push 2nd (OVER)</td> </tr> <tr> <td align="left"> _ </td> <td align="left"> (a--b) </td> <td align="left"> b: -a (Negate)</td> </tr> <tr> <td align="left"> xA </td> <td align="left"> (a--b) </td> <td align="left"> b: abs(a) (Absolute)</td> </tr> </tbody> </table> <h3>MEMORY</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> c@ </td> <td align="left"> (a--n) </td> <td align="left"> Fetch BYTE n from address a</td> </tr> <tr> <td align="left"> @ </td> <td align="left"> (a--n) </td> <td align="left"> Fetch CELL n from address a</td> </tr> <tr> <td align="left"> c! </td> <td align="left"> (n a--) </td> <td align="left"> Store BYTE n to address a</td> </tr> <tr> <td align="left"> ! </td> <td align="left"> (n a--) </td> <td align="left"> Store CELL n to address a</td> </tr> </tbody> </table> <h3>REGISTERS and LOCALS</h3> <p>NOTES:<br> - Register names are 1 to 3 UPPERCASE characters: [rA..rZZZ]<br> - LOCALS: S4 provides 10 locals per call [r0..r9].<br> - The number of registers is controlled by the NUM_REGS #define in "config.h".<br> - Register 'rI' is the FOR Loop index <strong>special</strong></p> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> rXXX </td> <td align="left"> (--n) </td> <td align="left"> read register/local XXX</td> </tr> <tr> <td align="left"> sXXX </td> <td align="left"> (n--) </td> <td align="left"> set register/local XXX to n</td> </tr> <tr> <td align="left"> iXXX </td> <td align="left"> (--) </td> <td align="left"> increment register/local XXX</td> </tr> <tr> <td align="left"> dXXX </td> <td align="left"> (--) </td> <td align="left"> decrement register/local XXX</td> </tr> <tr> <td align="left"> nXXX </td> <td align="left"> (--) </td> <td align="left"> increment register/local XXX by the size of a cell (next cell)</td> </tr> </tbody> </table> <h3>WORDS/FUNCTIONS</h3> <p>NOTES:<br> - Word/Function names are ProperCase identifiers.<br> - They must begin with [A..Z], and can include lowercase characters [a..z].<br> - The number of words is controlled by the NUM_FUNCS #define in "config.h"<br> - NUM_FUNCS needs to be a power of 2.<br> - If a word has already been defined, S4 prints "-redef-".</p> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> : </td> <td align="left"> (--) </td> <td align="left"> Define word/function. Copy chars to (HERE++) until closing ';'.</td> </tr> <tr> <td align="left"> ABC </td> <td align="left"> (--) </td> <td align="left"> Execute/call word/function ABC</td> </tr> <tr> <td align="left"> ; </td> <td align="left"> (--) </td> <td align="left"> Return: PC = rpop()</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - Returning while inside of a loop is not supported; break out of the loop first.</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - Use '|' to break out of a FOR or WHILE loop.</td> </tr> </tbody> </table> <h3>INPUT/OUTPUT</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> . </td> <td align="left"> (N--) </td> <td align="left"> Output N as decimal number.</td> </tr> <tr> <td align="left"> , </td> <td align="left"> (N--) </td> <td align="left"> Output N as character (Forth EMIT).</td> </tr> <tr> <td align="left"> " </td> <td align="left"> (?--?) </td> <td align="left"> Output characters until the next '"'.</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - %d outputs TOS as an integer (eg - 123"x%dx" outputs x123x)</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - %c outputs TOS as a character (eg - 65"x%cx" outputs xAx)</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - %n outputs CR/LF</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - %<x> output <x> (eg - "x%" %% %"x" outputs x" % "x) </x> </x></td> </tr> <tr> <td align="left"> 0..9 </td> <td align="left"> (--n) </td> <td align="left"> Scan DECIMAL number. For multiple numbers, separate them by space (47 33).</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - To enter a negative number, use "negate" (eg - 490_).</td> </tr> <tr> <td align="left"> hNNN </td> <td align="left"> (--h) </td> <td align="left"> h: NNN as a HEX number (0-9, A-F, UPPERCASE only).</td> </tr> <tr> <td align="left"> 'x </td> <td align="left"> (--n) </td> <td align="left"> n: the ASCII value of x</td> </tr> <tr> <td align="left"> <code>XXX</code> </td> <td align="left"> (a--a b) </td> <td align="left"> Copies XXX to address a, b is the next address after the NULL terminator.</td> </tr> <tr> <td align="left"> xZ </td> <td align="left"> (a--) </td> <td align="left"> Output the NULL terminated string starting at address a.</td> </tr> <tr> <td align="left"> xK? </td> <td align="left"> (--f) </td> <td align="left"> f: 1 if a character is waiting in the input buffer, else 0.</td> </tr> <tr> <td align="left"> xK@ </td> <td align="left"> (--c) </td> <td align="left"> c: next character from the input buffer. If no character, wait.</td> </tr> </tbody> </table> <h3>CONDITIONS/LOOPS/FLOW CONTROL</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> &lt; </td> <td align="left"> (a b--f) </td> <td align="left"> f: (a &lt; b) ? 1 : 0;</td> </tr> <tr> <td align="left"> = </td> <td align="left"> (a b--f) </td> <td align="left"> f: (a = b) ? 1 : 0;</td> </tr> <tr> <td align="left"> &gt; </td> <td align="left"> (a b--f) </td> <td align="left"> f: (a &gt; b) ? 1 : 0;</td> </tr> <tr> <td align="left"> ~ </td> <td align="left"> (n -- f) </td> <td align="left"> f: (a = 0) ? 1 : 0; (Logical NOT)</td> </tr> <tr> <td align="left"> ( </td> <td align="left"> (f--) </td> <td align="left"> IF: if (f != 0), continue into '()', else skip to matching ')'</td> </tr> <tr> <td align="left"> [ </td> <td align="left"> (F T--) </td> <td align="left"> FOR: start a For/Next loop. if (T &lt; F), swap T and F</td> </tr> <tr> <td align="left"> rI </td> <td align="left"> (--n) </td> <td align="left"> n: the index of the current FOR loop</td> </tr> <tr> <td align="left"> ] </td> <td align="left"> (--) </td> <td align="left"> NEXT: increment index (rI) and restart loop if (rI &lt;= T)</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> NOTE: A FOR loop always runs at least one iteration.</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> - It can be put it inside a '()' to keep it from running.</td> </tr> <tr> <td align="left"> { </td> <td align="left"> (f--f) </td> <td align="left"> WHILE: if (f == 0) skip to matching '}'</td> </tr> <tr> <td align="left"> } </td> <td align="left"> (f--f?) </td> <td align="left"> WHILE: if (f != 0) jump to matching '{', else drop f and continue</td> </tr> <tr> <td align="left"> uL </td> <td align="left"> (--) </td> <td align="left"> Unwind the loop stack. Use with ';'.eg = (uL;)</td> </tr> <tr> <td align="left"> uF </td> <td align="left"> (--) </td> <td align="left"> Exit FOR. Continue after the next ']'.</td> </tr> <tr> <td align="left"> uW </td> <td align="left"> (--) </td> <td align="left"> Exit WHILE. Continue after the next '}'.</td> </tr> <tr> <td align="left"> uC </td> <td align="left"> (--) </td> <td align="left"> Continue. Jump to the beginning of the current loop.</td> </tr> </tbody> </table> <h3>OTHER</h3> <table> <thead> <tr> <th align="left"> Opcode</th> <th align="left"> Stack </th> <th align="left"> Description </th> </tr> </thead> <tbody> <tr> <td align="left"> xBO </td> <td align="left"> (n m--fh) </td> <td align="left"> File: Block Open (block-nnn.s4, m: 0=&gt;read, 1=write)</td> </tr> <tr> <td align="left"> xBR </td> <td align="left"> (n a sz--) </td> <td align="left"> File: Block Read (block-nnn.s4, max sz bytee).</td> </tr> <tr> <td align="left"> xBW </td> <td align="left"> (n a sz--) </td> <td align="left"> File: Block Write (block-nnn.s4, sz bytes).</td> </tr> <tr> <td align="left"> xBL </td> <td align="left"> (n--) </td> <td align="left"> File: Load code from file (block-nnn.src). This can be nested.</td> </tr> <tr> <td align="left"> xFL </td> <td align="left"> (--) </td> <td align="left"> File: Load code from ./Code.S4.</td> </tr> <tr> <td align="left"> xFS </td> <td align="left"> (--) </td> <td align="left"> File: Save code to ./Code.S4.</td> </tr> <tr> <td align="left"> xFO </td> <td align="left"> (n m--h) </td> <td align="left"> File: Open - n: file name, m: mode, h: file handle (0 means not opened)</td> </tr> <tr> <td align="left"> xFC </td> <td align="left"> (h--) </td> <td align="left"> File: Close - h: file handle</td> </tr> <tr> <td align="left"> xFD </td> <td align="left"> (n--) </td> <td align="left"> File: Delete - n: file name</td> </tr> <tr> <td align="left"> xFR </td> <td align="left"> (h--c f) </td> <td align="left"> File: Read - h: file handle, c: char, n: success?</td> </tr> <tr> <td align="left"> xFW </td> <td align="left"> (c h--f) </td> <td align="left"> File: Write - h: file handle, c: char, n: success?</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> NOTE: File operations are enabled by #define <strong>FILES</strong></td> </tr> <tr> <td align="left"> xPI </td> <td align="left"> (p--) </td> <td align="left"> Arduino: Pin Input (pinMode(p, INPUT))</td> </tr> <tr> <td align="left"> xPU </td> <td align="left"> (p--) </td> <td align="left"> Arduino: Pin Pullup (pinMode(p, INPUT_PULLUP))</td> </tr> <tr> <td align="left"> xPO </td> <td align="left"> (p--) </td> <td align="left"> Arduino: Pin Output (pinMode(p, OUTPUT)</td> </tr> <tr> <td align="left"> xPRA </td> <td align="left"> (p--n) </td> <td align="left"> Arduino: Pin Read Analog (n = analogRead(p))</td> </tr> <tr> <td align="left"> xPRD </td> <td align="left"> (p--n) </td> <td align="left"> Arduino: Pin Read Digital (n = digitalRead(p))</td> </tr> <tr> <td align="left"> xPWA </td> <td align="left"> (n p--) </td> <td align="left"> Arduino: Pin Write Analog (analogWrite(p, n))</td> </tr> <tr> <td align="left"> xPWD </td> <td align="left"> (n p--) </td> <td align="left"> Arduino: Pin Write Digital (digitalWrite(p, n))</td> </tr> <tr> <td align="left"> xR </td> <td align="left"> (n--r) </td> <td align="left"> r: a random number in the range [0..(n-1)]</td> </tr> <tr> <td align="left"> </td> <td align="left"> </td> <td align="left"> NB: if n=0, r is the entire 32-bit random number</td> </tr> <tr> <td align="left"> xT </td> <td align="left"> (--n) </td> <td align="left"> Milliseconds (Arduino: millis(), Windows: GetTickCount())</td> </tr> <tr> <td align="left"> xN </td> <td align="left"> (--n) </td> <td align="left"> Microseconds (Arduino: micros(), Windows: N/A)</td> </tr> <tr> <td align="left"> xW </td> <td align="left"> (n--) </td> <td align="left"> Wait (Arduino: delay(), Windows: Sleep())</td> </tr> <tr> <td align="left"> xIAF </td> <td align="left"> (--a) </td> <td align="left"> INFO: a: address of first function vector</td> </tr> <tr> <td align="left"> xIAH </td> <td align="left"> (--a) </td> <td align="left"> INFO: a: address of HERE variable</td> </tr> <tr> <td align="left"> xIAR </td> <td align="left"> (--a) </td> <td align="left"> INFO: a: address of first register vector</td> </tr> <tr> <td align="left"> xIAS </td> <td align="left"> (--a) </td> <td align="left"> INFO: a: address of beeginning of system structure</td> </tr> <tr> <td align="left"> xIAU </td> <td align="left"> (--a) </td> <td align="left"> INFO: a: address of beeginning of user area</td> </tr> <tr> <td align="left"> xIF </td> <td align="left"> (--n) </td> <td align="left"> INFO: n: number of words/functions</td> </tr> <tr> <td align="left"> xIH </td> <td align="left"> (--n) </td> <td align="left"> INFO: n: value of HERE variable</td> </tr> <tr> <td align="left"> xIR </td> <td align="left"> (--n) </td> <td align="left"> INFO: n: number of registers</td> </tr> <tr> <td align="left"> xIU </td> <td align="left"> (--n) </td> <td align="left"> INFO: n: number of bytes in the USER area</td> </tr> <tr> <td align="left"> xSR </td> <td align="left"> (--) </td> <td align="left"> S4 system reset</td> </tr> <tr> <td align="left"> xQ </td> <td align="left"> (--) </td> <td align="left"> PC: Exit S4</td> </tr> </tbody> </table> <h1>Examples</h1> <p>; To enter a comment:<br> 0( here is a comment )<br> ; here is another comment</p> <p>; if (c) { print("Yes") } else { print("No") }<br> rC #("Yes")~("No")</p> <p>; x = (a == b) ? c : d;<br> rA rB=#(rC$)~(rD)sX;</p> <p>; To make sure F &lt; T<br> ; S4 code: %%&gt;($)<br> ; Forth equivalent: OVER OVER &gt; IF SWAP THEN<br> ; C equivalent: if (f &gt; t) { int x = f; f = t; t = x; }</p> <p>; To do something (in this case, execute LP) N times:<br> 1 rN[LP]</p> <p>; Increment Register x, decrement register Y<br> iX dY</p> <p>; To print numbers from F to T:<br> ; S4 code: rF rT[rI." "]<br> ; Forth equivalent: F @ T @ FOR I . NEXT<br> ; C equivalent: for (int i = F; i &lt;= T; i)) { printf("%d ", i); }</p> <p>; One way to copy N bytes from A to B (n f t--)<br> N A B s2 s1 1[r1 c@ r2 c! i1 i2]</p> <p>; One way to copy N CELLS from A to B (N A B--)<br> N A B s2 s1 1[r1 @ r2 ! n1 n2]</p> <p>; A simple benchmark for a 100 million FOR loop:<br> 1000#* 100* xT$ 1[] xT$-." ms"</p> <p>; A simple benchmark for a 100 million WHILE loop:<br> 1000#* 100* xT$ {1-} xT$-." ms"</p> <p>; Define a word to display the currently defined code:<br> :CODE xIAU xIH 1-[rI c@ #,';=(rI 1+ c@': =(13,10,))];</p> <p>; Exit S4:<br> xQ</p> <h1>Adding new functionality to S4:</h1> <p>In run(start) (S4.cpp), in the "switch" statement, there are cases available for direct support of new instructions, mostly unused lowercase characters. New functionality can be put there by comandeering one of the /* FREE */ cases.</p> <p>S4 also has "extended" instructions. These are triggered by the 'x' case. All extended instructions begin with 'x'. For extended instructions, function doExt() is called. It behaves in a similar way to run(), by setting 'ir = *(pc++)' and a switch statement. The "default:" case calls an external function, doCustom(ir, pc). That is where I usually put system-specific functionality; (eg - pin manipulation for Arduino Boards). Function doCustom(ir, pc) needs to return the address where pc should continue.</p> <p>As an example, I will add Gamepad/Joystick simulation to S4 as extended instructions. This example uses the HID-Project from NicoHood (<a href="https://github.com/NicoHood/HID">https://github.com/NicoHood/HID</a>).</p> <p>Import the library into Arduino using "Import Library".</p> <p>In the doCustom(ir, pc) function (S4.ino), add a new case for ir. In this case, I am adding case 'G'.</p> <pre><code>addr doCustom(byte ir, addr pc) { ir = *(pc++); switch (ir) { ... case 'G': pc = doGamePad(ir, pc); break; ... } return pc; } </code></pre> <p>Then it is a simple matter of implementing doGamePad(ir, pc):</p> <pre><code>\#ifdef __GAMEPAD__ \#include &lt;HID-Project.h&gt; \#include &lt;HID-Settings.h&gt; addr doGamePad(byte ir, addr pc) { ir = *(pc++); switch (ir) { case 'X': Gamepad.xAxis(pop()); break; case 'Y': Gamepad.yAxis(pop()); break; case 'P': Gamepad.press(pop()); break; case 'R': Gamepad.release(pop()); break; case 'A': Gamepad.dPad1(pop()); break; case 'B': Gamepad.dPad2(pop()); break; case 'L': Gamepad.releaseAll(); break; case 'W': Gamepad.write(); break; default: isError = 1; printString("-notGamepad-"); } return pc; } \#else addr doGamePad(addr pc) { printString("-noGamepad-"); return pc; } \#endif </code></pre> <p>The last thing to do is #define __GAMEPAD__. This is best done in "config.h".</p> <pre><code>... #elif __BOARD__ == XIAO #define __GAMEPAD__ #define STK_SZ 8 #define LSTACK_SZ 4 #define USER_SZ (22*1024) #define NUM_REGS (26*26) ... </code></pre> <p>Now in S4, you can do things like:</p> <p>3 xGP xGW 0(Press button 3)<br> 3 xGR xGW 0(Release button 3)</p> https://tedneward.github.io/Research/vms/q3vm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/q3vm/index.html <p><a href="https://github.com/jnz/q3vm">GitHub</a></p> <p>A lightweight (single file: vm.c) embeddable interpreter/Virtual Machine (VM) for compiled bytecode files (.qvm) based on good old C-language input (.c). A complete C compiler to generate .qvm files is included (LCC). The interpreter is based on the Quake III Arena virtual machine (hence the name q3vm) but the interpreter is not tied to Quake III Arena and can be used for any kind of project. For example code that needs to run in a sandbox.</p> <p>Introduction to Q3VM: <a href="http://fabiensanglard.net/quake3/qvm.php">http://fabiensanglard.net/quake3/qvm.php</a></p> https://tedneward.github.io/Research/vms/reading/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/reading/index.html <h2>Posts, Papers</h2> <ul> <li><a href="https://github.com/spencertipping/jit-tutorial">How to write a very simple JIT compiler</a></li> <li><a href="http://ittc.ku.edu/~kulkarni/teaching/EECS768/slides/chapter1.pdf">Introduction to Virtual Machines</a></li> <li><a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.186.4512&amp;rep=rep1&amp;type=pdf">An Introduction to Virtual Machines Implementation and Applications</a></li> <li>Crafting Interpreters: <a href="https://www.craftinginterpreters.com/a-virtual-machine.html">A Virtual Machine</a></li> <li><a href="https://www.jmeiners.com/lc3-vm/">Write your Own Virtual Machine</a> <a href="https://github.com/justinmeiners/lc3-vm">Source</a></li> <li><a href="http://merlintec.com/download/mario.pdf">"A Smalltalk Implementation in Self"</a></li> <li><a href="https://www.andreinc.net/2021/12/01/writing-a-simple-vm-in-less-than-125-lines-of-c">Writing a simple 16-bit VM in less than 125 lines of C</a> (<a href="https://github.com/nomemory/lc3-vm">Github</a>)</li> <li><a href="https://en.wikibooks.org/wiki/Creating_a_Virtual_Machine/Register_VM_in_C">Creating a Virtual Machine/Register VM in C</a></li> <li><a href="https://www.lua.org/doc/jucs05.pdf">The Implementation of Lua 5.0</a></li> <li><a href="https://www.gnu.org/software/guile/manual/html_node/A-Virtual-Machine-for-Guile.html">A Virtual Machine for Guile</a></li> <li><a href="https://www.cs.cornell.edu/people/egs/papers/kimera-sosp99.pdf">Design and implementation of a distributed virtual machine for networked computers</a></li> <li><a href="./VirtualMachinesAndAbstractCompilers.pdf">"Virtual Machines and Abstract Compilers"</a></li> <li><a href="./TitzerVEE10.pdf">"Improving Compiler-Runtime Separation with XIR"</a></li> <li><a href="./2204.06156.pdf">"Modular and Didactic Compiler Design with XML Inter-Phases Communication"</a></li> <li><a href="http://stevehanov.ca/blog/index.php?id=92">qb.js: An implementation of QBASIC in Javascript</a></li> </ul> <h2>Videos</h2> <p>Build Your Own Virtual Machine</p> <ol> <li><a href="https://www.youtube.com/watch?v=BNXP0w4Ppto&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=4">How to Build a Virtual Machine from Scratch</a></li> <li><a href="https://www.youtube.com/watch?v=Dw9URS-s9M8&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=3&amp;t=1s">Create Your Own Stack Machine</a></li> <li><a href="https://www.youtube.com/watch?v=UnGYvjDDPJI&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=2&amp;t=1s">Build Your Own Virtual Machine - Lexer</a></li> <li><a href="https://www.youtube.com/watch?v=A5Ioln_8Wbc&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=1&amp;t=1s">Build Your Own Virtual Machine - Assembler</a></li> <li><a href="https://www.youtube.com/watch?v=9Pa0iDObijA&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=5">Build Your Own Virtual Machine - Finish Assembler</a></li> <li><a href="https://www.youtube.com/watch?v=pPFP9yA1MF8&amp;list=PLSiFUSQSRYAOFwfP-aMzXJlWKVyIuWfPU&amp;index=6&amp;t=1s">Build Your Own Virtual Machine - Finish Up!</a></li> </ol> <p><a href="http://dmitrysoshnikov.com/courses/virtual-machine/">Building a Virtual Machine for Programming Language</a> - Commercial; <a href="https://www.youtube.com/playlist?list=PLGNbPb3dQJ_446PjTYQ0mCn2OGoHSKraB">first 5 parts</a></p> <h2>College/University Courses</h2> <ul> <li><a href="http://www.wolczko.com/CS294/">UCB CS294-113</a>: Virtual Machines and Managed Runtimes <ul> <li><a href="https://archive.org/download/vmss16/wolczko.pdf">A Concise and Opinionated History of Virtual Machines</a> or <a href="https://www.youtube.com/watch?v=QnQYhrpX39M">YouTube</a> <ul> <li><a href="https://soft-dev.org/events/vmss16/">Virtual Machines Summer School 2016</a></li> </ul> </li> </ul> </li> <li><a href="http://course.ece.cmu.edu/~ece600/fall17/lectures/lecture19.pdf">18-600: Fondations of Computer Systems/Lecture 19: Virtual Machine Design &amp; Implementation</a></li> <li><a href="https://ycpcs.github.io/cs340-fall2016/lectures/lecture19.html">CS 340: Lecture 19: Virtual Machines</a></li> <li><a href="http://www.wolczko.com/CS294/">UCB CS294-113: Virtual Machines and Managed Runtimes</a></li> </ul> https://tedneward.github.io/Research/vms/rubinius/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/rubinius/index.html <p><a href="https://rubinius.com/">Website</a> | <a href="https://github.com/rubinius/rubinius">Github</a></p> <p>From the Github Overview:</p> <blockquote> <p>Rubinius is a virtual computing machine. As a computing machine, it possesses several mechanisms that physical computers possess to support computation. It runs in an environment that provides various capabilities. These are presented to the machine through the Environment object. It has certain configuration values that influence how the machine operates. This is managed by the Configuration object. The machine has a boot process that carefully builds up the components in a precise order until the machine is ready to run user code. When the user code completes executing, the machine halts, again following a precise sequence of deconstructing the various components. The rest of the components of the machine are described below.</p> </blockquote> https://tedneward.github.io/Research/vms/python/cpython/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/cpython/index.html <p><a href="https://github.com/python/cpython">Source</a></p> <h2>Books</h2> <ul> <li><a href="https://leanpub.com/insidethepythonvirtualmachine/read">Inside the Python Virtual Machine</a></li> </ul> <h2>Posts</h2> <p><a href="https://opensource.com/article/18/4/introduction-python-bytecode">An introduction to Python bytecode</a></p> <p>Python behind the scenes:<br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-1-how-the-cpython-vm-works/">#1: how the CPython VM works</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-2-how-the-cpython-compiler-works/">#2: how the CPython compiler works</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-3-stepping-through-the-cpython-source-code/">#3: stepping through the CPython source code</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-4-how-python-bytecode-is-executed/">#4: how CPython bytecode is executed</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-5-how-variables-are-implemented-in-cpython/">#5: how variables are implemented in CPython</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-6-how-python-object-system-works/">#6: how Python object system works</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-7-how-python-attributes-work/">#7: how Python attributes work</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-8-how-python-integers-work/">#8: how Python integers work</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-9-how-python-strings-work/">#9: how Python strings work</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-10-how-python-dictionaries-work/">#10: how Python dictionaries work</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-11-how-the-python-import-system-works/">#11: how the Python import system works</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-12-how-asyncawait-works-in-python/">#12: how async/await works</a><br> * <a href="https://tenthousandmeters.com/blog/python-behind-the-scenes-13-the-gil-and-its-effects-on-python-multithreading/">#13: the GIL and its effects on Python multithreading</a></p> <h2>Slides</h2> <ul> <li><a href="https://troeger.eu/files/teaching/pythonvm08.pdf">Python 2.5: A Guided Tour (PDF)</a></li> </ul> https://tedneward.github.io/Research/vms/python/pypy/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/pypy/index.html <p><a href="https://www.pypy.org/">Website</a> | <a href="https://www.pypy.org/download.html">Downloads (Binaries and Source zips)</a></p> https://tedneward.github.io/Research/vms/python/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/index/index.html <h2>Implementations</h2> <ul> <li>C/Native: <a href="../cpython">CPython</a></li> <li>Java: <a href="../jython">Jython</a></li> <li>Python: <a href="../coco">Coco</a></li> <li><a href="https://aosabook.org/en/500L/a-python-interpreter-written-in-python.html">A Python in under 500 lines of Python</a> (<a href="https://github.com/nedbat/byterun">Source</a>)</li> <li><a href="../medusavm">MedusaVM</a></li> <li><a href="../pypy">PyPy</a></li> </ul> https://tedneward.github.io/Research/vms/python/jython/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/jython/index.html <p><a href="https://www.jython.org/">Website</a> | <a href="https://github.com/jython/jython/">Source</a></p> https://tedneward.github.io/Research/vms/python/medusavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/medusavm/index.html <p><a href="https://github.com/lispyclouds/medusa">Source</a></p> <p>Archived in 2015.</p> https://tedneward.github.io/Research/vms/js/spidermonkey/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/spidermonkey/index.html <p><a href="https://spidermonkey.dev/">Website</a> | <a href="https://github.com/mozilla-spidermonkey">Source</a></p> https://tedneward.github.io/Research/vms/jvm/ffi/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/ffi/index.html <ul> <li><a href="https://github.com/zakgof/java-native-benchmark">java-native-benchmark</a>: Benchmarking Java's native call APIs: JNI, JNA, JNR, BridJ and Project Panama</li> </ul> <h1>JNI (Java Native Interface)</h1> <p><a href="https://docs.oracle.com/en/java/javase/17/docs/specs/jni/index.html">Specification (Java 17)</a></p> <p>Basically broken into three parts:</p> <ul> <li>Java calling native code (static methods marked <code>native</code> and dynamically linked at runtime)</li> <li>Native code calling back into the JVM (native code accessing fields and methods of Java objects)</li> <li>Invocation/hosting (bringing the JVM up inside a native process--which is what the <code>java</code> host does)</li> </ul> <h3>Helpers</h3> <ul> <li><a href="https://github.com/mapbox/jni.hpp">jni.hpp</a>: a modern, type-safe, header-only, C++14 wrapper for JNI</li> <li><a href="https://github.com/bytedeco/javacpp">JavaCPP</a>: The missing bridge between Java and native C++</li> <li><a href="https://github.com/SwiftAndroid/swift-jni">swift-jni</a>: Swift wrappers around JNI</li> </ul> <h1>JNA</h1> <p><a href="https://github.com/java-native-access/jna">Source</a></p> <h1>JNR (jnr-ffi): Java Abstracted Foreign Function Layer</h1> <p>Java library for loading native libraries without writing JNI code by hand or using tools such as SWIG. JNR is a comparingly young project that target the same problem. Similarly as JNA or Bridj it does not require native programming. There's not much documentation or reviews at the moment, but JNR is often called promising.</p> <p><a href="https://github.com/jnr/jnr-ffi">Source</a></p> <h1>BridJ</h1> <p><a href="https://github.com/nativelibs4java/BridJ">Source</a></p> <h1>Project Panama</h1> <p><a href="http://openjdk.java.net/projects/panama/">Website (OpenJDK)</a></p> <ul> <li>Panama: A Foreign Policy for Java <ul> <li>Devoxx 2018; Maurizio Cimadamore</li> <li><a href="https://www.youtube.com/watch?v=cfxBrYud9KM">https://www.youtube.com/watch?v=cfxBrYud9KM</a></li> </ul> </li> <li>Native Interfaces: The Phantom Menace <ul> <li>NYJavaSIG, February 7, 2018</li> <li>Tony Printezis - JVM Engineer - Twitter</li> <li>JNI, Project Panama</li> <li><a href="https://www.eventbrite.com/e/nyjavasig-native-interfaces-the-phantom-menace-and-java-9-compatibility-tickets-42519931259#">https://www.eventbrite.com/e/nyjavasig-native-interfaces-the-phantom-menace-and-java-9-compatibility-tickets-42519931259#</a></li> <li><a href="https://www.youtube.com/watch?v=YxHb6kAGZ10">https://www.youtube.com/watch?v=YxHb6kAGZ10</a></li> </ul> </li> </ul> <h2>GraalVM</h2> <p>Supports native execution of LLVM bitcode using <a href="https://github.com/graalvm/sulong">Sulong</a>. This enables interop to C/C++, Fortran, and other languages that can be compiled to LLVM bitcode.</p> <h1>Other FFI</h1> https://tedneward.github.io/Research/vms/jvm/jikes/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/jikes/index.html <p><a href="https://www.jikesrvm.org/">Website</a> | <a href="https://github.com/JikesRVM/JikesRVM">Source</a></p> <p>No updates since 2016.</p> https://tedneward.github.io/Research/vms/jvm/nanovm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/nanovm/index.html <p><a href="http://www.harbaum.org/till/nanovm/index.shtml">Website</a> | <a href="https://github.com/harbaum/NanoVM">Source</a></p> <p>No updates since 2015 or so.</p> https://tedneward.github.io/Research/vms/jvm/waba/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/waba/index.html <p><a href="https://wabasoft.com/">Website</a> | <a href="http://waba.sourceforge.net/">Source</a></p> <p>Looks to not have been updated since 2001. <a href="https://sourceforge.net/projects/waba/files/waba/20010521/">This</a> looks to be the latest source drop. Might be interesting for pedagogical purposes.</p> https://tedneward.github.io/Research/vms/minivm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/minivm/index.html <p><a href="https://github.com/FastVM/minivm">Source</a></p> <p>The most common way to get code running on MiniVM is to use <a href="/languages/paka">Paka</a>.</p> https://tedneward.github.io/Research/vms/nanovm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/nanovm/index.html <p><a href="http://www.harbaum.org/till/nanovm/index.shtml">Website</a></p> <p>No updates since 2005.</p> https://tedneward.github.io/Research/vms/parrot/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/parrot/index.html <p><a href="http://parrot.org/">Website</a> | <a href="https://github.com/parrot/parrot">Github</a></p> <p>Parrot work appears to have stopped in 2014.</p> https://tedneward.github.io/Research/vms/js/jscore/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/jscore/index.html <p><a href="https://docs.webkit.org/Deep%20Dive/JSC/JavaScriptCore.html">Website</a></p> https://tedneward.github.io/Research/vms/js/v8/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/v8/index.html <p>The engine that drives NodeJS. Embeddable.</p> <p><a href="https://github.com/danbev/learning-v8">Learning V8 project</a></p> <p><a href="https://v8.dev/blog">V8 Dev Blog</a>:</p> <ul> <li><a href="/libraries/oilpan">Oilpan</a> is a standalone garbage collection library for C++ hosted by the V8 project.</li> <li>WebAssembly topics: <ul> <li><a href="https://v8.dev/blog/4gb-wasm-memory">Up to 4GB of memory</a>: Thanks to recent work in Chrome and Emscripten, you can now use up to 4GB of memory in WebAssembly applications. That’s up from the previous limit of 2GB.</li> <li><a href="https://v8.dev/blog/wasm-decompile">Introducing <code>wasm-decompile</code></a>: wasm-decompile produces output that tries to look like a "very average programming language" while still staying close to the Wasm it represents.</li> <li><a href="https://v8.dev/blog/emscripten-standalone-wasm">Standalone WebAssembly binaries using Emscripten</a>: Emscripten has always focused first and foremost on compiling to the Web and other JavaScript environments like Node.js. But as WebAssembly starts to be used without JavaScript, new use cases are appearing, and so we've been working on support for emitting standalone Wasm files from Emscripten, that do not depend on the Emscripten JS runtime!</li> <li><a href="https://v8.dev/blog/emscripten-llvm-wasm">Emscripten and the LLVM WebAssembly backend</a>: WebAssembly is normally compiled from a source language, which means that developers need tools to use it. Because of that, the V8 team works on relevant open-source projects like LLVM, Emscripten, Binaryen, and WABT. This post describes some of the work we’ve been doing on Emscripten and LLVM, which will soon allow Emscripten to switch to the LLVM WebAssembly backend by default — please test it and report any issues!</li> <li><a href="https://v8.dev/blog/wasm-code-caching">Code caching for WebAssembly developers</a>: There’s a saying among developers that the fastest code is code that doesn’t run. Likewise, the fastest compiling code is code that doesn’t have to be compiled. WebAssembly code caching is a new optimization in Chrome and V8 that tries to avoid code compilation by caching the native code produced by the compiler. We’ve written about how Chrome and V8 cache JavaScript code in the past, and best practices for taking advantage of this optimization. In this blog post, we describe the operation of Chrome’s WebAssembly code cache and how developers can take advantage of it to speed up loading for applications with large WebAssembly modules.</li> </ul> </li> <li><a href="https://v8.dev/blog/sparkplug">Sparkplug</a>: a non-optimizing JavaScript compiler</li> <li><a href="https://v8.dev/blog/short-builtin-calls">Short builtin calls</a>: we’ve realized that function calls between embedded builtins and JIT compiled code can come at a considerable performance penalty. This cost depends on the microarchitecture of the CPU. In this post we’ll explain why this is happening, what the performance looks like, and what we’re planning to do to resolve this long-term.</li> <li><a href="https://v8.dev/blog/fast-super">Super fast property access</a>: The super keyword can be used for accessing properties and functions on an object’s parent.</li> <li><a href="https://v8.dev/blog/adaptor-frame">Faster JavaScript calls</a>: JavaScript allows calling a function with a different number of arguments than the expected number of parameters, i.e., one can pass fewer or more arguments than the declared formal parameters. The former case is called under-application and the latter is called over-application.</li> <li><a href="https://v8.dev/blog/non-backtracking-regexp">An additional non-backtracking RegExp engine</a>: V8 ships with a new experimental non-backtracking RegExp engine (in addition to the existing Irregexp engine) which guarantees execution in linear time with respect to the size of the subject string.</li> <li><a href="https://v8.dev/blog/system-analyzer">Indicium</a>: This system analyzer is a unified web interface to trace, debug and analyse patterns of how Inline Caches (ICs) and Maps are created and modified in real-world applications. V8 already has a tracing infrastructure for ICs and Maps which can process and analyse IC events using the IC Explorer and Map events using Map Processor. However, previous tools didn't allow us to analyze maps and ICs holistically and this is now possible with system analyzer.</li> <li><a href="https://v8.dev/blog/slack-tracking">Slack tracking</a>: Slack tracking is a way to give new objects an initial size that is larger than what they may actually use, so they can have new properties added quickly. And then, after some period of time, to magically return that unused space to the system. Neat, huh?</li> <li><a href="https://v8.dev/blog/high-performance-cpp-gc">High-performance garbage collection for C++</a>: Because the C++ object graph around the DOM is heavily tangled with Javascript objects, the Chromium team switched a couple of years ago to a garbage collector, called Oilpan, for managing this kind of memory. Oilpan is a garbage collector written in C++ for managing C++ memory that can be connected to V8 using cross-component tracing that treats the tangled C++/JavaScript object graph as one heap.</li> <li>Understanding the ECMAScript spec: <a href="https://v8.dev/blog/understanding-ecmascript-part-1">Part 1</a> | <a href="https://v8.dev/blog/understanding-ecmascript-part-2">Part 2</a> | <a href="https://v8.dev/blog/understanding-ecmascript-part-3">Part 3</a> | <a href="https://v8.dev/blog/understanding-ecmascript-part-4">Part 4</a></li> <li>Blazingly-fast parsing: <a href="https://v8.dev/blog/scanner">Part 1: the scanner</a> | <a href="https://v8.dev/blog/preparser">Part 2: lazy parsing</a></li> <li><a href="https://v8.dev/blog/pointer-compression">Pointer compression in V8</a>: There is a constant battle between memory and performance. As users, we would like things to be fast as well as consume as little memory as possible. Unfortunately, usually improving performance comes at a cost of memory consumption (and vice versa).</li> <li><a href="https://v8.dev/blog/regexp-tier-up">Improving V8 regular expressions</a>: In its default configuration, V8 compiles regular expressions to native code upon the first execution. As part of our work on JIT-less V8, we introduced an interpreter for regular expressions. Interpreting regular expressions has the advantage of using less memory, but it comes with a performance penalty. In this blog post we describe how we take advantage of the upsides of interpreting regular expressions while mitigating the downsides.</li> <li><a href="https://v8.dev/blog/v8-lite">A lighter V8</a>: In late 2018 we started a project called V8 Lite, aimed at dramatically reducing V8’s memory usage. Initially this project was envisioned as a separate Lite mode of V8 specifically aimed at low-memory mobile devices or embedder use-cases that care more about reduced memory usage than throughput execution speed. However, in the process of this work, we realized that many of the memory optimizations we had made for this Lite mode could be brought over to regular V8 thereby benefiting all users of V8.</li> <li><a href="https://v8.dev/blog/code-caching-for-devs">Code caching for Javascript developers</a>: In this blog post, we offer a few pieces of advice for JS developers who want to make the best use of code caching to improve the startup of their websites. This advice focuses on the implementation of caching in Chrome/V8, but most of it is likely transferable to other browsers’ code caching implementations too.</li> <li><a href="https://v8.dev/docs/cross-compile-ios">Cross-compiling for iOS</a></li> <li><a href="https://jayconrod.com/posts/51/a-tour-of-v8--full-compiler">V8: full compiler</a></li> <li><a href="https://jayconrod.com/posts/55/a-tour-of-v8-garbage-collection">V8: Garbage Collection</a></li> <li><a href="https://v8.dev/blog/static-roots">Static Roots: Objects with Compile-Time Constant Addresses</a></li> </ul> <p>(Lots more posts there, sorted by category as well: <a href="https://v8.dev/blog/tags/internals">internals</a> <a href="https://v8.dev/blog/tags/ecmascript">ECMAScript spec</a> <a href="https://v8.dev/blog/tags/webassembly">WebAssembly</a> <a href="https://v8.dev/blog/tags/memory">memory</a>)</p> <p>Videos <a href="https://www.youtube.com/@theavocoder">by Lydia Hallie</a>:</p> <ul> <li><a href="https://www.youtube.com/watch?v=zdGfo6I1yrA&amp;t=1s">Execution Contexts</a></li> <li><a href="https://www.youtube.com/watch?app=desktop&amp;v=Xs1EMmBLpn4">Promise Execution</a></li> <li><a href="https://www.youtube.com/watch?app=desktop&amp;v=eiC58R16hb8">Event Loop</a></li> </ul> <p><a href="https://github.com/cmdr2/v8-android">v8-android</a>: Example Android Studio project that embeds v8 (plus some notes on compiling v8 for android)</p> <h2>FFI</h2> <p><a href="https://github.com/eclipsesource/J2V8">Java bindings for V8</a>: J2V8 is a set of Java bindings for V8. J2V8 focuses on performance and tight integration with V8. It also takes a 'primitive first' approach, meaning that if a value can be accessed as a primitive, then it should be. This forces a more static type system between the JS and Java code, but it also improves the performance since intermediate Objects are not created. We developed J2V8 as a high performance engine for our multi-platform mobile toolkit tabris.js and it is a great choice for executing JavaScript on Android devices.</p> <p><a href="https://github.com/pmed/v8pp">v8pp</a>: Bind C++ functions and classes into V8 JavaScript engine</p> https://tedneward.github.io/Research/vms/jvm/graalvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/graalvm/index.html <p><a href="https://www.graalvm.org/">Website</a> | <a href="https://github.com/oracle/graal">Source</a> | <a href="http://www.graalvm.org/docs/">Docs</a></p> <blockquote> <p>"GraalVM is a high-performance JDK distribution designed to accelerate the execution of applications written in Java and other JVM languages along with support for JavaScript, Ruby, Python, and a number of other popular languages. GraalVM’s polyglot capabilities make it possible to mix multiple programming languages in a single application while eliminating foreign language call costs."</p> </blockquote> <hr> <h2>Notes</h2> <p>Notes on Graal and Truffle</p> <h3>Introductory Material</h3> <ul> <li> <p>Add Graal JIT Compilation to Your JVM Language in 5 Steps, A Tutorial<br> <a href="http://stefan-marr.de/2015/11/add-graal-jit-compilation-to-your-jvm-language-in-5-easy-steps-step-1/">http://stefan-marr.de/2015/11/add-graal-jit-compilation-to-your-jvm-language-in-5-easy-steps-step-1/</a></p> </li> <li> <p>The SimpleLanguage, an example of using Truffle with great JavaDocs; official getting-started project:<br> <a href="https://github.com/graalvm/simplelanguage">https://github.com/graalvm/simplelanguage</a></p> </li> <li> <p>Truffle Tutorial, Christan Wimmer, PLDI 2016, 3h recording<br> <a href="https://youtu.be/FJY96_6Y3a4">https://youtu.be/FJY96_6Y3a4</a> <a href="https://lafo.ssw.uni-linz.ac.at/pub/papers/2016_PLDI_Truffle.pdf">Slides</a></p> </li> <li> <p>Truffle Tutorial, Nicolas Laurent<br> <a href="https://norswap.com/truffle-tutorial/">https://norswap.com/truffle-tutorial/</a></p> </li> <li> <p>Series of blog posts on implementing a Lisp<br> <a href="http://cesquivias.github.io/">http://cesquivias.github.io/</a></p> </li> <li> <p>Truffle Tutorials, Presentations, Publications<br> <a href="https://github.com/graalvm/graal/blob/master/docs/Publications.md">https://github.com/graalvm/graal/blob/master/docs/Publications.md</a></p> </li> <li> <p>Graal VM and Graal.JS on the Oracle Technology Network<br> <a href="http://www.oracle.com/technetwork/oracle-labs/program-languages/overview/index-2301583.html">http://www.oracle.com/technetwork/oracle-labs/program-languages/overview/index-2301583.html</a></p> </li> <li> <p>Papers on Truffle<br> <a href="http://ssw.jku.at/Research/Projects/JVM/Truffle.html">http://ssw.jku.at/Research/Projects/JVM/Truffle.html</a></p> </li> <li> <p>Papers on Graal<br> <a href="http://ssw.jku.at/Research/Projects/JVM/Graal.html">http://ssw.jku.at/Research/Projects/JVM/Graal.html</a></p> </li> </ul> <h3>Language Implementations</h3> <h4>Maintained</h4> <ul> <li>FastR, an R implementation: <a href="https://github.com/graalvm/fastr">https://github.com/graalvm/fastr</a></li> <li>TruffleRuby: <a href="https://github.com/graalvm/truffleruby">https://github.com/graalvm/truffleruby</a></li> <li>SOM Smalltalk: <a href="https://github.com/SOM-st/TruffleSOM">https://github.com/SOM-st/TruffleSOM</a></li> <li>SOMns, a Newspeak: <a href="https://github.com/smarr/SOMns">https://github.com/smarr/SOMns</a></li> <li>SimpleLanguage: <a href="https://github.com/graalvm/simplelanguage">https://github.com/graalvm/simplelanguage</a></li> <li>Graal.JS, a JavaScript implementation: <a href="http://www.oracle.com/technetwork/oracle-labs/program-languages/overview/index-2301583.html">http://www.oracle.com/technetwork/oracle-labs/program-languages/overview/index-2301583.html</a>, <a href="https://github.com/graalvm/graaljs/">https://github.com/graalvm/graaljs/</a></li> <li>GraalPython, a Python 3 implementation: <a href="https://github.com/graalvm/graalpython/">https://github.com/graalvm/graalpython/</a></li> <li>Sulong, a LLVM IR implementation on top of the JVM <a href="https://github.com/graalvm/sulong">https://github.com/graalvm/sulong</a></li> <li>RegexLanguage, a JavaScript-flavored regex language <a href="https://github.com/oracle/graal/blob/master/regex/src/com.oracle.truffle.regex/src/com/oracle/truffle/regex/RegexLanguage.java#L38">https://github.com/oracle/graal/blob/master/regex/src/com.oracle.truffle.regex/src/com/oracle/truffle/regex/RegexLanguage.java#L38</a></li> </ul> <h4>Status Unknown</h4> <ul> <li>ZipPy, a Python 3: <a href="https://github.com/securesystemslab/zippy">https://github.com/securesystemslab/zippy</a></li> </ul> <h4>Experiments</h4> <ul> <li> <p>TruffleSqueak, Squeak interpreter on top of Truffle <a href="https://github.com/timfel/trufflesqueak">https://github.com/timfel/trufflesqueak</a></p> </li> <li> <p>TruffleC, a C on top of the JVM: <a href="http://dl.acm.org/citation.cfm?id=2647528">http://dl.acm.org/citation.cfm?id=2647528</a></p> </li> <li> <p>Mumbler: A Lisp<br> <a href="https://github.com/cesquivias/mumbler">https://github.com/cesquivias/mumbler</a> &amp; <a href="http://cesquivias.github.io/">http://cesquivias.github.io/</a></p> </li> <li> <p>Lua<br> <a href="https://github.com/lucasallan/LuaTruffle">https://github.com/lucasallan/LuaTruffle</a> &amp; <a href="http://www.luatruffle.org/">http://www.luatruffle.org/</a></p> </li> <li> <p>Hex<br> <a href="https://bitbucket.org/hexafraction/truffles">https://bitbucket.org/hexafraction/truffles</a> &amp; <a href="https://hextruffle.wordpress.com/">https://hextruffle.wordpress.com/</a></p> </li> <li> <p>Golo (a minimal Truffle backend)<br> <a href="https://github.com/smarr/golo-lang/tree/truffle">https://github.com/smarr/golo-lang/tree/truffle</a></p> </li> <li> <p>TruffleMATE, a Smalltalk with a completely reified runtime system<br> <a href="https://github.com/charig/TruffleMATE">https://github.com/charig/TruffleMATE</a></p> </li> <li> <p>PureScript<br> <a href="https://github.com/slamdata/truffled-purescript">https://github.com/slamdata/truffled-purescript</a></p> </li> <li> <p>Mozart-Graal: An implementation of the Oz language<br> <a href="https://github.com/eregon/mozart-graal">https://github.com/eregon/mozart-graal</a></p> </li> <li> <p>DynSem: A Language Specification-Language, as a meta-interpreter in Truffle<br> <a href="https://github.com/metaborg/dynsem">https://github.com/metaborg/dynsem</a></p> </li> <li> <p>Cover: A Safe Subset of C++<br> <a href="https://github.com/gerard-/cover">https://github.com/gerard-/cover</a></p> </li> <li> <p>TrufflePascal<br> <a href="https://github.com/Aspect26/TrufflePascal/">https://github.com/Aspect26/TrufflePascal/</a></p> </li> <li> <p>Reactive Ruby: Truffle Ruby meets Reactive Programming<br> <a href="https://github.com/guidosalva/ReactiveRubyTruffle">https://github.com/guidosalva/ReactiveRubyTruffle</a></p> </li> <li> <p>PorcE: An Orc implementation on top of Truffle<br> <a href="https://github.com/orc-lang/orc/tree/master/PorcE">https://github.com/orc-lang/orc/tree/master/PorcE</a></p> </li> <li> <p>shen-truffle<br> <a href="https://github.com/ragnard/shen-truffle">https://github.com/ragnard/shen-truffle</a></p> </li> <li> <p>NQP - Not Quite Perl (6)<br> <a href="https://github.com/perl6/nqp/tree/truffle">https://github.com/perl6/nqp/tree/truffle</a></p> </li> <li> <p>Go<br> <a href="https://github.com/PDZaninov/GoLang-Compiler">https://github.com/PDZaninov/GoLang-Compiler</a></p> </li> </ul> <h3>Key Papers</h3> <ul> <li> <p>The Truffle paper: “Self-Optimizing AST Interpreters”<br> <a href="http://lafo.ssw.uni-linz.ac.at/papers/2012_DLS_SelfOptimizingASTInterpreters.pdf">http://lafo.ssw.uni-linz.ac.at/papers/2012_DLS_SelfOptimizingASTInterpreters.pdf</a></p> </li> <li> <p>How to represent Objects Efficiently: “An Object Storage Model for the Truffle Language Implementation Framework”<br> <a href="http://chrisseaton.com/rubytruffle/pppj14-om/pppj14-om.pdf">http://chrisseaton.com/rubytruffle/pppj14-om/pppj14-om.pdf</a></p> </li> <li> <p>The Truffle+Graal paper: "One VM to Rule Them All"<br> <a href="http://lafo.ssw.uni-linz.ac.at/papers/2013_Onward_OneVMToRuleThemAll.pdf">http://lafo.ssw.uni-linz.ac.at/papers/2013_Onward_OneVMToRuleThemAll.pdf</a></p> </li> <li> <p>The Partical Evaluation paper: "Practical partial evaluation for high-performance dynamic language runtimes"<br> accessible via: <a href="https://chrisseaton.com/rubytruffle/pldi17-truffle/pldi17-truffle.pdf">https://chrisseaton.com/rubytruffle/pldi17-truffle/pldi17-truffle.pdf</a></p> </li> </ul> <h3>Other Related Papers</h3> <ul> <li>One Compiler: Deoptimization to Optimized Code (work on SubstrateVM, AOT compilation of Truffle code)<br> <a href="http://doi.org/10.1145/3033019.3033025">http://doi.org/10.1145/3033019.3033025</a></li> </ul> https://tedneward.github.io/Research/vms/jvm/jvmulator/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/jvmulator/index.html <p><a href="https://github.com/alblue/jvmulator">Source</a></p> <p>This provides a simple emulator for Java bytecode as well as an in-memory Java compiler to allow bytecode to be generated. The generated code can be executed as well as emulated to allow stepping through bytecode line by line, and seeing what the content of the local variables or stack happens to be.</p> https://tedneward.github.io/Research/vms/jvm/openjdk/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/openjdk/index.html <p><a href="http://openjdk.java.net/groups/hotspot/">OpenJDK</a> (<a href="https://github.com/openjdk">Source</a>); see <a href="https://foojay.io/">foojay</a> for interesting details on OpenJDK</p> <h3>Performance-related</h3> <p><a href="https://www.oracle.com/java/technologies/whitepaper.html">Java HotSpot Performance Engine Architecture</a></p> <p><a href="https://www.oracle.com/java/technologies/tuning-garbage-collection-v50-java-virtual-machine.html">Tuning Garbage Collection with the 5.0 JVM</a></p> <p><a href="https://github.com/moditect/jfr-analytics">jfr-analytics</a>: An exploration for running analytics on JDK Flight Recorder recordings. There's two areas of interest: (1) Pull-based SQL queries on JFR recording files, using Apache Calcite, and (2) Streaming queries on realtime JFR event streams (implementation tbd., e.g. via Apache Flink or Akka Streams).</p> https://tedneward.github.io/Research/vms/liblg/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/liblg/index.html <p><a href="https://github.com/codr7/liblg">Source</a> | <a href="https://news.ycombinator.com/item?id=24388687">HN</a></p> https://tedneward.github.io/Research/vms/moarvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/moarvm/index.html <p><a href="https://moarvm.org">Website</a> | <a href="https://github.com/MoarVM/MoarVM">Github</a></p> <p>From Github:</p> <blockquote> <p>MoarVM (short for Metamodel On A Runtime Virtual Machine) is a runtime built for the 6model object system. It is primarily aimed at running NQP and Rakudo, but should be able to serve as a backend for any compilers built using the NQP compiler toolchain.</p> </blockquote> https://tedneward.github.io/Research/vms/neko/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/neko/index.html <p><a href="http://nekovm.org/">Website</a> | <a href="https://github.com/HaxeFoundation/neko">Source</a></p> https://tedneward.github.io/Research/vms/pearpc/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/pearpc/index.html <p><a href="http://pearpc.sourceforge.net/">Website</a> | <a href="https://github.com/sebastianbiallas/pearpc">Source</a></p> https://tedneward.github.io/Research/vms/js/primjs/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/primjs/index.html <p><a href="https://github.com/lynx-family/primjs">Source</a></p> <p>Built on top of <a href="../quickjs">QuickJS</a>.</p> https://tedneward.github.io/Research/vms/jsemu/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jsemu/index.html <p>A list of emulators written in the JavaScript programming language.</p> <p>This list started as a compilation of JavaScript emulators posted to Echo JS<br> over the years. If you know about any missing emulators, please consider<br> adding them to the collection: the source for this page is available on<br> <a href="https://github.com/fcambus/jsemu">GitHub</a>. Thank you in advance.</p> <h2>Acorn</h2> <ul> <li><a href="https://floooh.github.io/tiny8bit/atom.html">Atom Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="http://econet.network/atom/">Acorn Atom Emulator</a> - by Phil Mainwaring. Software archive <a href="http://econet.network/atom/menu.html">here</a>. (Older version <a href="http://phils-place.co.uk/HTeMuLator/atom/">here</a>: Type "OLD" for an Easter Egg.)</li> <li><a href="https://elkjs.azurewebsites.net/">ElkJS</a> - JavaScript based Acorn Electron emulator (<a href="https://github.com/dmcoles/elkjs">Source</a>)</li> <li><a href="https://bbc.godbolt.org">JSBeeb</a> - JavaScript BBC Micro emulator (<a href="https://github.com/mattgodbolt/jsbeeb">Source</a>) (<a href="https://xania.org/Emulation">Development blog</a>)</li> <li><a href="https://jsacorn.commandercoder.com/?model=atom">JSAtom</a> - JavaScript Acorn Atom emulator including Software Archive (<a href="https://github.com/CommanderCoder/JSATOM">Source</a>) Based on JSBeeb</li> </ul> <h2>Altair</h2> <ul> <li><a href="https://s2js.com/altair/">MITS Altair Simulator</a> Front panel simulation of the 8080-based Altair, by Ian Davies, built on 8080.js</li> </ul> <h2>Amstrad</h2> <ul> <li><a href="https://web.archive.org/web/20130302051242/http://www.cpcbox.com/">CPCBox</a> - Amstrad CPC emulator in JavaScript (archive link)</li> <li><a href="https://floooh.github.io/tiny8bit/cpc.html">CPC Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li>Roland - An Amstrad CPC emulator written in JavaScript. (<a href="https://sourceforge.net/p/emuscriptoria/code/HEAD/tree/jcpc.js">Source</a>)</li> </ul> <h2>Apple</h2> <ul> <li><a href="https://www.scullinsteel.com/apple1/">Apple 1js</a> - by Will Scullin</li> <li><a href="https://inindev.github.io/apple2e/">Apple2e</a> - Apple IIe emulator by John Clark (<a href="https://github.com/inindev/apple2e">Source</a>)</li> <li><a href="https://www.scullinsteel.com/apple2/">Apple IIjs</a> - An Apple ][ Emulator in JavaScript</li> <li><a href="https://porkrind.org/a2/">a2</a> - A fast, WebGL optimized Apple ][+ emulator</li> <li><a href="https://github.com/gmegidish/apple2js">Apple2JS</a> - A JavaScript emulator for the Apple II</li> <li><a href="https://github.com/jsdf/macemu">BasiliskII</a> - An Emscripten Port of BasiliskII (<a href="https://jamesfriend.com.au/projects/basiliskii/BasiliskII-worker.html">Demo 1</a>, <a href="http://macintosh-js-html.herokuapp.com/">Demo 2</a>)</li> <li><a href="https://github.com/yksoft1/minivmac-em">MiniVMac-Em</a> - An Emscripten Port of MiniVMac (<a href="https://yksoft1.github.io/mvmacdemo.html">Demo</a>)</li> <li><a href="http://www.skibo.net/6502/apple2/">Yet Another Apple 2+ in JavaScript</a> - by Thomas Skibo</li> </ul> <h2>Atari</h2> <ul> <li><a href="https://8bitworkshop.com/?platform=vcs&amp;file=examples%2Fhello">8bit Workshop</a> - VCS IDE by Steven Hugg, building on Javatari.js</li> <li><a href="https://estyjs.azurewebsites.net">EstyJS</a> - A pretty fast and functional JavaScript Atari ST emulator (<a href="https://github.com/dmcoles/estyjs">Source</a>)</li> <li><a href="https://javatari.org">Javatari.js</a> - Atari 2600 emulator by Paulo Augusto Peccin. (<a href="https://www.2600online.com/adventure.php">Example cartridges online</a>) (<a href="https://github.com/ppeccin/javatari.js">Source</a>)</li> <li><a href="http://zerstoerung.de/jsa8e/">jsA8E</a> - JavaScript version of the A8E Atari 800 XL Emulator</li> <li><a href="https://raz0red.github.io/js7800/">JS7800</a> - JavaScript Atari 7800 Emulator</li> </ul> <h2>Commodore</h2> <ul> <li><a href="http://commodore-lcd.lgb.hu/jsemu/">Commodore LCD emulator</a> - by Gábor Lénárt "LGB"</li> <li><a href="http://www.skibo.net/6502/pet2001/">Commodore PET</a> - by Thomas Skibo</li> <li><a href="https://www.kingsquare.nl/jsc64/">JSC64</a> - Commodore 64 emulator written in JavaScript (<a href="https://github.com/Reggino/jsc64">Source</a>)</li> <li><a href="https://mborgbrant.github.io/c64js/">c64js</a> - Commodore 64 emulator written in JavaScript by Mikael Borgbrant (<a href="https://github.com/mborgbrant/c64js">Source</a>)</li> <li><a href="https://floooh.github.io/tiny8bit/c64.html">C64 Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="http://www.robsayers.com/jskim1/">Kim1</a> - emulation in JavaScript by Rob Sayers (<a href="https://github.com/rsayers/jskim1">Source</a>)</li> <li><a href="http://scriptedamigaemulator.net">SAE</a> - Scripted Amiga Emulator (<a href="https://github.com/naTmeg/ScriptedAmigaEmulator">Source</a>)</li> <li><a href="https://www.mdawson.net/vic20chrome/vic20.php">VIC-20 Emulator</a> - JavaScript VIC-20 emulator</li> <li><a href="https://vice.janicek.co/c64/">VICE.js</a> - Versatile Commodore Emulator for JavaScript (<a href="https://github.com/rjanicek/vice.js">Source</a>)</li> </ul> <h2>Data General</h2> <ul> <li><a href="http://www.novasareforever.org/novajs/index.php">Novas Are Forever</a> - by Wild Hare Computer Systems, for the 50th anniversary.</li> </ul> <h2>DEC</h2> <ul> <li><a href="https://www.masswerk.at/spacewar/">PDP-1 running the SpaceWar game</a> and the <a href="https://www.masswerk.at/minskytron/">Minskytron demo</a> - by Norbert Landsteiner</li> <li><a href="https://programmer209.wordpress.com/2011/01/30/pdp-8-assembly-language-part-2/">PDP-8</a> and <a href="https://programmer209.wordpress.com/2011/08/14/pdp-11-assembly-language-simulator/">PDP-11</a> simulators with assembly language interfaces (explanatory articles with full source, not live site) - by programmer209</li> <li><a href="https://skn.noip.me/pdp11/pdp11.html">JavaScript PDP 11</a> - PDP-11/70 emulator with simulated front panel and a choice of operating systems. By Paul Nankervis</li> <li><a href="http://pdp11.aiju.de/">PDP-11 Emulator</a> - A JavaScript PDP-11 emulator running UNIX Sixth Edition. By Julius Schmidt</li> <li><a href="https://pavel-krivanek.github.io/pdp11/">with teletype</a> - PDP-11 emulator with teletype interface</li> <li><a href="https://takahirox.github.io/pdp11-js/unixv6.html">pdp11-js</a> - PDP-11 emulator with UNIX V6. By Takahiro Aoyagi (<a href="https://github.com/takahirox/pdp11-js">Source</a>)</li> </ul> <h2>Nintendo</h2> <ul> <li><a href="https://github.com/schibo/1964js">1964js</a> - JavaScript port of the 1964 N64 emulator for Windows</li> <li><a href="https://github.com/jpikl/cfxnes">CfxNES</a> - JavaScript NES Emulator (<a href="https://cfxnes.herokuapp.com/">Demo</a>)</li> <li><a href="https://app.7io.org/CycloaJS/">CycloaJS</a> - JavaScript NES Emulator (<a href="https://github.com/ledyba/CycloaJS">Source</a>)</li> <li><a href="https://bitbucket.org/tsone/em-fceux">em-fceux</a> - an Emscripten port of FCEUX, an emulator of NES, Famicom, Famicom Disk System (FDS), and Dendy consoles. <a href="https://tsone.kapsi.fi/em-fceux/">Demo site</a></li> <li><a href="https://endrift.github.io/gbajs/">GBA.js</a> - Game Boy Advance in the Browser (<a href="https://github.com/endrift/gbajs">Source</a>)</li> <li><a href="https://taisel.github.io/IodineGBA/">IodineGBA</a> - A Game Boy Advance emulator written entirely in JavaScript (<a href="https://github.com/taisel/IodineGBA">Source</a>)</li> <li><a href="https://gba.ninja/">gba.ninja</a> - JavaScript port of VisualBoyAdvance-M, a Game Boy Advance emulator (<a href="https://github.com/simon-paris/gba.ninja">Source</a>)</li> <li><a href="https://jsnes.org">JSNES</a> - A JavaScript NES emulator (<a href="https://github.com/bfirsh/jsnes">Source</a>)</li> <li><a href="http://koen.kivits.com/nesnes/">NESNES</a> - JavaScript NES emulator, also available as a <a href="https://github.com/koenkivits/x-nes">web component</a> (<a href="https://github.com/koenkivits/nesnes">Source</a>)</li> <li><a href="http://zelex.net/nezulator/">Nezulator</a> - A NES emulator in JavaScript</li> <li><a href="https://neilb.net/n64wasm/">N64Wasm</a> - A web based N64 Emulator (<a href="https://github.com/nbarkhina/N64Wasm">Source</a>)</li> <li><a href="https://tjwei.github.io/xnes/">XNES</a> - Experimental JavaScript Super Nintendo emulators (<a href="https://github.com/tjwei/xnes">Source</a>)</li> <li>jsGB - A Game Boy emulator in JavaScript (<a href="https://github.com/Two9A/jsGB">Source</a>)</li> <li><a href="https://ardean.github.io/jsGBC-web/">jsGBC</a> - Game Boy Color Emulator written in JavaScript (<a href="https://github.com/ardean/jsGBC">Source</a>)</li> <li><a href="https://jquesnelle.github.io/mupen64plus-ui-console/">mupen64plus</a> - A port of the popular Nintendo 64 emulator for the Web (<a href="https://github.com/jquesnelle/mupen64plus-ui-console/">Source</a>)</li> <li><a href="https://hulkholden.github.io/n64js/">n64js</a> - An N64 emulator in JavaScript (<a href="https://github.com/hulkholden/n64js">Source</a>)</li> <li><a href="https://koute.github.io/pinky-web/">pinky</a> - A Rust based NES emulator ported to the web via WebAssembly (<a href="https://github.com/koute/pinky/tree/master/pinky-web">Source</a>)</li> <li><a href="https://github.com/44670/desmume-wasm">DeSmuME-wasm</a> - A WebAssembly port of the DeSmuME Nintendo DS emulator (<a href="https://ds.44670.org/">Demo</a>)</li> <li><a href="https://github.com/Unzor/desmond">Desmond.js</a> - A portable/embeddable version of DeSmuME-wasm (<a href="https://js-emulators.github.io/desmond/">Demo</a>)</li> <li><a href="https://github.com/skylersaleh/SkyEmu">SkyEmu</a> - SkyEmu is a low level Game Boy, Game Boy Color and Game Boy Advance emulator</li> <li><a href="https://github.com/linoscope/CAMLBOY">CAMLBOY</a> - A Game Boy emulator that runs in your browser written in OCaml</li> <li><a href="https://github.com/Unzor/TinyGB">TinyGB</a> - A Game Boy emulator that runs only in the terminal written in NodeJS</li> <li><a href="https://boytacean.joao.me">Boytacean</a> - A Game Boy emulator written in Rust that works in the Browser using WebAssembly (<a href="https://github.com/joamag/boytacean">Source</a>)</li> </ul> <h2>Robotron / VEB Mikroelektronik</h2> <ul> <li><a href="https://lanale.de/kc85_emu/KC85_Emu.html">KC85_Emu</a> - KC85/3 and KC85/4 emulator by Alexander Lang</li> <li>KC85/2 family emulators by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>): <ul> <li><a href="https://floooh.github.io/tiny8bit/kc85.html?type=kc85_2">KC85/2 Tiny Emu</a></li> <li><a href="https://floooh.github.io/tiny8bit/kc85.html?type=kc85_3">KC85/3 Tiny Emu</a></li> <li><a href="https://floooh.github.io/tiny8bit/kc85.html?type=kc85_4">KC85/4 Tiny Emu</a></li> </ul> </li> <li><a href="https://floooh.github.io/tiny8bit/lc80.html">LC-80</a> - single board U880-based trainer from the DDR (warning: auto-plays sound at boot), via emscripten. By Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="https://floooh.github.io/tiny8bit/z1013.html">Z1013 Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="https://floooh.github.io/tiny8bit/z9001.html?type=kc87">KC87 Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> </ul> <h2>Sega</h2> <ul> <li><a href="https://gmarty.github.io/jsSMS/">jsSMS</a> - JavaScript Sega Master System &amp; Game Gear emulator (<a href="https://github.com/gmarty/jsSMS">Source</a>)</li> <li><a href="https://xania.org/miracle/miracle.html">Miracle</a> - Sega Master System emulator (<a href="https://github.com/mattgodbolt/Miracle">Source</a>)</li> </ul> <h2>Sinclair</h2> <ul> <li><a href="http://jsspeccy.zxdemo.org">JSSpeccy</a> - A ZX Spectrum emulator in JavaScript (<a href="https://github.com/gasman/jsspeccy2">Source</a>)</li> <li><a href="http://www.zx81stuff.org.uk/zx81/jtyone.html">JtyOne Online ZX81 Emulator</a> - by Simon Holdsworth</li> <li><a href="http://torinak.com/qaop">Qaop/JS</a> - ZX Spectrum emulator</li> <li><a href="https://nocanvas.zame-dev.org/0004/">ZX80 Emulator</a> - JavaScript ZX80 Emulator</li> <li><a href="http://em.ulat.es/machines/SinclairZX80">EMF ZX80</a> - EMF-based ZX80 Emulator - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-sinclair-zx80">Source</a>)</li> <li><a href="http://em.ulat.es/machines/SinclairZX81">EMF ZX81</a> - EMF-based ZX81 Emulator - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-sinclair-zx81">Source</a>)</li> <li><a href="http://www.dougrice.plus.com/dev/seg_mk14.htm">Science of Cambridge MK14 simulator</a> - by Doug Rice, based on Paul Robson's offline emulator.</li> <li><a href="https://floooh.github.io/tiny8bit/zx.html?type=zx48k">ZX Spectrum 48K Tiny Emy</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="https://floooh.github.io/tiny8bit/zx.html?type=zx128">ZX Spectrum 128 Tiny Emu</a> - by Andre Weissflog (<a href="https://github.com/floooh/chips-test">source</a>)</li> </ul> <h2>Sony</h2> <ul> <li><a href="https://kootstra-rene.github.io/enge-js/">eNGE</a> - JavaScript browser based PSX emulator (runs games at full speed) (<a href="https://github.com/kootstra-rene/enge-js">Source</a>)</li> <li>PSeudo - JavaScript/WebGL/WebAudio browser based PLAYSTATION emulator (aka PSX) (needs boot ROM image, not supplied) (<a href="https://github.com/dkoluris/pseudo">Source</a>)</li> <li><a href="https://tjwei.github.io/pcsxjs/">PCSXjs</a> - Modified PCSX-Reloaded compiled with Emscripten (<a href="https://github.com/tjwei/pcsxjs">Source</a>)</li> <li><a href="https://github.com/js-emulators/WASMpsx">WASMpsx</a> - Easily embeddable fork of PCSXjs</li> <li><a href="https://github.com/kpspemu/kpspemu">kpspemu</a> - PSP Emulator written in Kotlin for JVM, JS and Native</li> <li><a href="https://playjs.purei.org">Play!.js</a> - This is a port of Play!, a PlayStation2 emulator, running in a web browser</li> </ul> <h2>Tandy</h2> <ul> <li><a href="https://bitchin100.com/CloudT/#/M100Display">CloudT TRS-80 Model 100 Emulator</a> - by John R. Hogerhuis (<a href="https://www.mail-archive.com/m100@lists.bitchin100.com/msg02714.html">Announcement</a>)</li> <li><a href="http://mc-10.com/">MC-10 Emulator</a> - Emulator for the TRS-80 MC-10 microcomputer</li> <li><a href="http://trsjs.48k.ca/trs80.html">TRS-80 Model III Emulator</a> a JavaScript emulator for the TRS-80 Model III, by Peter Phillips</li> <li><a href="http://www.haplessgenius.com/mocha/">JS Mocha</a> - The HTML5 CoCo 2 Emulator</li> </ul> <h2>Multi-system Emulators</h2> <ul> <li><a href="https://archive.org/details/historicalsoftware">JSMESS examples</a> - The JavaScript MESS (Multi Emulator Super System) (<a href="https://github.com/jsmess/jsmess">Source</a>) (<a href="https://github.com/jsmess/jsmess/wiki">Notes</a>)</li> <li><a href="https://github.com/jsdf/pce">PCE</a> - PC emulators in JavaScript (Atari ST, <a href="https://jamesfriend.com.au/pce-js/ibmpc-win/">IBM PC 5150</a>, <a href="https://jamesfriend.com.au/pce-js/">Macintosh</a>, RC759 Piccoline)</li> <li><a href="https://toadking.com/retroarch/">RetroArch</a> - JavaScript port of RetroArch (bundles Gambatte (Game Boy), Genesis Plus GX, Handy (Lynx), Snes9x Next, VBA Next (Game Boy Advance), Tyrquake and FinalBurn Alpha)</li> <li><a href="http://retroweb.maclab.org">RetroWeb</a> - collection of JavaScript emulators and boot media, including Apple-IIe (VisiCalc), Macintosh (System 1.0), Atari 1040ST, Commodore 64, Amiga 500 (Workbench 1.3), IBM PC Model 5150 (PC-DOS, CP/M-86, Cassette Basic), IBM PC XT (DOS, GEM 1.2, VisiCalc, Windows 1.01, 8088 Corruption demo), RC759 Piccoline (Eliza, Bil-simulation, Concurrent CP/M-86), TRS-80.</li> <li><a href="https://floooh.github.io/virtualkc/">YAKC</a> - Z1013, Z9001, KC85/2 family, Speccy, CPC, Acorn Atom, C64, with integrated debugging UI (<a href="https://github.com/floooh/yakc">source</a>)</li> <li><a href="https://floooh.github.io/tiny8bit/">Tiny Emulators</a> - based on the same chip- and system-emulator source code as YAKC, but as minimal WASM apps without fluff (<a href="https://github.com/floooh/chips-test">source</a>)</li> <li><a href="https://emulatorjs.netlify.app/">EmulatorJS</a> - RetroArch compiled to emscripten with a nice wrapper (<a href="https://github.com/emulatorjs/emulatorjs">source</a>)</li> </ul> <h2>PC Emulators</h2> <ul> <li><a href="https://github.com/dreamlayers/em-dosbox">Em-DOSBox</a> - An Emscripten port of DOSBox (<a href="http://playdosgamesonline.com/">Demo</a>)</li> <li><a href="https://github.com/nepx/halfix/">Halfix</a> - x86 PC emulator that runs both natively and in the browser, via WebAssembly (<a href="https://nepx.github.io/halfix-demo/">Demo 1</a>, <a href="https://pixelsuft.github.io/onwin/">Demo2</a>)</li> <li><a href="https://js-dos.com">js-dos</a> - WebAssembly port of DOSBox (fork of Em-DOSBox with better js API) (<a href="https://js-dos.com/#js-dos-622-demo">Demo</a>,<a href="https://github.com/caiiiycuk/js-dos">Source</a>)</li> <li><a href="https://bellard.org/jslinux/">JS/Linux</a> - JavaScript PC emulator</li> <li><a href="http://jsdosbox.sourceforge.net">JsDOSBox</a> - JavaScript PC DOS emulator (<a href="https://sourceforge.net/projects/jsdosbox/files/">Source</a>)</li> <li><a href="https://www.pcjs.org">PCjs</a> - IBM PC Model 5150 emulator (<a href="https://github.com/jeffpar/pcjs">Source</a>)</li> <li><a href="https://github.com/atrosinenko/qemujs">QemuJS</a> - An Emscripten port of QEMU (<a href="https://atrosinenko.github.io/qemujs-demo">Demo</a>)</li> <li><a href="https://copy.sh/v86/">Virtual x86</a> - x86 virtualization in your browser, recompiling x86 to wasm on the fly (<a href="https://github.com/copy/v86">Source</a>)</li> <li><a href="http://jemul8.com">jemul8</a> - An object-oriented JavaScript x86 emulator for Node.js and the Browser (<a href="https://github.com/asmblah/jemul8">Source</a>)</li> <li>jsbochs - Bochs PC emulator for the Browser (<a href="https://github.com/codinguncut/jsbochs">Source</a>)</li> <li><a href="http://www.boxedwine.org/">Boxedwine</a> - An emulator that runs Windows applications using Wine and emulating a Linux kernel and CPU (<a href="https://github.com/danoon2/Boxedwine">Source</a>)</li> </ul> <h2>Bare CPUs</h2> <ul> <li><a href="http://www.xlnsresearch.com/scelbal.htm">8008 running SCELBAL</a> by Mark G. Arnold. ("SCELBAL is the only open-source, floating-point, high-level language ever implemented on the 8008")</li> <li><a href="http://typedarray.org/wp-content/projects/Intel8080/index.html">8080 CPU emulator</a> - Intel 8080 CPU emulator running Space Invaders ROM (<a href="https://github.com/thibaultimbert/Intel8080">Source</a>)</li> <li><a href="https://wokwi.com/arduino/projects/297787059514376717">AVR8js</a> - AVR architecture emulator, capable of running Arduino code (<a href="https://github.com/wokwi/avr8js">source</a>)</li> <li><a href="http://em.ulat.es/machines/TaitoSpaceInvaders">EMF Arcade Invaders</a> - EMF-based 8080 Emulator with Space Invaders ROM - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-arcade_invaders">Source</a>)</li> <li><a href="https://www.netpipe.ca/apps/8086tiny/8086tiny.html">8086tiny running FreeDOS</a> - An Emscriptem port of Adrian Cable's 8086tiny. Source, and other emulators ported by Clay Shippy, at <a href="emscripten-projects">https://github.com/tecan/emscripten-projects</a>.</li> <li><a href="https://github.com/riscv/riscv-angel">Angel</a> - JavaScript RISC-V ISA simulator booting Linux in a web-browser</li> <li><a href="https://github.com/JeremyLikness/6502emulator#readme">Angular 2 6502</a> written with TypeScript and Angular 2, by Jeremy Likness</li> <li><a href="https://biged.github.io/arm-js/arm-js.html">ARM-js</a> - An ARM emulator written in JavaScript (<a href="https://github.com/ozaki-r/arm-js">Source</a>)</li> <li><a href="https://www.asm80.com">ASM80</a> - Online assembler for 8-bit microprocessors by Martin Malý. Includes emulation of several machines: 8080, Z80, 6502, 6809. (<a href="https://github.com/maly">Sources</a>)</li> <li>Basic MIPS functional simulator by Mianzhi Wang (morriswmz). (<a href="https://github.com/morriswmz/SimpleMIPS.js">Source</a>)</li> <li><a href="https://skilldrick.github.io/easy6502/">Easy6502</a> - JavaScript 6502 tutorial and emulator (<a href="https://github.com/skilldrick/easy6502">Source</a>)</li> <li><a href="https://web.edumips.org/">EduMIPS64</a> - <a href="https://www.edumips.org/">Educational MIPS64 CPU</a>, ported from Java by Andrea Spadaccini using GWT (see blog <a href="http://edumips64.blogspot.com/2016/06/towards-javascript-port-of-edumips64.html">here</a>.) (<a href="https://github.com/lupino3/edumips64">Source</a>)</li> <li><a href="http://ivanzuzak.info/FRISCjs/webapp/">FRISCjs</a> - an 8-register educational RISC from the University of Zagreb, with both assembler and front panel, by Ivan Žužak. <a href="https://github.com/izuzak/FRISCjs">Source</a></li> <li><a href="https://www.mdawson.net/asm6502/6502asm.php">Imaginary 6502</a> - 6502 Emulator and Assembler</li> <li><a href="http://e4004.szyc.org/">Intel 4004 emulator</a> - by Maciej Szyc. Includes assembler and disassembler.</li> <li><a href="https://www.tramm.li/i8080/">Intel 8080 CPU Emulator</a> - Emulates a minimal Intel 8080 Microcomputer that runs CP/M</li> <li><a href="https://polysoftit.co.uk/irisc-web/">iRISC</a> - Interactive ARMv7 assembly language interpreter and computer architecture simulator (<a href="https://github.com/rtybanana/irisc-web/">Source</a>)</li> <li><a href="https://bluishcoder.co.nz/js8080/">JavaScript 8080 Emulator</a> - 8080 arcade game emulator in JavaScript</li> <li><a href="http://blog.loadzero.com/demo/mipsdis/demo.html">Mipsdis</a> - MIPS disassembler that runs in the browser</li> <li><a href="https://www.enscope.nl/rrca/">RISC Relay Computer</a> - Relay based computer project with a 16 bit RISC CPU. Emulator includes an assembler and source for a calculator program. By RJH. See <a href="http://www.enscope.nl/rrc">website</a>.</li> <li><a href="https://bellard.org/riscvemu/js/">RISCVEMU</a> - RISC-V emulator boots 64-bit Linux. By Fabrice Bellard.</li> <li><a href="https://wokwi.com/arduino/projects/297323005822894602">RP2040js</a> - Raspberry Pi Pico (RP2040 / Arm Cortex-M0+) emulator with C/C++/MicroPython/CircuitPython support (<a href="https://github.com/wokwi/rp2040js">Source</a>)</li> <li><a href="https://schweigi.github.io/assembler-simulator/">Simple 8-bit Assembler Simulator</a> - Provides a simplified assembler syntax (based on NASM) and is simulating a x86 like CPU (<a href="https://github.com/Schweigi/assembler-simulator">Source</a>)</li> <li>uARM.wasm - ARMv5TE emulator running Linux in browsers (<a href="https://github.com/TonyLianLong/uARM.wasm">Source</a>)</li> <li><a href="https://alexaltea.github.io/unicorn.js/">Unicorn.js</a> - The Unicorn emulator framework, now available for JavaScript (<a href="https://github.com/AlexAltea/unicorn.js">Source</a>)</li> <li><a href="http://visual6502.org/sim/varm/armgl.html">Visual ARM1</a> - JavaScript/WebGL for ARM's first CPU, modelling 25000 transistors at switch level and animating the original chip layout - in 3D. See <a href="http://blog.visual6502.org/2015/11/the-visual-arm1.html">the blog post</a></li> <li><a href="http://www.visual6502.org/JSSim/">Visual 6502</a> - JavaScript simulator for the 6502 CPU, modelling thousands of transistors at switch level and animating the original chip layout. See also <a href="http://www.visual6502.org/JSSim/expert.html?steps=10">expert mode</a>.</li> <li><a href="http://www.visual6502.org/JSSim/expert-6800.html?steps=10">Visual 6800</a> - JavaScript simulator for the Motorola 6800 CPU, modelling thousands of transistors at switch level and animating the original chip layout.</li> <li><a href="http://www.visual6502.org/JSSim/expert-z80.html">Visual Z80</a> - JavaScript simulator for the Z80 CPU, modelling thousands of transistors at switch level and animating the original chip layout.</li> <li><a href="http://thlorenz.com/visulator/">Visulator</a> - x86 machine emulator that visualizes how each instruction is processed (<a href="https://github.com/thlorenz/visulator">Source</a>)</li> <li><a href="http://yamd.xuanji.li">YAMD</a> - Yet Another MIPS Debugger (<a href="https://github.com/zodiac/yamd">Source</a>)</li> <li><a href="https://s-macke.github.io/jor1k/">jor1k</a> - OpenRISC OR1K JavaScript emulator running Linux with network support (<a href="https://github.com/s-macke/jor1k/">Source</a>)</li> <li><a href="https://www.ubercomp.com/jslm32/src/">jslm32</a> - JavaScript LatticeMico32 emulator running Linux (<a href="https://github.com/ubercomp/jslm32/">Source</a>)</li> <li><a href="http://www.nablaman.com/relay/sim/zusie.html">ZUSIE</a> - JavaScript simulation of <a href="http://www.nablaman.com/relay/">Fredrik Andersson's homebrew relay machine</a> inspired by Zuse's machines.</li> </ul> <h2>Early machines</h2> <ul> <li><a href="http://ed-thelen.org/bab/bab-diff-JavaScript.html">Babbage's Difference Engine</a> (First funded 1823, first full build in 1855, first full rebuild in 1991)</li> <li><a href="https://www.fourmilab.ch/babbage/aem/">Babbage's Analytical Engine</a> in JavaScript, by John Walker. (First described 1837, never completed, not yet rebuilt.) (<a href="https://www.fourmilab.ch/babbage/emulator.html">Documentation</a>)</li> <li><a href="http://rendell-attic.org/gol/TMapplet/index.htm">Turing machine</a> simulated in JavaScript. See <a href="http://rendell-attic.org/gol/utm/">here</a> for more information. (1936)</li> <li><a href="http://zuse-z1.zib.de/simulations/z1/adders/wgl/">Z1 machine's adder in 3D</a> JavaScript/WebGL interactive simulation of the mechanical adder of Zuse's first machine. By Jakob Mischek (<a href="https://github.com/daign/zuse-z1.js">Source</a>) (1938)</li> <li><a href="http://gymoberwil.educanet2.ch/a.hu/projektarbeit/zuse/addition/addition.htm">Z3 machine's adder</a> - ripple-carry electromechanical adder simulated in JavaScript, by Henry Raymond, Patrick Seewald and Vijeinath Tissaveerasingham. <a href="http://gymoberwil.educanet2.ch/a.hu/projektarbeit/zuse/simu.htm">Explanation</a> (1941)</li> <li><a href="http://edmundgriffiths.com/jsssem.html">JsSSEM</a> - Manchester Small-Scale Experimental Machine emulator (Also check <a href="http://www.edmundgriffiths.com/czero.html">Computer/zero</a> which is very loosely based on the SSEM, and its <a href="http://www.edmundgriffiths.com/degreezero.html">tutorial</a>) (1948)</li> <li><a href="https://danieljabailey.github.io/c88-js/">C88</a> - C88 computer simulation (<a href="http://bitofahack.com/post/1434913931">The Homebrew CPU inspired by the SSEM</a>) (1948)</li> <li><a href="http://nhiro.org/learn_language/repos/EDSAC-on-browser/index.html">EDSAC on Browser</a> - by NISHIO Hirokazu (<a href="https://www.dcs.warwick.ac.uk/~edsac/Software/EdsacTG.pdf">Programming guide</a>) (1949)</li> <li><a href="http://em.ulat.es/machines/Elliott903">EMF Elliott</a> - EMF-based Elliott Emulator - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-elliott-903">Source</a>)</li> <li>WITCH Emulator - The Harwell Dekatron Machine, by Justin King. (<a href="https://github.com/jsking/witch-e">Source and example programs</a>) (1951)</li> <li><a href="https://www.masswerk.at/virtualunivac/">UNIVAC I emulator</a> - JavaScript emulator by Norbert Landsteiner (1951)</li> <li><a href="http://www.phkimpel.us/ElectroData-205/webUI/D205.html">ElectroData/Burroughs Datatron 205 Emulator</a> - by Paul Kimpel (<a href="https://github.com/pkimpel/retro-205">Source</a>) (1954)</li> <li><a href="https://www.wizforest.com/tech/tx-0/#">TX-0</a> emulator by @wizforest, including several programs from the time. (<a href="https://www.wizforest.com/tech/tx-0/manual.html">Instructions</a>) (1956)</li> <li><a href="http://members.aon.at/nkehrer/mailuefterl/mailuefterl.html">Mailüfterl</a> - JavaScript emulator by Norbert Kehrer (1958)</li> <li><a href="http://trinary.ru/projects/setunws/">Setun (Russian)</a> and <a href="https://web.archive.org/web/20170114102246/http://en.trinary.ru/projects/setunws/">English (archived)</a> - JavaScript emulator by Обухов Александр. Ternary machine from Soviet Union (1958)</li> <li><a href="http://digicomp-1.appspot.com">Digi-Comp 1</a> (<a href="https://web.archive.org/web/20140222193139/http://www.scoopsfolks.com/digicomp1">previously</a>) - educational sliding-rods plastic computer. Emulator by Larry Groebe and Kevin Williams. (1963)</li> <li><a href="https://museum.syssrc.com/joda/">Digi-Comp II</a> - educational falling-marbles computer. Emulator by Joda Redfearn. (1965)</li> <li><a href="https://retro-b5500.blogspot.com">Burroughs B5500 emulator</a> - Burroughs B5500 emulator in JavaScript (<a href="https://github.com/pkimpel/retro-b5500">Source</a>) (1964)</li> <li><a href="http://svtsim.com/moonjs/agc.html">Apollo Guidance Computer</a> - Moonjs a port by Shahriar Iravanian of Ronald Burkey's Virtual AGC. (1966)</li> <li><a href="http://www.mailcom.com/besm6/runitnew.cgi">BESM-6</a> - with examples in Algol, Fortran, and assembler. By Leonid A. Broukhis and Michael Yaroslavtsev. More <a href="http://www.mailcom.com/besm6/">here</a>. (1968)</li> <li><a href="https://www.cs.drexel.edu/~bls96/museum/cardsim.html">CARDIAC</a> - Bell Labs' CARDIAC cardboard computer from 1969. <a href="https://www.cs.drexel.edu/~bls96/museum/cardiac.html">Instructions</a> (1969)</li> <li><a href="http://www.neocomputer.org/kenbak/kenbak1-JS.html">Kenbak-1</a> - John Blankenbaker's TTL-based 256byte personal computer. <a href="http://www.neocomputer.org/kenbak/">More information</a> (1970)</li> <li><a href="http://nguyen.univ-tln.fr/share/Archi/ordinapoche.html">Ordinapoche</a> - A paper computer from France, invented 1969, popularised in 1981 and 1985. (<a href="http://nguyen.univ-tln.fr/ordinapoche.html">More here</a> and <a href="https://www.abandonware-magazines.org/affiche_mag.php?mag=85&amp;num=3809&amp;album=oui">see also the 1981 magazine</a>)</li> <li><a href="https://www.aconit.org/histoire/Gamma-3/Simulateur/">BullGammaTor</a> - A JavaScript emulator for the Bull Gamma 3 ET computer (<a href="https://github.com/lutrampal/bullgammator/">Source</a>) (1952)</li> </ul> <h2>Calculator emulators</h2> <h3>Microcode-level calculators</h3> <ul> <li><a href="https://www.sydneysmith.com/products/hp21u/run/index.html">HP21u</a>, <a href="https://www.sydneysmith.com/products/hp25u/run/index.html">HP25u</a>, and <a href="https://www.sydneysmith.com/products/hp29u/run/index.html">HP29u</a> - by Greg Sydney-Smith</li> <li><a href="https://ashleyf.github.io/HP35/">HP-35</a> - bug-compatible emulator by Ashley Feniello <a href="https://blogs.msdn.microsoft.com/ashleyf/2010/09/25/microcode-level-hp-35-emulator-in-javascript/">explained here</a> using <a href="https://nonpareil.brouhaha.com/">Eric Smith</a>'s and <a href="https://web.archive.org/web/20210601085601/http://home.citycable.ch/pierrefleur/Jacques-Laporte/">Jacques Laporte</a>'s work</li> <li><a href="https://torotrak.wordpress.com/hp-classic/">HP-35, HP-45, HP-55, HP-65, HP-80</a> - collection of HP Classics, based on Feniello's work, by Francois Roulet</li> <li><a href="http://members.aon.at/nkehrer/hp45_js/hp45.html">HP-45</a> - statically recompiled ROM by Norbert Kehrer</li> <li><a href="https://jadegame.com/jsemu48/">jsEmu48</a> - emscripten port by Julien Meyer of HP EMU By Daniel Nilsson. (<a href="https://github.com/brizzly/jsEmu48">Source</a>)</li> <li><a href="https://www.sydneysmith.com/products/hp55u/run/">HP-55</a>, <a href="https://www.sydneysmith.com/wordpress/hp65/">HP-65</a> and <a href="https://www.sydneysmith.com/wordpress/1178/hp67-microcode-emulator/">HP-67</a> - with extra debug menu, by Greg Sydney-Smith</li> <li><a href="http://mk-61.moy.su/emulator.html">мк-61</a> - Elektronika's programmable calculators MK-61, Б3-34, МК-54, and МК-56, also the Феликс-М (Felix M) arithmometer, a slide rule and an abacus. By Felix Lazarev, Andrey_emu, Sergey Tarasov and others.</li> <li><a href="https://files.righto.com/sinclair">Sinclair Scientific</a> and <a href="https://files.righto.com/ti">TI-1500</a> - calculator simulations including full description of the algorithms and the reverse-engineering process. By Ken Shirriff. Further work by Phil Mainwaring shows <a href="http://phils-place.co.uk/calculators/?calc=sc">Sinclair Cambridge</a>, <a href="http://phils-place.co.uk/calculators/?calc=ti">TI-1500</a> and <a href="http://phils-place.co.uk/calculators/?calc=sc">Sinclair Scientific</a>, each using different customisations of TI's 080x chip.</li> <li><a href="https://www.pcjs.org/devices/ti42/machine/">TI-42 "MBA" Programmable Calculator</a> by Jeff Parsons (PCjs)</li> <li><a href="https://www.pcjs.org/devices/ti55/">TI-55</a> by Jeff Parsons (PCjs)</li> <li><a href="https://www.pcjs.org/devices/ti57/machine/">TI-57 Programmable Calculator</a> by Jeff Parsons (PCjs)</li> <li><a href="https://www.ocf.berkeley.edu/~pad/emu/v11.html">TI-92 Plus emulator</a> - JavaScript emulator for the TI-92 Plus</li> <li><a href="https://tiplanet.org/emu68k_fork/">JavaScript TI-89 / TI-92+ / TI-V200 / TI-89T emulator</a> - by Patrick Davidson and Lionel Debroux</li> </ul> <h3>Workalike calculators</h3> <ul> <li><a href="https://stendec.io/ctb/hp11c.html">HP-11C</a>, <a href="https://stendec.io/ctb/hp12c-platinum.html">HP-12C Platinum</a>, <a href="https://stendec.io/ctb/hp16c.html">HP-16C</a> - by Elvis Pfützenreuter</li> <li><a href="https://jrpn.jovial.com/">HP-16C (Jovial RPN)</a> - by Bill Foote</li> <li><a href="https://hp15c.com/web/hp15c.html">HP-15C</a> - by Greg Hewgill. (<a href="https://github.com/ghewgill/hp15c">Source</a>)</li> <li><a href="http://www.sydneysmith.com/products/gss-hp21/run/index.html#">HP-21</a> and <a href="http://www.sydneysmith.com/products/gss-hp29/run/index.html#">HP-29</a> by Greg Sydney-Smith (See <a href="http://www.sydneysmith.com/wordpress/hp21/">here</a> and <a href="http://www.sydneysmith.com/wordpress/hp29/">here</a>)</li> <li><a href="http://www.jcczone.net/hp-25.html">HP-25</a> - by John Clenance</li> <li><a href="https://neil.fraser.name/software/hp-35/">HP-35</a> - JavaScript emulator by Neil Fraser</li> <li><a href="http://hansklav.home.xs4all.nl/rpn/hp35sos.html">HP-35 SOS</a> - modified HP-35 with stack overflow sensing LED, by Hans Klaver, based on Fraser's work</li> <li><a href="http://www.poleyland.com/hp48/index.html">HP-48</a> - JavaScript implementation of the most commonly used HP-48 functions. More info <a href="http://www.poleyland.com/hp48/help.html">here</a>, by Josh Poley</li> <li><a href="https://biged.github.io/P101/p101viewport.html">Olivetti Programma 101</a> - JavaScript emulation of the first commercial programmable desktop computer, by "Fabioamd87". <a href="https://github.com/Fabioamd87/P101">Source on GitHub</a></li> <li><a href="http://www.caffnib.co.uk/calculators/sinclair_prog.html">Sinclair Cambridge Programmable</a> by Nigel Bromley. (Source <a href="http://www.caffnib.co.uk/calculators/sinclair_prog.js">here</a>.)</li> </ul> <h2>Miscellaneous</h2> <ul> <li><a href="https://www.pcjs.org/docs/c1pjs/">C1Pjs</a> - JavaScript simulation of the Challenger 1P (PCjs)</li> <li><a href="https://archive.org/details/canoncat">Canon Cat (in JSMess)</a> - Jef Raskin's Forth-capable 68000-based word processor. (<a href="https://github.com/mamedev/mame/blob/master/src/mame/drivers/cat.cpp#L16">Instructions</a> and <a href="http://www.canoncat.net/">more info</a>)</li> <li><a href="https://www.bedroomlan.org/hardware/cft/microcode-emulator">CFT</a> - JavaScript simulation of Alexios Chouchoulas' 16-bit homebrew TTL machine. (<a href="https://www.bedroomlan.org/hardware/cft/">More information</a> including documentation and a video.)</li> <li><a href="https://biged.github.io/Chip-8-Emulator/">Chip-8 virtual machine</a> by Alexander Dickson - see <a href="http://blog.alexanderdickson.com/javascript-chip-8-emulator">blog entry</a></li> <li><a href="http://pachisystems.com/JSEmu/chip8.html">Chip-8 virtual machine</a> by Brian Milton (<a href="https://github.com/PachiSystems/JSEmu">Source</a> may target several CPUs)</li> <li><a href="https://xem.github.io/chip8/">Chip8 1k</a> - a code-golfed chip 8 emulator in 1k by Maxime Euzière and others</li> <li><a href="http://em.ulat.es/machines/Chip8">EMF Chip 8</a> - EMF-based Elliott Emulator - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-chip-8">Source</a>)</li> <li><a href="http://em.ulat.es/machines/Megaprocessor">EMF Megaprocessor</a> - EMF-based Megaprocessor Emulator - by Steven Goodwin (@MarquisdeGeek) of the room-sized machine built by James Newman and housed in the Museum of Computing History, in Cambridge, England. (<a href="https://github.com/MarquisdeGeek/emf-emulator-megaprocessor">Source</a>)</li> <li><a href="https://github.com/doodlewind/merry8">Merry8</a> - JavaScript Chip-8 emulator by Yifeng Wang</li> <li><a href="http://compucolor.org/emu/ccemu.html">Compucolor II Emulator</a> - JavaScript Compucolor II Emulator</li> <li><a href="http://www.avoncliff.com/uk101/">Compukit UK101</a> - by David Stevenson</li> <li><a href="https://archives.loomcom.com/contraltojs/">ContrAltoJS</a> - Pure JavaScript implementation of the ContrAlto Xerox Alto emulator (<a href="https://github.com/sethm/ContrAltoJS">Source</a>)</li> <li><a href="http://www.donnelly-house.net/programming/cdp1802/simelf/">COSMAC Elf-ish</a> - simulator by William Donnelly</li> <li><a href="http://dcmo5.free.fr/online/">DCMO5 Online</a> - Thomson MO5 JavaScript emulator</li> <li><a href="https://play.dodolabs.io/?code=51447f62">Dodo Playground</a> - IDE and simulator for 6502-based Dodo homebrew game system by Peter Noyes</li> <li><a href="http://em.ulat.es/machines/Dragon32">EMF Dragon</a> - EMF-based Dragon Emulator - by Steven Goodwin (@MarquisdeGeek)</li> <li><a href="http://em.ulat.es/machines/JupiterAce">EMF Jupiter Ace</a> - EMF-based ZX80 Emulator - by Steven Goodwin (@MarquisdeGeek) (<a href="https://github.com/MarquisdeGeek/emf-emulator-jupiter-ace">Source</a>)</li> <li><a href="http://ep.lgb.hu/jsep/demo/?autostart=yes">Enterprise-128 JavaScript Emulator</a> - by Gábor Lénárt "LGB", based on JSSpeccy</li> <li><a href="https://corax89.github.io/esp8266Game/index.html">ESP8266 Game Engine</a> - a console-game-oriented virtual machine running on ESP8266, by Corax. (<a href="https://github.com/corax89/esp8266_game_engine">Source</a>)</li> <li><a href="https://gigatron.io/emu/">Gigatron</a> - emulator of the present-day TTL console computer with Tiny Basic and games. Also Wozmon on a nested emulation of 6502. (<a href="https://gigatron.io/">Website</a>)</li> <li><a href="https://diversen.github.io/hack-emulator-js/">Hack</a> - emulator of the computer from <a href="https://nand2tetris.org">nand2tetris</a>, by diversen. (<a href="https://github.com/diversen/hack-emulator-js">Source</a>.)</li> <li><a href="http://phils-place.co.uk/demos/et3400/">Heathkit ET3400</a> - by Phil Mainwaring. Click "Do" then "0000". (<a href="http://phils-place.co.uk/demos/et3400/heathkit_3400_trainer_manual_pages_46_to_104_edit_1.txt">Instructions</a>)</li> <li><a href="https://norbertkehrer.github.io/ibm_5110/emu5110.html">IBM 5110 Emulator</a> by Norbert Kehrer runs Basic or APL.</li> <li><a href="https://js99er.net/">JS99'er</a> - TI-99/4A emulator written in JavaScript (<a href="https://github.com/Rasmus-M/Js99er">Source</a>)</li> <li><a href="http://heathkit.garlanger.com/emulator/jsH89/">jsH89</a> - Heathkit H89 emulator (runs CP/M) by Mark Garlanger</li> <li>JSGS - Experimental JavaScript implementation of the Pico-8 fantasy console (<a href="https://github.com/burakcan/jsgs">Source</a>)</li> <li><a href="http://jsmsx.sourceforge.net/">jsMSX</a> - The first MSX emulator 100% written in JavaScript</li> <li><a href="http://jspspemu.com">JsPspEmu</a> - JavaScript PSP emulator (<a href="https://github.com/jspspemu/jspspemu">Source</a>)</li> <li><a href="http://www.twitchasylum.com/jsvecx/">JSVecX</a> - JavaScript port of the VecX Vectrex emulator</li> <li><a href="http://jupiler.zxtres.com">jupiler</a> - Jupiter Ace emulator written in JavaScript (<a href="https://sourceforge.net/p/emuscriptoria/code/HEAD/tree/jupiler.js">Source</a>)</li> <li><a href="http://hp.vector.co.jp/authors/VA016157/kmz80web10/kmz80web.html">KM-Z80 web</a> emulator for Sharp MZ-80K, by Katsumi Morimatsu. GOTO $1200 to start KM-BASIC. (<a href="http://hp.vector.co.jp/authors/VA016157/kmz80web10/">More information</a>)</li> <li><a href="https://nippur72.github.io/laser500emu/">laser500emu</a> emulator for Video Technology (VTech) Laser 350/500/700, by Antonino Porcino.</li> <li><a href="http://matt.krutar.org/LMC4/">Little Man Computer</a> a minimal CPU for teaching - emulator by Matthew Krutar. (<a href="https://en.wikipedia.org/wiki/Little_man_computer">Background</a>)</li> <li><a href="https://blog.paulhankin.net/lmc/lmc.html">Little Man Computer</a> a minimal CPU for teaching - emulator by Paul Hankin. (<a href="https://en.wikipedia.org/wiki/Little_man_computer">Background</a>)</li> <li><a href="https://peterhigginson.co.uk/LMC/">Little Man Computer</a> a minimal CPU for teaching - emulator by Peter Higginson. (<a href="https://en.wikipedia.org/wiki/Little_man_computer">Background</a>)</li> <li><a href="https://js1k.com/2010-first/details/455">Machine in a machine</a> - Turing Machine Implementation in 1k from the <a href="https://js1k.com/">JS1k competition</a></li> <li><a href="https://js1k.com/2016-elemental/details/2415">Marmmodore-1K</a> - minimal 8-bit computer in 1k by Felipe Alfonso</li> <li><a href="http://nanowasp.org">NanoWasp</a> - A MicroBee emulator</li> <li><a href="http://thorn.ws/jsnascom/jsnascom.html">Nascom 2 emulator</a> by Tommy Thorn. (J to start Basic) (<a href="https://github.com/tommythorn/jsnascom">Source</a>)</li> <li><a href="https://www.cs.drexel.edu/~bls96/oisc/OISC.html">One Instruction Set Computer (OISC)</a> by Peter Crampton, presented by Brian L. Stuart. (<a href="https://www.cs.drexel.edu/~bls96/oisc/">Explanation</a>)</li> <li><a href="http://orao.hrvoje.org/">Orao</a> - Orao emulator by Hrvoje Cavrak (<a href="https://github.com/hrvach/OraoJs">More information</a>)</li> <li><a href="https://torguet.net/oric/Oricutron.html">Oricutron</a> - An emulator for the ORIC series of home computers, by Peter Gordon (<a href="https://github.com/pete-gordon/oricutron">Source</a>)</li> <li>PC-01 Lviv - An emulator for the PC-01 Lviv (Ukrainian home computer) (<a href="https://github.com/dolgarev/emulator-pc01-lviv">Source</a>)</li> <li><a href="https://www.peterhigginson.co.uk/RISC/">RISC Simulator</a> - RISC Simulator with Fetch/Execute and register based CPU model, by Peter Higginson</li> <li><a href="https://pebble.github.io/rockyjs/">RockyJS</a> - Pebble watch interpreter/emulator</li> <li><a href="https://rk86.ru">Radio-86RK</a> Radio-86RK emulator in JavaScript (Intel 8080 based 8-bit Russian home computer) (<a href="https://github.com/begoon/rk86-js">Source</a>)</li> <li><a href="http://www.edmundgriffiths.com/setunka.html">Setunka</a> - Ternary computing for the browser</li> <li><a href="https://ellisgl.github.io/SAP-1-CPU/">SAP-1</a> JavaScript emulation of the Simple As Possible TTL machine from Malvino's book, Digital Computer Electronics. By Kenneth Ellis McCall. (<a href="https://github.com/ellisgl/SAP-1-CPU">Source</a>)</li> <li><a href="https://www.tejotron.com/">Tejotron</a> - Virtual breadboard, inspired by Ben Eater's 6502 computer</li> <li><a href="https://github.com/lkesteloot/turbopascal">Turbo Pascal</a> - A web-based Pascal compiler that runs a subset of Turbo Pascal 5.5 code</li> <li><a href="http://sensi.org/scalar/ware/468/">Vector06js</a> - Vector-06C (AKA Вектор-06Ц) JavaScript emulator by Viacheslav Slavinsky. Also runnable <a href="https://svofski.github.io/vector06js/">here</a>. (<a href="https://github.com/svofski/vector06js">Source</a>.)</li> <li>Virt.js - JavaScript emulation library (<a href="https://github.com/arcanis/virtjs">Source</a>)</li> <li><a href="http://www1.idc.ac.il/vic/software/computer/">Visual Computer</a> a minimal CPU for teaching by Shimon Schocken. (<a href="https://sites.google.com/site/schocken/thevisualcomputer">Web site</a>)</li> <li><a href="https://webmsx.org/">WebMSX</a> - WebMSX, or simply WMSX, is a new MSX emulator designed for the Web (<a href="https://github.com/ppeccin/webmsx">Source</a>)</li> <li><a href="https://dested.com/projects/wire/">Wireworld computer</a> - JavaScript port by Salvatore Aiello of the prime-generating computer implemented in the Wireworld cellular automaton, as described <a href="https://www.quinapalus.com/wi-index.html">here</a>.</li> <li><a href="https://github.com/neophob/wpc-emu">WPCEmu</a> - Williams Pinball Emulator by Michael Vogt</li> </ul> <h2>Adventure Game Engines</h2> <ul> <li><a href="https://github.com/juj/emscripten-scummvm">Emscripten ScummVM</a> - Emscripten fork of the ScummVM engine</li> <li><a href="https://github.com/curiousdannii/ifvms.js">IFVMS</a> - Infocom/Inform Web interpreter</li> <li><a href="https://github.com/curiousdannii/parchment">Parchment</a> - Infocom/Inform Web interpreter</li> <li><a href="https://github.com/bstreiff/zztjs">ZZTJS</a> - ZZT game engine in JavaScript</li> <li><a href="https://github.com/Utodev/ngPAWS">ngPAWS</a> - Professional Adventure Writer (PAW) Web interpreter</li> </ul> <h2>See also (lists of JavaScript emulators elsewhere)</h2> <ul> <li><a href="https://archive.vg/blog/a-big-list-of-browser-based-emulators-and-ports-of-classic-games">A Big List of Browser-Based Emulators</a> by Richard Moss</li> <li><a href="http://torinak.com/qaop/links">Qaop/JS – Emulator links</a> by Jan Bobrowski</li> </ul> https://tedneward.github.io/Research/vms/jvm/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/index/index.html <p>Bytecode interpreted. JIT compiled (depending on implementation). Dynamic code loaded.</p> <p><a href="https://en.wikipedia.org/wiki/Java_virtual_machine">Wikipedia</a></p> <p><a href="/platforms/jvm/specifications">JVM Specifications</a></p> <p><a href="https://github.com/jtransc/jtransc">JVM conversion</a>: Bytecode to source converting Java &amp; Kotlin code into JavaScript, C++, D, C#, PHP, AS3, Dart and Haxe and run it everywhere. Also use JVM code in your favourite language as a library. <a href="https://jtransc.soywiz.com/">Website</a></p> <h2>Articles</h2> <ul> <li><a href="https://www.kdgregory.com/index.php?page=java.refobj">Java Reference Objects</a>: One of the best articles on the Java "Reference" types (SoftReference, WeakReference, PhantomReference, ReferenceQueue) that I've ever read</li> <li><a href="/reading/memory-management">General memory mgmt/GC reading and links</a></li> <li><a href="https://inside.java/">Inside Java</a>: News and views from members of the Java team at Oracle <ul> <li><a href="https://inside.java/sip/">Sip of Java</a>: Short takes on interesting Java ideas</li> <li><a href="https://openjdk.org/projects/babylon/articles/linq">Emulating C# LINQ using Code Reflection</a></li> </ul> </li> </ul> <h2>Implementations:</h2> <ul> <li><a href="/vms/jvm/openjdk">OpenJDK</a></li> <li><a href="https://www.eclipse.org/openj9/">OpenJ9</a> (<a href="https://github.com/eclipse/openj9">Source</a>)</li> <li><a href="https://adoptopenjdk.net/">AdoptOpenJDK</a> builds JDK downloads of OpenJDK and OpenJ9 builds.</li> <li><a href="/vms/jvm/graalvm">GraalVM</a></li> <li>Zulu</li> <li>Amazon Coretto</li> <li><a href="https://www.oracle.com/java/jrockit.html">JRockit</a></li> <li><a href="http://www.kaffe.org/">Kaffe</a> (<a href="https://github.com/kaffe/kaffe">Source</a>)</li> <li><a href="https://github.com/alblue/jvmulator">jvmulator</a>: "a simple emulator for Java bytecode as well as an in-memory Java compiler to allow bytecode to be generated. The generated code can be executed as well as emulated to allow stepping through bytecode line by line, and seeing what the content of the local variables or stack happens to be."</li> </ul> <p>WebAssembly:</p> <ul> <li><a href="https://github.com/konsoletyper/teavm">TeaVM</a> - an ahead-of-time translating compiler (transpiler) of Java bytecode, that's capable of emitting JavaScript and WebAssembly.</li> <li><a href="https://github.com/i-net-software/JWebAssembly">JWebAssembly</a> - A Java bytecode to WebAssembly compiler. It can generate the WebAssembly binary or text format. It is written in Java itself and can be integrated with other Java build tools.</li> <li><a href="https://github.com/mirkosertic/Bytecoder">Bytecoder</a> - A Rich Domain Model for Java Bytecode and Framework to interpret and transpile it to other languages such as JavaScript, OpenCL or WebAssembly.</li> <li><a href="https://github.com/leaningtech/cheerpj-meta">CheerpJ</a> - A Java compiler for the web that converts any Java client application into standard HTML5/WebAssembly/JavaScript.</li> <li><a href="https://github.com/AmazingRise/rise-jvm">Rise JVM</a> - Rise JVM is a minimal Java VM based on WASM. You can try it out <a href="https://risehere.net/rise-jvm">here</a>.</li> </ul> <p>Research/Experiments:</p> <ul> <li><a href="../maxine-vm">Maxine</a>: Research JVM; "meta-circular" (meaning, JVM written in Java)</li> <li><a href="https://github.com/tomatsu/squawk">Squawk</a>: A JVM for micro-devices <a href="https://en.wikipedia.org/wiki/Squawk_virtual_machine">Wikipedia</a></li> <li><a href="https://www.jikesrvm.org/">Jikes RVM</a>: Research JVM; "meta-circular" (meaning, JVM written in Java) <a href="https://en.wikipedia.org/wiki/Jikes_RVM">Wikipedia</a> <a href="https://github.com/JikesRVM">Source</a></li> <li><a href="https://github.com/xranby/jamvm">JamVM</a>: "JamVM is a new Java Virtual Machine conforming to the JVM specification edition 2 (blue book). (Clone of <a href="git://git.code.sf.net/p/jamvm/code">git://git.code.sf.net/p/jamvm/code</a>)"</li> <li><a href="https://inside.java/">Inside Java</a>: News and views from members of the Java team at Oracle</li> </ul> <h2>Support/tooling</h2> <ul> <li><a href="https://github.com/corretto/heapothesys">HeapOTheSys</a>: a heap allocation JVM benchmark developed by the Amazon Corretto team</li> </ul> https://tedneward.github.io/Research/vms/jvm/lljvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/lljvm/index.html <p><a href="https://github.com/davidar/lljvm">Source</a></p> <p>Unmaintained but looks like a good source of research/investigation.</p> https://tedneward.github.io/Research/vms/jvm/specifications/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/specifications/index.html <p>(Oracle) Java Virtal Machine Specification (JVMS):</p> <ul> <li><a href="https://docs.oracle.com/javase/specs/jvms/se17/html/index.html">JVM Spec, Java SE 17</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se17/jvms17.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se16/html/index.html">JVM Spec, Java SE 16</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se16/jvms16.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se15/html/index.html">JVM Spec, Java SE 15</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se15/jvms15.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se14/html/index.html">JVM Spec, Java SE 14</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se14/jvms14.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se13/html/index.html">JVM Spec, Java SE 13</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se13/jvms13.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se12/html/index.html">JVM Spec, Java SE 12</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se12/jvms12.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se11/html/index.html">JVM Spec, Java SE 11</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se11/jvms11.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se10/html/index.html">JVM Spec, Java SE 10</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se10/jvms10.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se9/html/index.html">JVM Spec, Java SE 9</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se9/jvms9.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se8/html/index.html">JVM Spec, Java SE 8</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se8/jvms8.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jls/se7/html/index.html">JVM Spec, Java SE 7</a> (<a href="https://docs.oracle.com/javase/specs/jvms/se7/jvms7.pdf">PDF</a>)</li> <li><a href="https://docs.oracle.com/javase/specs/jvms/se6/html/VMSpecTOC.doc.html">JVM Spec, 2nd Edition (Java SE 6)</a></li> </ul> <p>(Previous-to-Java8 versions are on the Oracle website; I don't know that anybody should care about versions prior to 8. Prior to Java6, the VM was essentially unchanged since Java2/JDK 1.2.)</p> https://tedneward.github.io/Research/vms/luavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/luavm/index.html <ul> <li> <p><a href="http://luaforge.net/docman/view…/ANoFrillsIntroToLua51VMInstructions.pdf">A No-Frills Introduction to Lua 5.1 VM Instructions</a>, by Kein-Hong Man.</p> </li> <li> <p><a href="http://www.lua.org/doc/jucs05.pdf">The Implementation of Lua 5.0</a>, by Roberto Ierusalimschy et al.</p> </li> <li> <p><a href="https://sillycross.github.io/2022/11/22/2022-11-22/">Building the fastest Lua interpreter automatically!</a></p> </li> </ul> https://tedneward.github.io/Research/vms/mri/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/mri/index.html <h2>Articles/Posts</h2> <ul> <li><a href="https://www.rubyguides.com/2016/02/exploring-mri/">Exploring MRI</a></li> </ul> https://tedneward.github.io/Research/vms/omr/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/omr/index.html <p><a href="https://www.eclipse.org/omr/">Website</a> | <a href="https://github.com/eclipse/omr">Github</a></p> <p>From the website:</p> <blockquote> <p>OMR is enterprise-class runtime technology that delivers high performance without sacrificing versatility or robustness. OMR components are built to be extensible. It is easy to get started, and then to incrementally add powerful capabilities to your runtime. OMR takes care of the core capabilities that are common to all runtimes, so that you can focus on the unique requirements of your language.</p> </blockquote> https://tedneward.github.io/Research/vms/pl0/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/pl0/index.html <p><a href="https://programming.dojo.net.nz/study/pl0-language-tools/index">Website</a> | <a href="https://github.com/oriontransfer/PL0-Language-Tools">Source</a></p> <p>PL/0 is a collection of tools for learning about parsers, interpreters, compilers and virtual machines. It has been designed from the ground up to be a minimal implementation of several common interpreter and compiler functions, including the following:</p> <ul> <li>Lexical analysis and tokenisation.</li> <li>Recursive descent parsing to an abstract syntax tree.</li> <li>Direct syntax-tree interpretation.</li> <li>Static compilation to a stack based assembly language.</li> <li>Assembly from symbolic assembly language to virtual op-codes.</li> <li>Virtual machine execution using a stack-based computer.</li> </ul> https://tedneward.github.io/Research/vms/js/quickjs/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/quickjs/index.html <p><a href="https://bellard.org/quickjs/">Website</a> | <a href="https://github.com/bellard/quickjs">Source</a></p> <p>Small and easily embeddable: just a few C files, no external dependency, 210 KiB of x86 code for a simple hello world program. An online demonstration of the QuickJS engine with its mathematical extensions is available at numcalc.com. It was compiled from C to WASM/asm.js with Emscripten. <em>(There is something deeply satisfying about compiling a JS engine to WASM and running JS inside of JS.)</em></p> https://tedneward.github.io/Research/vms/jvm/avian/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/avian/index.html <p><a href="https://readytalk.github.io/avian/">Website</a> | <a href="https://github.com/ReadyTalk/avian">Source</a></p> <p>No longer maintained or supported.</p> https://tedneward.github.io/Research/vms/jvm/jcip/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/jcip/index.html <p><em>(by Brian Goetz)</em></p> <h4>Chapter 1: Introduction</h4> <ul> <li><em>Safety</em> means "nothing bad ever happens," while <em>liveness</em> means "something good eventually happens."</li> <li>Compromising safety often means compromising correctness. An example is a race condition between multiple threads.</li> <li>A liveness failure in a single-threaded program can be an infinite loop. In a multithreaded program, it could be deadlock, starvation, or livelock.</li> <li>While liveness means that something good <em>eventually</em> happens, performance means that it will happen <em>quickly</em>.</li> <li>Context switches have significant costs, including saving and restoring execution context, less of locality, and spending CPU time on scheduling threads instead of running them.</li> </ul> <h4>Chapter 2: Thread Safety</h4> <ul> <li>Writing thread-safe code is, at its core, about managing access to shared, mutable state.</li> <li>Whenever more than one thread accesses a given state variable, and one of them might write to it, they all must coordinate their access to it using synchronization.</li> <li>A broken program shares mutable state without synchronization. Either don't share the state, make it immutable, or use synchronization upon every access.</li> <li>It is far easier to design a class to be thread-safe than to retrofit it for safety later.</li> <li>The same object-oriented techniques that help you write well-organized, maintainable classes -- such as encapsulation and data hiding -- can also help you create thread-safe classes.</li> <li>A program that consists entirely of thread-safe classes may not be thread-safe, and a thread-safe program may contain classes that are not thread-safe.</li> </ul> <h5>2.1: What is thread safety?</h5> <ul> <li>A class is thread-safe if it behaves correctly when accessed from multiple threads, regardless of their scheduling or interleaving of execution, and with no additional synchronization or coordination by the calling code.</li> <li>Thread-safe classes encapsulate any needed synchronization so that clients need not provide their own.</li> </ul> <h5>2.2: Atomicity</h5> <ul> <li>A race condition occurs when the correctness of a computation depends on the relative timing or interleaving of multiple threads.</li> <li>The most common type of race condition is <em>check-then-act</em>, where an observation could have become invalid between the time you observed it and the time you acted on it, causing a problem.</li> <li>Read-modify-write operations require knowing a previous value and ensuring that no one else changes or uses that value while you are in mid-update.</li> <li>Check-then-act and read-modify-write sequences are compound actions, or sequences that must be executed atomically in order to remain thread-safe.</li> </ul> <h5>2.3: Locking</h5> <ul> <li>Mutual exclusion locks, or mutexes, means that at most one thread may own a lock. If thread A attempts to acquire a lock by thread B, it blocks until B releases it.</li> <li>Reentrancy means that locks are acquired on a per-thread rather than per-invocation basis. Each lock is associated with an acquisition count and an owning thread.</li> </ul> <h5>2.4: Guarding state with locks</h5> <ul> <li>For each mutable state variable that may be accessed by more than one thread, all accesses to that variable must be performed by the same lock held. The variable is <em>guarded</em> by the lock.</li> <li>When a class has invariants that involve more than one state variable, we must ensure that each variable participating in the invariant must be guarded by the same lock.</li> </ul> <h5>2.5: Liveness and performance</h5> <ul> <li>Acquiring and releasing a lock has some overhead, so do not break down guarded blocks too far, even if this would not compromise atomicity.</li> <li>When implementing a synchronization policy, resist the temptation to prematurely sacrifice simplicity (potentially compromising safety) for the sake of performance.</li> <li>Avoid holding locks during lengthy computations or operations at risk of not completing quickly, such as when performing disk or network I/O.</li> </ul> <h4>Chapter 3: Sharing Objects</h4> <h5>3.1: Visibility</h5> <ul> <li>To ensure visibility of memory writes across threads, you must use synchronization.</li> <li>There is no guarantee that operations in one thread will be performed in the order given by the program, as long as the reordering is not detectable from within <em>that</em> thread.</li> <li>Because of reordering, attempts to reason about the order in which memory actions "must" happen in insufficiently synchronized multithreaded programs will almost certainly be incorrect.</li> <li>Unless synchronization is used every time a variable is accessed, it is possible for a thread to read a stale value for that variable.</li> <li>Stale data can cause serious and confusing failures such as unexpected exceptions, corrupted data structures, inaccurate computations, and infinite loops.</li> <li><em>Out-of-thin-air safety</em> is when a variable reads a thread without synchronization, it reads a value that was actually written by another thread instead of some random value.</li> <li>With the Java Memory Model, the values of variables that were visible to thread A prior to releasing a lock are guaranteed to be visible to thread B upon acquiring the same lock.</li> <li>Variables marked with the <code>volatile</code> keyword are not reordered with other memory operations, and are not put in caches where they are hidden from other processors.</li> <li>A <code>volatile</code> variable can be used for a completion, interruption, or status flag, but the semantics are not strong enough to make the increment operation atomic, for example.</li> </ul> <h5>3.2: Publication and escape</h5> <ul> <li><em>Publishing</em> an object means making it available to code outside of its current scope. An object that is published when it should not have been is said to have <em>escaped</em>.</li> <li>Any object that is <em>reachable</em> from a published object by following some chain of nonprivate field references and method calls has also been published.</li> <li>Once an object escapes, you have to assume that another class or thread may, maliciously or carelessly, misuse it. This is a compelling reason to use encapsulation.</li> <li>An object is in a predictable, consistent state only after its constructor returns, so publishing an object (via <code>this</code>) from within its constructor can publish an incompletely constructed object.</li> <li>You can avoid this improper construction by using a private constructor and a public static factory method.</li> </ul> <h5>3.3: Thread confinement</h5> <ul> <li>If data is only accessed by a single thread, then no synchronization is needed. This technique is called <em>thread confinement</em>.</li> <li><em>Ad-hoc thread confinement</em> is when the responsibility for maintaining thread confinement falls entirely on the implementation.</li> <li>Thread confinement is often a consequence of deciding to implement a particular subsystem, such as the GUI, as a single thread. The simplicity benefit of such a system outweighs the fragility of ad-hoc thread confinement.</li> <li><em>Stack confinement</em> is a special case of thread confinement in which an object can only be reached through local variables. Local variables are intrinsically confined to the executing thread.</li> </ul> <h5>3.4: Immutability</h5> <ul> <li>Immutable objects are simple, because they can only have one state. Immutable objects are also safe, because you can freely share and publish them without the need to make defensive copies.</li> <li>An object is immutable if its state cannot be modified after construction, all its fields are <code>final</code>, and it is properly constructed, i.e. the <code>this</code> reference does not escape during construction.</li> <li>Whenever a group of related data items but be acted upon atomically, consider creating an immutable holder class for them.</li> </ul> <h5>3.5: Safe publication</h5> <ul> <li>Simply storing a reference to an object into a public field is not enough to publish that object safely. Improper publication allows another thread to observe a partially constructed object.</li> <li>However, immutable objects can be used safely by any thread without additional synchronization, even when synchronization is not used to publish them.</li> <li>To publish an object safely, both the reference to the object and the object's state must be made visible to other threads at the same time.</li> <li>Objects that are not technically immutable, but whose state will not be modified after publication, are called <em>effectively immutable</em>. Such objects, when safely published, can be used safely by any thread without additional synchronization.</li> <li>While effectively immutable objects must be safely published, mutable objects must be safely published, and mus the either thread-safe or guarded by a lock.</li> <li>Many concurrency errors stem from failing to understand the "rules of engagement" for a shared object. When you publish an object, document how it should be accessed.</li> </ul> <h4>Chapter 4: Composing Objects</h4> <h5>4.1: Designing a thread-safe class</h5> <ul> <li>Designing a thread-safe class requires identifying the invariants that constrain the state variables, and establishing a policy for managing concurrent access to them.</li> <li>The smaller the <em>state space</em> of an object or variable, the easier it is to reason about. Use <code>final</code> fields to reduce the state space.</li> <li>Operations with state-based preconditions are called <em>state-dependent</em>.</li> <li>Ownership implies control, but once you publish a reference to a mutable object, you no longer have exclusive control, but at best "shared ownership."</li> </ul> <h5>4.2: Instance confinement</h5> <ul> <li>Encapsulating data within an object confines access to the data to the object's methods, making it easier to ensure that the data is always accessed with the appropriate lock held.</li> <li>Instance confinement also allows different state variables to be held by different locks.</li> <li>Confined objects can also escape by publishing other objects such as iterators or inner class instances that may indirectly publish the confined objects.</li> <li>Using a private lock prohibits client code from acquiring it, whereas a publicly accessible lock allows client code to participate in its synchronization policy, perhaps incorrectly.</li> </ul> <h5>4.3: Delegating thread safety</h5> <ul> <li>A class with multiple independent thread-safe state variables and no operations that have any invalid state transitions can delegate thread safety to the underlying state variables.</li> <li>If a state variable is thread-safe, does not participate in any invariants that constrain its value, and has no prohibited state transitions for any of its operations, then it can be safely published.</li> </ul> <h5>4.4: Adding functionality to existing thread-safe classes</h5> <ul> <li>Extending a class to support a thread-safe operation is more fragile than adding code directly to the class, as its synchronization policy is now distributed over multiple source files.</li> <li>Just as subclassing violates encapsulation of implementation, client-side locking violates encapsulation of synchronization policy.</li> </ul> <h5>4.5: Documenting synchronization policies</h5> <ul> <li>Document a class's thread safety guarantees for its clients; document its synchronization policy for its maintainers.</li> <li>If you want clients to be able to create new atomic operations on your class, you must document which locks they should acquire to do so safely.</li> <li>If you must guess whether a class is thread-safe, improve the quality of your guess by interpreting the specification by someone who must implement it versus someone who will merely use it.</li> </ul> <h4>Chapter 10: Avoiding Liveness Hazards</h4> <h5>10.1: Deadlock</h5> <ul> <li>When thread A holds lock L and tries to acquire lock M, but at the same time thread B holds lock M and tries to acquire L, this is deadlock, or <em>deadly embrace</em>.</li> <li>When a database system detects that a set of transactions has deadlocked by searching the is-waiting-for graph for cycles, it picks a victim and aborts that transaction, thereby releasing its held locks.</li> <li>A program will be free of lock-ordering deadlocks if all threads acquire the locks they need in a fixed global order. Sometimes we must induce this ordering.</li> <li>Invoking an alien method with a lock held is asking for liveness trouble. That method might risk deadlock by acquiring other locks, or block for an unexpectedly long time and stall other threads on the held lock.</li> <li>Calling a method with no locks held is called an <em>open call</em>, and classes that rely on open calls are more well-behaved and composable than classes that make calls with locks held.</li> <li>Resource deadlocks occur when thread A holds resource X and tries to acquire resource Y, but at the same time thread B holds resource Y and tries to acquire resource X.</li> </ul> <h5>10.2: Avoiding and diagnosing deadlocks</h5> <ul> <li>Try acquiring locks with a timeout. By using a timeout that is much longer than you expect acquiring the lock to take, you can regain control when something unexpected happens.</li> <li>Thread dumps not only include a stack trace for each running thread, but locking information such as which locks are held by a thread, and which lock a thread is waiting to acquire.</li> </ul> <h5>10.3: Other liveness hazards</h5> <ul> <li><em>Starvation</em> is when a thread is perpetually denied access to resources it needs in order to make progress; the most commonly starved resource is CPU cycles.</li> <li><em>Livelock</em> is a liveness failure in which a thread, while not blocked, still cannot make progress because it keeps retrying an operation that will always fail.</li> <li>Livelock can occur when multiple cooperating threads change their state in response to the others in such a way that no thread can ever make progress. The solution is to introduce randomness into the retry mechanism.</li> </ul> <h3>Chapter 11: Performance and Scalability</h3> <h5>11.1: Thinking about performance</h5> <ul> <li>Improving performance means doing more work with fewer resources. When performance of an activity is limited by availability of a particular resource, it is <em>bound</em> by that resource.</li> <li>Using concurrency to achieve better performance means using the processing resources we have more effectively, and enable a program to exploit additional processing resources that become available.</li> <li><em>Scalability</em> describes the ability to improve throughput or capacity when additional computing resources (such as CPUs, memory, storage, or I/O bandwidth) are added.</li> <li>The "how much" aspects like scalability, throughput, and capacity are concern for server applications. The "how fast" aspects like service time or latency are concern for client applications.</li> <li>Most optimizations are premature because they are often undertaken before a clear set of requirements is available.</li> <li>Make it right, then make it fast. And if attempting to make it fast, measure. Don't guess.</li> </ul> <h5>11.2: Amdahl's law</h5> <ul> <li>Amdahl's law describes how much a program can theoretically be sped up by additional computing resources, based on the proportion of parallelizable to serial components.</li> <li>If F is the faction of the calculation that must be executed serially, then on a machine with N processors, we can achieve a speedup of most: 1/(F+(1-F)/N).</li> <li>When evaluating an algorithm, thinking "in the limit" about what would happen with hundreds or thousands of processors can offer some insight into where scaling limits might appear.</li> </ul> <h5>11.3: Costs introduced by threads</h5> <ul> <li>When a new thread is switched in, the data it needs is unlikely to be in the local processor cache, and so a context switch causes a flurry of cache misses and runs a little slower at first.</li> <li>Schedulers give each runnable thread a certain minimum time quantum, thereby amortizing the cost of the context switch and its consequences over more interrupted execution time.</li> <li>A program that does more blocking has more of its threads suspended and switched out. The program therefore incurs more context switches, increasing scheduling overhead and reducing throughput.</li> <li>Special instructions called <em>memory barriers</em> can flush or invalidate caches and flush hardware write buffers. They inhibit compiler optimizations; most operations cannot be reordered with them.</li> <li><em>Lock elision</em> optimizes away lock acquisitions. <em>Lock coarsening</em> merges together adjacent blocks holding the same lock, reducing synchronization overhead and helping the optimizer.</li> <li>When a lock is contended, the losing threads must block. This can be implemented either by <em>spin-waiting</em> or by <em>suspending</em> the blocked thread through the operating system.</li> </ul> <h5>11.4: Reducing lock contention</h5> <ul> <li>Two factors influence the likelihood of contention for a lock: How often that lock is requested, and how long it is held once acquired.</li> <li><em>Lock splitting</em> and <em>lock striping</em> involve using separate locks to guard multiple independent state variables previously guarded by a single lock.</li> <li>Splitting a lock into two offers the greatest possibility for improvement when a lock is experiencing moderate but not heavy contention.</li> <li>Lock striping extends lock splitting by partitioning locking on a variable-sized set of independent objects. But locking the collection for exclusive access is more difficult and costly.</li> <li>If your class has a small number of hot fields that do not participate in invariants with other variables, then replacing them with atomic variables may improve scalability.</li> <li>Tools like <code>vmstat</code> or <code>mpstat</code> can show whether your application is CPU-bound, while tools like <code>iostat</code> or <code>perfmon</code> can show whether your application is I/O-bound.</li> <li>The tool <code>vmstat</code> has a column reporting the number of threads that are runnable but not currently running because a CPU is not available.</li> </ul> https://tedneward.github.io/Research/vms/jvm/maxine-vm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/maxine-vm/index.html <p><a href="https://github.com/beehive-lab/Maxine-VM">Github</a> | <a href="https://en.wikipedia.org/wiki/Maxine_Virtual_Machine">Wikipedia</a> | <a href="https://maxine-vm.readthedocs.io/_/downloads/en/stable/pdf/">Docs</a></p> <p>From the <a href="https://maxine-vm.readthedocs.io/en/latest/">docs page</a>:</p> <blockquote> <p>In this era of modern, managed languages we demand ever more from our virtual machines: better performance, more scalability, and support for the latest new languages. Research and experimentation is essential but challenging in the context of mature, complex, production VMs written in multiple languages. The Maxine VM is a next generation platform that establishes a new standard of productivity in this area of research. It is written entirely in Java, completely compatible with modern Java IDEs and the standard JDK, features a modular architecture that permits alternate implementations of subsystems such as GC and compilation to be plugged in, and is accompanied by a dedicated development tool (the Maxine Inspector) for debugging and visualizing nearly every aspect of the VM’s runtime state.</p> </blockquote> https://tedneward.github.io/Research/vms/jvm/tornadovm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jvm/tornadovm/index.html <p><a href="https://www.tornadovm.org/">Website</a> | <a href="https://github.com/beehive-lab/TornadoVM">Source</a></p> https://tedneward.github.io/Research/vms/mame/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/mame/index.html <p>From the website:</p> <blockquote> <p>MAME’s purpose is to preserve decades of software history. As electronic technology continues to rush forward, MAME prevents this important "vintage" software from being lost and forgotten. This is achieved by documenting the hardware and how it functions. The source code to MAME serves as this documentation. The fact that the software is usable serves primarily to validate the accuracy of the documentation (how else can you prove that you have recreated the hardware faithfully?). Over time, MAME (originally stood for Multiple Arcade Machine Emulator) absorbed the sister-project MESS (Multi Emulator Super System), so MAME now documents a wide variety of (mostly vintage) computers, video game consoles and calculators, in addition to the arcade video games that were its initial focus.</p> </blockquote> <p><a href="https://www.mamedev.org/">Website</a> | <a href="https://git.redump.net/mame/tree/">Source</a> and <a href="https://github.com/mamedev/mame">Source</a></p> https://tedneward.github.io/Research/vms/mu/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/mu/index.html <p><a href="https://microvm.github.io/">Website</a> | <a href="https://gitlab.anu.edu.au/mu/mu-spec">Specification</a> | <a href="https://mu-tutorial.readthedocs.io/en/latest/">Guide</a> | <a href="https://gitlab.anu.edu.au/groups/mu">Project</a></p> <p><em>Looks like the project paused/stopped development around 2019 or so</em></p> <h2>Implementations</h2> <ul> <li><a href="https://gitlab.anu.edu.au/mu/mu-impl-ref2">Holstein</a>: reference</li> <li><a href="https://gitlab.anu.edu.au/mu/mu-impl-fast">Zebu</a>: high-perf, in development</li> </ul> <p>A large fraction of today’s software is written in managed languages. These languages increase software productivity by offering rich abstractions for managing memory, executing code concurrently, and hiding the complexity of modern hardware. Examples include JavaScript, PHP, Objective-C, Java, C#, Python, and Ruby. These languages are economically important. Unfortunately, most of these languages are inefficient, imposing overheads as large as a factor of fifty compared to orthodox language choices such as C.</p> <p>The project will define, develop, evaluate, and refine the essential components of a new foundation layer for managed language implementation. In doing so, it will address a key source of systemic inefficiency, by pioneering micro virtual machines as an efficient high-performance substrate for managed language implementation. The relationship between a micro virtual machine and existing managed language implementations is analogous to the one between an operating system micro kernel and monolithic operating systems such as Linux. A micro virtual machine captures the insight that there exists a well-defined foundation common to most modern languages that can take responsibility for fundamental abstractions over hardware, concurrency, and memory. By isolating and exposing this substrate, a micro virtual machine embodies state-of-the-art base technology available to language implementers while isolating them from the pernicious complexities of these abstractions, freeing them to focus on all-important language-specific optimizations. This project will enable more efficient software and a distinctly sharper focus for language implementation research and development.</p> <h2>Resources</h2> <h3>Papers</h3> <ul> <li>Yi Lin, "An efficient implementation of a micro virtual machine", Ph.D. thesis, College of Engineering and Computer Science, The Australian National University, 2019. <a href="https://openresearch-repository.anu.edu.au/bitstream/1885/158122/1/Yi%20Lin%20Thesis%202019.pdf" title="Download pdf">pdf</a> <a href="https://doi.org/10.25911/5ca1dad991dbf">url</a></li> <li>K. Wang, S. M. Blackburn, A. L. Hosking, and M. Norrish, "Hop, Skip, &amp; Jump: Practical On-Stack Replacement for a Cross-Platform Language-Neutral VM", in 14th ACM SIGPLAN/SIGOPS International Conference on Virtual Execution Environments (VEE 2018), 2018. <a href="https://wks.github.io/downloads/pdf/osr-vee-2018.pdf">pdf</a> <a href="https://doi.org/10.1145/3186411.3186412">url</a></li> <li>Kunshan Wang, "Micro Virtual Machines: A Solid Foundation for Managed Language Implementation", Ph.D. thesis, College of Engineering and Computer Science, The Australian National University, 2017. <a href="https://openresearch-repository.anu.edu.au/bitstream/1885/147871/1/Wang%20Thesis%202018.pdf" title="Download pdf">pdf</a> <a href="http://hdl.handle.net/1885/147871">url</a></li> <li>Y. Lin, S. M. Blackburn, A. L. Hosking, and M. Norrish, "Rust as a Language for High Performance GC Implementation", in Proceedings of the Sixteenth ACM SIGPLAN International Symposium on Memory Management, ISMM ‘16, Santa Barbara, CA, June 13, 2016, 2016. <a href="http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rust-ismm-2016.pdf" title="Download pdf">pdf</a></li> <li>Y. Lin, K. Wang, S. M. Blackburn, M. Norrish, and A. L. Hosking, "Stop and Go: Understanding Yieldpoint Behavior", in Proceedings of the Fourteenth ACM SIGPLAN International Symposium on Memory Management, ISMM ‘15, Portland, OR, June 14, 2015, 2015. <a href="http://users.cecs.anu.edu.au/~steveb/downloads/pdf/yieldpoint-ismm-2015.pdf" title="Download pdf">pdf</a></li> <li>K. Wang, Y. Lin, S. M. Blackburn, M. Norrish, and A. L. Hosking, "Draining the Swamp: Micro Virtual Machines as Solid Foundation for Language Development", in 1st Summit on Advances in Programming Languages (SNAPL 2015), 2015. <a href="http://drops.dagstuhl.de/opus/volltexte/2015/5034/pdf/24.pdf">pdf</a> <a href="http://drops.dagstuhl.de/opus/frontdoor.php?source_opus=5034">url</a></li> </ul> https://tedneward.github.io/Research/vms/opensmalltalk/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/opensmalltalk/index.html <p><a href="http://opensmalltalk.org/">Website</a> | <a href="https://github.com/OpenSmalltalk/opensmalltalk-vm">Github</a></p> <p>First, opensmalltalk-vm (a.k.a. the Cog VM) is the virtual machine beneath the Cuis, Pharo and Squeak Smalltalk dialects.</p> <p>Second, the core VM, which comprises the execution engine and garbage collector, and the core plugins, is developed in Smalltalk, using the VM Simulator. This repository contains the code generated by the Simulator, and the platform support code for the entire VM, its CI infrastructure and so on. The core VM should not be developed by editing the generated code. The core VM should be developed using Smalltalk. The source code repository for the VM is <a href="http://source.squeak.org/VMMaker.html">http://source.squeak.org/VMMaker.html</a>.</p> <p>You can find scripts to build a Smalltalk image in which to do core VM development in the image directory in this repository. You can read about the Simulator here:</p> <ul> <li><a href="https://www.researchgate.net/publication/328509577_Two_Decades_of_Smalltalk_VM_Development_Live_VM_Development_through_Simulation_Tools">https://www.researchgate.net/publication/328509577_Two_Decades_of_Smalltalk_VM_Development_Live_VM_Development_through_Simulation_Tools</a></li> <li><a href="https://hal.archives-ouvertes.fr/hal-01883380/document">https://hal.archives-ouvertes.fr/hal-01883380/document</a> Please look at the wiki on this site for a description of on-going projects.</li> </ul> <p>Cog is an evolution of the Squeak Back-to-the-future Smalltalk virtual machine that provides a number of different Smalltalk virtual machines. The VMs are developed in Smalltalk, using all the dynamic and reflective facilities of the Squeak/Pharo Smalltalk system. As such, developing in Cog is a delight. The Smalltalk framework comprising the various Cog VMs is translated into C by its Slang component to produce VM source that is combined with platform-specific support sources and compiled via a C compiler to obtain a fast production VM. This directory tree includes the output of Slang for various configurations of "Cog VM" and the associated platform support code, plus build directories that can be used to produce production VMs.</p> <p>This directory tree also includes an instance of the Smalltalk Cog development system, suitable for developing the VM in Smalltalk, and for generating new VM sources.</p> https://tedneward.github.io/Research/vms/python/coco/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/python/coco/index.html <p>C++: <a href="https://kentdlee.github.io/CoCoPages/html/index.html">Website</a> | <a href="http://github.com/kentdlee/CoCo.git">Source</a></p> <p>Java: <a href="https://kentdlee.github.io/JCoCoPages/_build/html/index.html#">Website</a> | <a href="http://github.com/kentdlee/JCoCo">Source</a></p> <p>Used as part of Kent Lee's book on language implementations.</p> https://tedneward.github.io/Research/vms/corewar/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/corewar/index.html <p><a href="https://github.com/emilwallner/Corewar">Source</a></p> <p>Core War was inspired by a malicious virus written in the 80’s. To deal with the self-replicating virus, a white hat hacker invented Reaper. It was a virus designed to spread and eliminate the malware. He fought fire with fire.</p> <p>This inspired A. K. Dewdney to coin the idea for Core War.</p> <p>The idea was simple. You compete by designing viruses to overtake a computer. You win by protecting your own program and overwriting your opponent's programs. This is all happening on a virtual computer. Think, a simple computer within your computer.</p> <p>This is what is looks like in action:</p> <p><img src="https://camo.githubusercontent.com/ecdb2786375656c7e085c85e38421ea11946a2f553ce29a6817d2eb482307d85/687474703a2f2f672e7265636f726469742e636f2f7079794167675963576d2e676966" alt=""></p> <p>Let’s focus on the high-level game dynamics:</p> <ul> <li>The game board, the memory of our virtual computer. It’s represented in a 64 X 64 grid of bytes.</li> <li>The players, small programs represented in different colors. The white parts have blank memory.</li> <li>Cursors, the moving parts with inverted color. They read from the game board.</li> </ul> <p>The cursors have a couple of features. They can jump on the game board, store and write values, and clone themselves. The cursors belong to no-one, they just read from the game board.</p> <p>When the game starts, all players have one cursor at the beginning of their program. The game advances as the cursors read the commands stored in their programs. If they end up on a faulty command or a blank memory, it moves to the next byte.</p> <p>Below, we see how the pink player starts by cloning their cursors. It then starts attacking the blue player.</p> <p><img src="https://camo.githubusercontent.com/716b61e9b563a2925b9f89d6ba671a7d88c0eb2afceae9f26a7f368d7e7476c8/687474703a2f2f672e7265636f726469742e636f2f593972394537384656592e676966" alt=""></p> <p>Let’s get into a little bit more depth.</p> <p>Every byte you see, the pairs of numbers or letters, are represented in hexadecimal. Each hexadecimal has the value of a digit, from 0 - 255.</p> <p>There are 16 operations in the game. The operations decide if the cursor should jump to a different address, clone itself and so on. The first 16 hexadecimal are coding bytes, they all store a different operation. The remaining digits, 17 - 255, are regular numbers.</p> <p>The coding byte decides which operation to execute and how many bytes to read. In the image above, the pink player goes to the hexadecimal 0c. It's 12 in decimal, telling the cursor to clone itself. The two bytes after decide where the cloned cursor starts.</p> <p><img src="https://camo.githubusercontent.com/2998dc8f55ea9e1d22fbb9ce00389d9888e463acfe0852a2e5559b13d013454d/687474703a2f2f672e7265636f726469742e636f2f58516468566d717672562e676966" alt=""></p> <p>There are three main components to determine who wins:</p> <ul> <li>Game rounds, every game round is measured in cycles. It determines how much each cursor can read from the game board.</li> <li>Lives, if a cursor reads a player life from the game board, they are given a life for that round. These are visualized in the colorful progress bars.</li> <li>Cycle to die, for every game round, the number of bytes a cursor can read from the game board is reduced.</li> </ul> <p>You win if your program is the last one to receive a life.</p> <p>There are more nuances to the game that I don’t cover. The best way to learn about them is to run it.</p> https://tedneward.github.io/Research/vms/fex/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/fex/index.html <p><a href="https://fex-emu.com/">Website</a> | <a href="https://github.com/FEX-Emu/FEX">Source</a></p> <p>FEX allows you to run x86 applications on ARM64 Linux devices, similar to qemu-user and box64. It offers broad compatibility with both 32-bit and 64-bit binaries, and it can be used alongside Wine/Proton to play Windows games.</p> <p>It supports forwarding API calls to host system libraries like OpenGL or Vulkan to reduce emulation overhead. An experimental code cache helps minimize in-game stuttering as much as possible. Furthermore, a per-app configuration system allows tweaking performance per game, e.g. by skipping costly memory model emulation. We also provide a user-friendly FEXConfig GUI to explore and change these settings.</p> https://tedneward.github.io/Research/vms/hashlink/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/hashlink/index.html <p><a href="https://hashlink.haxe.org/">Website</a> | <a href="https://github.com/HaxeFoundation/hashlink/">Source</a></p> <p>Can be either executed directly or converted to C.</p> <p>Appears to replace <a href="./neko">NekoVM</a> in practice.</p> <p>Performance-oriented features:</p> <ul> <li>HL bytecode is strictly typed</li> <li>Fast anonymous objects access using typed "virtuals"</li> <li>Static dispatch of object methods when no override exists</li> <li>Compact memory representation for all values</li> <li>Inference to reduce null field access checks to the minimum (soon)</li> <li>Low level access for pointers, single float, 8 and 16 bits integers</li> </ul> https://tedneward.github.io/Research/vms/hlvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/hlvm/index.html <p><a href="https://forge.ocamlcore.org/projects/hlvm/">Website</a> (not much there) | <a href="https://github.com/Tipoca/hlvm">Source</a></p> <p>Welcome to the High-Level Virtual Machine (HLVM) project. This open source project is designed to provide a virtual machine to support modern statically-typed functional programming languages easily and with performance characteristics ideally suited to scientific computing.</p> <p>HLVM currently sports the following features:</p> <ul> <li>Unit, bool, int and float primitive types.</li> <li>Tuples.</li> <li>Arrays.</li> <li>Union types.</li> <li>Function pointers.</li> <li>Tail call elimination of all tail calls.</li> <li>Generic printing.</li> <li>Foreign function interface to call C directly.</li> <li>POSIX threads.</li> <li>Mark and sweep garbage collector that allows threads to run in parallel.</li> <li>Incremental JIT compilation to high-performance native code suitable for a REPL.</li> </ul> <p>HLVM aspires to provide the following features in the future:</p> <ul> <li>Closures.</li> <li>Parametric polymorphism.</li> <li>Global variables.</li> <li>Generic comparison, equality, hashing and serialization functions.</li> <li>A standard library.</li> </ul> https://tedneward.github.io/Research/vms/js/hermes/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/js/hermes/index.html <p><a href="https://hermesengine.dev/">Website</a> | <a href="https://github.com/facebook/hermes">Source</a></p> <p>It features ahead-of-time static optimization and compact bytecode.</p> https://tedneward.github.io/Research/vms/clr/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/clr/index/index.html <p>List of CLR VM implementations:<br> * <a href="/vms/clr/coreclr">CoreCLR</a>, part of <a href="/platforms/clr/dotnetcore">.NET Core</a><br> * <a href="https://www.mono-project.com/docs/advanced/runtime/">Mono</a>, which was acquired by Microsoft a few years ago | <a href="https://github.com/mono/mono">Source</a></p> <p>There used to be more (below), but interest appears to have waned after an early surge.</p> <ul> <li> <p>Shared Source CLI (aka "Rotor"); <a href="/vms/clr/sscli-essentials">SSCLI Essentials</a> is a good place to start to understand the CLR codebase ("runtime") itself.</p> </li> <li> <p><a href="https://github.com/lap1nou/CLR_Heap_encryption">CLR Heap Encryption</a> proof of concept</p> </li> </ul> <h2>Articles</h2> <ul> <li>Writing a .NET Profiler in C# <a href="https://minidump.net/writing-a-net-profiler-in-c-part-1-d3978aae9b12">Part 1</a> <a href="https://minidump.net/writing-a-net-profiler-in-c-part-2-8039da001e43">Part 2</a> <a href="https://minidump.net/writing-a-net-profiler-in-c-part-3-7d2c59fc017f">Part 3</a> <a href="https://minidump.net/writing-a-net-profiler-in-c-part-4-c54df903b9ce/">Part 4</a> <a href="https://minidump.net/writing-a-net-profiler-in-c-part-5/">Part 5</a></li> <li><a href="https://minidump.net/2025-28-01-writing-a-net-gc-in-c-part-1/">Writing a .NET Garbage Collector in C#</a></li> </ul> https://tedneward.github.io/Research/vms/dev/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/dev/index.html <h2>Reading</h2> <h3>Articles, Blogs, Essays</h3> <h3>Books</h3> <ul> <li>Crafting Interpreters: <ul> <li><a href="http://craftinginterpreters.com/a-virtual-machine.html#design-note">A Virtual Machine</a></li> <li><a href="http://craftinginterpreters.com/chunks-of-bytecode.html">Bytecode Chunks</a></li> </ul> </li> </ul> <h3>References</h3> <ul> <li><a href="http://wiki.luajit.org/Bytecode-2.0">LuaJIT Bytecode 2.0</a></li> <li><a href="https://docs.python.org/3/library/dis.html">Python3 dis module</a></li> <li><a href="http://www.complang.tuwien.ac.at/projects/interpreters.html">Anton Ertl's page</a></li> <li><a href="https://inko-lang.org/manual/virtual-machine/bytecode/">bytecode format for Inko</a></li> </ul> <h3>Papers</h3> <ul> <li><a href="https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf">https://www.usenix.org/legacy/events/vee05/full_papers/p153-yunhe.pdf</a></li> </ul> https://tedneward.github.io/Research/vms/firecracker/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/firecracker/index.html <p><a href="https://firecracker-microvm.github.io/">Website</a> | <a href="https://github.com/firecracker-microvm/firecracker">Source</a></p> https://tedneward.github.io/Research/vms/hercules/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/hercules/index.html <p><a href="http://www.hercules-390.eu/">Website</a></p> https://tedneward.github.io/Research/vms/hvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/hvm/index.html <p><a href="https://github.com/HigherOrderCO/HVM">Source</a> | <a href="https://github.com/HigherOrderCO/HVM1">HVM1 Source</a> | <a href="https://github.com/HigherOrderCO/HVM/blob/main/paper/PAPER.pdf">Paper</a></p> <p>This repository provides a low-level IR language for specifying the HVM2 nets and a compiler from that language to C and CUDA. It is not meant for direct human usage. If you're looking for a high-level language to interface with HVM2, check Bend instead.</p> <p>HVM is a low-level compile target for high-level languages. It provides a raw syntax for wiring interaction nets. For example:</p> <pre><code>@main = a &amp; @sum ~ (28 (0 a)) @sum = (?(((a a) @sum__C0) b) b) @sum__C0 = ({c a} ({$([*2] $([+1] d)) $([*2] $([+0] b))} f)) &amp;! @sum ~ (a (b $([+] $(e f)))) &amp;! @sum ~ (c (d e)) </code></pre> <p>The file above implements a recursive sum.</p> <hr> <h1>HVM1</h1> <p>Pretty hyperbolic:</p> <blockquote> <p>That is possible due to a new model of computation, the Interaction Net, which combines the Turing Machine with the Lambda Calculus. Previous implementations of this model have been inefficient in practice, however, a recent breakthrough has drastically improved its efficiency, giving birth to the HVM. Despite being a prototype, it already beats mature compilers in many cases, and is set to scale towards uncharted levels of performance.</p> </blockquote> <p>Example: <code>main.hvm</code>; looks like untyped Haskell:</p> <pre><code>// Creates a tree with `2^n` elements (Gen 0) = (Leaf 1) (Gen n) = (Node (Gen(- n 1)) (Gen(- n 1))) // Adds all elements of a tree (Sum (Leaf x)) = x (Sum (Node a b)) = (+ (Sum a) (Sum b)) // Performs 2^n additions in parallel (Main n) = (Sum (Gen n)) </code></pre> <p>The program above creates a perfect binary tree with 2^n elements and adds them up. Since it is recursive, HVM will parallelize it automatically.</p> <p>Run and compile:</p> <pre><code>hvm r main 10 # runs it with n=10 hvm c main # compiles HVM to C clang -O2 main.c -o main -lpthread # compiles C to BIN ./main 30 # runs it with n=30 </code></pre> <p>The program above runs in about 6.4 seconds in a modern 8-core processor, while the identical Haskell code takes about 19.2 seconds in the same machine with GHC. This is HVM: write a functional program, get a parallel C runtime.</p> <p>Benefits are solely to parallelizable (recursive) work;</p> <h2>Example</h2> <p>HVM is a low-level compile target for high-level languages. It provides a raw syntax for wiring interaction nets. For example:</p> <pre><code>@main = a &amp; @sum ~ (28 (0 a)) @sum = (?(((a a) @sum__C0) b) b) @sum__C0 = ({c a} ({$([*2] $([+1] d)) $([*2] $([+0] b))} f)) &amp;! @sum ~ (a (b $(:[+] $(e f)))) &amp;! @sum ~ (c (d e)) </code></pre> https://tedneward.github.io/Research/vms/clr/specifications/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/clr/specifications/index.html <p><a href="/platforms/clr">CLR/CLI</a> Specifications:</p> <ul> <li> <p><a href="https://www.ecma-international.org/publications-and-standards/standards/ecma-335/">ECMA-335</a>: Common Language Infrastructure (CLI) | <a href="https://www.ecma-international.org/publications/files/ECMA-ST/ECMA-335.pdf">PDF</a></p> </li> <li> <p><a href="https://github.com/dotnet/runtime/blob/main/docs/design/specs/Ecma-335-Augments.md"><code>dotnet/runtime/docs/design/specs</code> ECMA-335 CLI Addenda</a>: Explicitly adds CLI support for C# 8.0 default interface methods, among other things</p> </li> </ul> https://tedneward.github.io/Research/vms/dis/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/dis/index.html <p><a href="https://www.vitanuova.com/inferno/papers/dis.html">Specification</a> (<a href="https://www.vitanuova.com/inferno/papers/dis.pdf">PDF</a>) | <a href="https://www.vitanuova.com/inferno/papers/hotchips.html">"The Design of the Inferno Virtual Machine"</a> (<a href="https://www.vitanuova.com/inferno/papers/hotchips.pdf">PDF</a>)</p> <blockquote> <p>The virtual machine models a CISC-like, three operand, memory-to-memory architecture. Code can either be interpreted by a C library or compiled on-the-fly into machine code for the target architecture. ... The Dis machine performs both reference counted and real time mark and sweep garbage collection.</p> </blockquote> https://tedneward.github.io/Research/vms/gameboy/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/gameboy/index.html <h3>Resources</h3> <ul> <li><a href="https://izik1.github.io/gbops/">gbops</a>: A table of Game Boy’s instruction set. Information necessary for decoding instructions is summarized here.</li> <li><a href="http://marc.rawer.de/Gameboy/Docs/GBCPUman.pdf">Game Boy CPU Manual</a>: CPU manual.</li> <li><a href="https://gbdev.io/pandocs/">Pandocs</a>: A wiki with details on how each hardware module should work.</li> </ul> <h3>Implementations</h3> <ul> <li><a href="https://imrannazar.com/GameBoy-Emulation-in-JavaScript">"Game Boy Emulation in JavaScript"</a> (<a href="http://github.com/Two9A/jsGB">Source</a>): A tutorial on how to implement a Game Boy emulator in JavaScript.</li> <li><a href="https://linoscope.github.io/writing-a-game-boy-emulator-in-ocaml/">"Writing a Game Boy Interpreter in OCaml"</a> With resource links to:</li> </ul> https://tedneward.github.io/Research/vms/hhvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/hhvm/index.html <p><a href="https://hhvm.com/">Website</a> | <a href="https://github.com/facebook/hhvm">Github</a> | <a href="languages/hack.html">Hack</a></p> https://tedneward.github.io/Research/vms/ikvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/ikvm/index.html <p><a href="https://github.com/ikvm-revived/ikvm">GitHub</a></p> <p>IKVM is an implementation of Java for the Microsoft .NET platform. It can be used to quickly and easily:</p> <ul> <li>Execute compiled Java code (bytecode) on .NET Framework or .NET Core</li> <li>Convert bytecode to a .NET assembly to directly access its API in a .NET project</li> </ul> <p>These tasks can be done without porting source code to .NET.</p> <p>IKVM Components</p> <ul> <li>A Java virtual machine (JVM) implemented in .NET</li> <li>A .NET implementation of the Java class libraries</li> <li>A tool that translates Java bytecode (JAR files) to .NET IL (DLL or EXE files).</li> <li>Tools that enable Java and .NET interoperability</li> <li>A full JRE/JDK 8 runtime image.</li> </ul> https://tedneward.github.io/Research/vms/clr/sscli-essentials/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/clr/sscli-essentials/index.html <p><a href="../SSCLI2Internals-DRAFT.pdf">SSCLI Essentials 2.0 (Draft) PDF</a>: This is the last (draft) copy I have of the SSCLI Essentials 2.0 book that Joel Pobar and I worked on for Microsoft Research, as a follow-on to the first edition by David, Geoff, and myself. Most high-level details appear to be in sync with the .NET Core release, although as one might expect, there is some source drift--not unexpected, since this book was done in 2006.</p> https://tedneward.github.io/Research/vms/dolphinsmalltalk/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/dolphinsmalltalk/index.html <p><a href="https://github.com/dolphinsmalltalk/DolphinVM">Source</a></p> <p>This repository is now obsolete. It has been merged into the main Dolphin repository for Dolphin 7.1.</p> https://tedneward.github.io/Research/vms/glulx/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/glulx/index.html <p><a href="https://www.eblong.com/zarf/glulx/">Website</a> | <a href="http://github.com/erkyrath/glulxe">Source</a></p> <p>Glulx is a portable VM, like the Z-machine. Unlike the Z-machine, it uses 32-bit data and addresses, so it can handle game files up to four gigabytes long. Also unlike the Z-machine, it has native support for Glk I/O, so game files can use any capability Glk provides. However, like the Z-machine -- again -- you can write games in the Inform language and compile them to Glulx game files.</p> <hr> <h1>Quixe: a Glulx VM interpreter in Javascript</h1> <p><a href="http://github.com/erkyrath/quixe">Website</a> | <a href="https://github.com/erkyrath/lectrote">Lectrote (Quixe in an Electron app)</a></p> https://tedneward.github.io/Research/vms/higgs/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/higgs/index.html <p>[Source]<a href="https://github.com/higgsjs/Higgs">https://github.com/higgsjs/Higgs</a>)</p> https://tedneward.github.io/Research/vms/jpc/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/jpc/index.html <p><a href="https://github.com/ianopolous/JPC">Source</a></p> https://tedneward.github.io/Research/vms/clr/ffi/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/clr/ffi/index.html <p>-ClrDebug: Managed wrappers around the Unmanaged API<br> - <a href="https://github.com/lordmilko/ClrDebug">https://github.com/lordmilko/ClrDebug</a><br> - CoreRTDemo: Building a native library in C# and calling it from C++<br> - <a href="https://github.com/encrypt0r/CoreRTDemo">https://github.com/encrypt0r/CoreRTDemo</a><br> - Writing Native Libraries in C# and using them in other languages<br> - <a href="https://dev.to/encrypt0r/writing-native-libraries-in-c-3kl">https://dev.to/encrypt0r/writing-native-libraries-in-c-3kl</a><br> - CppSharp<br> - Tools and libraries to glue C/C++ APIs to high-level languages<br> - <a href="https://github.com/mono/CppSharp">https://github.com/mono/CppSharp</a><br> - P/Invoke: A library containing all P/Invoke code so you don't have to import it every time. Maintained and updated to support the latest Windows OS.<br> - <a href="https://github.com/AArnott/pinvoke">https://github.com/AArnott/pinvoke</a><br> - The .NET Inter-Operability Operation<br> - Derbycon 2017; James Forshaw<br> - <a href="http://www.irongeek.com/i.php?page=videos/derbycon7/s13-the-net-inter-operability-operation-james-forshaw">http://www.irongeek.com/i.php?page=videos/derbycon7/s13-the-net-inter-operability-operation-james-forshaw</a><br> - Using Span for high performance interop with unmanaged libraries<br> - <a href="https://ericsink.com/entries/utf8z.html">https://ericsink.com/entries/utf8z.html</a></p> https://tedneward.github.io/Research/vms/clr/coreclr/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/clr/coreclr/index.html <p><a href="https://github.com/dotnet/runtime">Source</a></p> <p><a href="../sscli-essentials">SSCLI Essentials</a> is a good place to start to understand the CLR itself.</p> <p><a href="https://github.com/dotnet/coreclr/tree/master/Documentation/botr">"Book of the Runtime"</a>: Guides to the internals of .NET Core. Anything in here that conflicts with SSCLI Essentials should be considered the more accurate of the two.</p> <p><a href="https://github.com/dotnet/cli/blob/rel/1.0.0/Documentation/specs/corehost.md">corehost.exe runtime assembly resolution</a> (Note that the <code>cli</code> repo has been pulled into the <code>sdk</code> repo now, so it's not clear how accurate this link is at this point)</p> <h2>List of blog posts that provide a 'deep-dive' into the CoreCLR source code</h2> <ul> <li><a href="https://mattwarren.org/2016/07/04/How-the-dotnet-CLI-tooling-runs-your-code/">"How the <code>dotnet</code> CLI runs your code"</a>: Great breakdown of how the <code>dotnet</code> tool works, particular emphasis to the environment variables mentioned (<code>COREHOST_TRACE</code> and <code>DOTNET_CLI_CAPTURE_TIMING</code>)</li> </ul> <h3>Posts that take a high-level overview of releases</h3> <ul> <li><a href="https://www.ageofascent.com/2017/11/05/perfromance-dotnet-core-2-corestart-conference/">Corestart 2.0: What's new for performance in .NET Core 2.0</a></li> <li><a href="https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core/">Performance improvements in .NET Core 2.0</a></li> <li><a href="https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-2-1/">Performance improvements in .NET Core 2.1</a></li> <li><a href="https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-core-3-0/">Performance improvements in .NET Core 3.0</a></li> <li><a href="https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-5/">Performance improvements in .NET 5</a></li> </ul> <h3>Posts that take a high-level look at the entire source:</h3> <ul> <li><a href="http://mattwarren.org/2017/03/23/Hitchhikers-Guide-to-the-CoreCLR-Source-Code/">A Hitchhikers Guide to the CoreCLR Source Code</a></li> <li><a href="http://mattwarren.org/2017/02/07/The-68-things-the-CLR-does-before-executing-a-single-line-of-your-code/">The 68 things the CLR does before executing a single line of your code</a></li> <li><a href="http://mattwarren.org/2016/12/12/Research-papers-in-the-.NET-source/">Research papers in the .NET source</a></li> </ul> <h3>Posts that reference specific parts of the source:</h3> <ul> <li><a href="http://mattwarren.org/2017/08/02/A-look-at-the-internals-of-boxing-in-the-CLR/">A look at the internals of 'boxing' in the CLR</a></li> <li><a href="http://mattwarren.org/2017/07/10/Memory-Usage-Inside-the-CLR/">Memory Usage Inside the CLR</a></li> <li><a href="http://mattwarren.org/2017/06/15/How-the-.NET-Rutime-loads-a-Type/">How the .NET Runtime loads a Type</a></li> <li><a href="http://mattwarren.org/2017/05/19/Adding-a-new-Bytecode-Instruction-to-the-CLR/">Adding a new Bytecode Instruction to the CLR</a> (Peter Drayton did this back in the first release of Rotor, and did several conference talks on it; unfortunately his materials were sort of lost to the world when he went into hiding inside Microsoft)</li> <li><a href="http://mattwarren.org/2017/05/08/Arrays-and-the-CLR-a-Very-Special-Relationship/">Arrays and the CLR - a Very Special Relationship</a></li> <li><a href="http://mattwarren.org/2017/04/13/The-CLR-Thread-Pool-Thread-Injection-Algorithm/">The CLR Thread Pool 'Thread Injection' Algorithm</a></li> <li><a href="http://mattwarren.org/2017/03/30/The-.NET-IL-Interpreter/">The .NET IL Interpreter</a></li> <li><a href="http://mattwarren.org/2017/01/25/How-do-.NET-delegates-work/">How do .NET delegates work?</a></li> <li><a href="http://mattwarren.org/2016/12/14/Why-is-Reflection-slow/">Why is reflection slow?</a></li> <li><a href="http://mattwarren.org/2016/08/16/Preventing-dotNET-Garbage-Collections-with-the-TryStartNoGCRegion-API/">Preventing .NET Garbage Collections with the TryStartNoGCRegion API</a></li> <li><a href="http://mattwarren.org/2016/08/08/GC-Pauses-and-Safe-Points/">GC Pauses and Safe Points</a></li> <li><a href="http://mattwarren.org/2016/05/31/Strings-and-the-CLR-a-Special-Relationship/">Strings and the CLR - a Special Relationship</a></li> </ul> https://tedneward.github.io/Research/vms/atom/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/atom/index.html <p><a href="https://atomvm.sourceforge.net/">Website</a> | <a href="https://sourceforge.net/projects/atomvm/files/">Download</a></p> <p>The VM can be programmed using a neo-assembler language called Atom Assembler. This document discusses the VM and explains each opcode in Atom Assembler.</p> <p>Atom is a register based Virtual Machine (VM). Atom consists of one bank of registers numbered 1 through 64, of type Variant. The VM is programmed with Atom Assembler, a high-level, neo-assembler language.</p> <p>Atom is currently in alpha stage, and as such, lacks a few features. Most notably missing is file I/O and a byte-code compiler. Both of these items are on the to-do list. However, it does have over 100 commands (opcodes), and fairly complex programs can be written in Atom Assembler and executed by the VM.</p> <p>Atom is written in Delphi. It is released under the GPL and the source code can be downloaded from the Sourceforge files section.</p> <p>I created Atom as a learning exercise in building a virtual machine. See my article Why Atom, for more details. At some point in time, after the VM is feature complete, I plan on creating a high-level language (named Atom of course) that will interface with the VM.</p> <p>I hope that eventually, Atom will become useful. Even if it doesn't though, it will have served the purpose of being a vehicle for learning the intricacies of creating a programming language.</p> <hr> <p>Includes Screen Opcodes (!) and File Opcodes, not to mention Widgets. Not your typical VM.</p> https://tedneward.github.io/Research/vms/bochs/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/bochs/index.html <p><a href="http://bochs.sourceforge.net/">Website</a></p> https://tedneward.github.io/Research/vms/android/dalvik/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/android/dalvik/index.html <p><a href="https://source.android.com/devices/tech/dalvik">Dalvik and ART home page</a></p> <hr> <p><a href="https://github.com/jserv/simple-dvm">Simple-DVM</a>: A simplified educational Dalvik virtual machine implemetnation</p> https://tedneward.github.io/Research/vms/avmplus/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/avmplus/index.html <p><a href="https://github.com/adobe-flash/avmplus">Source</a> -- no updates since 2013, research/investigation use only</p> <p><a href="https://github.com/adobe/avmplus">Source</a> -- no updates since 2016, research/investigation use only. Appears to contain all contains of prior link, plus an additional drop of some files three years later.</p> <p>Both appear to be straight source-drop-dumps of the AVM when it was discontinued.</p> https://tedneward.github.io/Research/vms/brainfuck-vm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/brainfuck-vm/index.html <p><a href="https://github.com/Dominik-Salawa/Brainfuck-VM">Source</a></p> https://tedneward.github.io/Research/vms/android/smali/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/android/smali/index.html <p><a href="https://github.com/JesusFreke/smali">Source</a> <em>(Last update 2022)</em></p> https://tedneward.github.io/Research/vms/baremetalvmm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/baremetalvmm/index.html <p><a href="https://github.com/raesene/baremetalvmm">Source</a></p> <p>The goal of the project is to be useful in cases where you want something like Docker, but want some more isolation that Docker provides, or you want to do lower level tasks in the VM that don't suit Docker well. N.B we're not there yet!</p> https://tedneward.github.io/Research/vms/chip-8/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/chip-8/index.html <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="http://devernay.free.fr/hacks/chip8/C8TECH10.HTM">Cowgod's Chip-8 Technical Reference</a></li> </ul> https://tedneward.github.io/Research/vms/aquavm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/aquavm/index.html <p><a href="https://github.com/fluencelabs/aquavm">Github</a></p> https://tedneward.github.io/Research/vms/beamvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/beamvm/index.html <p><a href="https://github.com/erlang/otp/tree/master/erts/emulator/beam">Source (I think)</a> - Looks like it's bundled with language and platform source</p> <h2>Reading</h2> <ul> <li><a href="http://beam-wisdoms.clau.se/en/latest/">BEAM Wisdoms</a> (HTML)</li> <li><a href="https://blog.stenmans.org/theBeamBook">The BEAM Book</a> (HTML) / <a href="https://happi.github.io/theBeamBook/">https://happi.github.io/theBeamBook/</a> : Very good book which covers almost all the aspects of the Erlang RunTime System and BEAM subsystems.</li> <li><a href="http://erlang.org/doc/efficiency_guide/introduction.html">Erlang Efficiency Guide</a>.</li> <li><a href="http://erlang.org/doc/efficiency_guide/advanced.html#id71365">How much memory</a> do native Erlang data types occupy and also some important system limits.</li> <li><a href="http://www.erlang-factory.com/upload/presentations/247/erlang_vm_1.pdf">Evolution of the Erlang VM</a></li> <li><a href="http://jlouisramblings.blogspot.ch/2013/01/how-erlang-does-scheduling.html">How Erlang does scheduling</a></li> <li><a href="http://webcem01.cem.itesm.mx:8005/erlang/cd/downloads/hopl_erlang.pdf">A History of Erlang</a> by Joe Armstrong.</li> </ul> <h2>Notes</h2> <p><a href="https://happi.github.io/theBeamBook/#CH-Beam_loader">https://happi.github.io/theBeamBook/#CH-Beam_loader</a>: A very interesting part from the aforementioned book about the BEAM code loader and transformation from generic to specific instructions. Basically the Erlang bytecode loader does a large amount of work rewriting the generic "transport format" bytecode in the object files into the concrete internal bytecode operations that are actually executed. This recognizes common sequences and replaces them with optimized single-opcode versions.</p> <p><a href="http://gomoripeti.github.io/beam_by_example/">http://gomoripeti.github.io/beam_by_example/</a>: A peak into the Erlang compiler and BEAM bytecode. This is an overview of all the intermediate representations of the Erlang code. Steps described in the article are as follows: <code>Erlang source code --&gt; Abstract Syntax Tree ('P') --&gt; expanded AST ('E') --&gt; Core Erlang ('to_core') --&gt; BEAM byte-code</code>. However I would also add two missing steps namely BEAM generic bytecode --&gt; BEAM specific bytecode and Core Erlang --&gt; Kernel Erlang.<br> So to sum up, the main compilation stages for Erlang are:</p> <ul> <li>preprocessing (macros, conditional compilation, includes) <ul> <li>source-level expansions, such as records to tuples</li> <li>translation to Core Erlang + optimizations and optional inlining</li> <li>translation to "Kernel Erlang" + pattern matching compilation and further optimizations</li> <li>translation to Beam assembler + some more optimizations</li> <li>encoding as Beam bytecode</li> </ul> </li> </ul> <p>StackOverflow question with an interesting answer which contains some details about the Erlang pattern match compiler:</p> <p><a href="https://stackoverflow.com/questions/587996/erlang-beam-bytecode">https://stackoverflow.com/questions/587996/erlang-beam-bytecode</a></p> <p>Source code of the Kernel Erlang compiler (<code>v3_kernel.erl</code>) with useful comments:</p> <p><a href="https://github.com/erlang/otp/blob/master/lib/compiler/src/v3_kernel.erl#L20">https://github.com/erlang/otp/blob/master/lib/compiler/src/v3_kernel.erl#L20</a></p> <p>A nice thread descibing compilation to BEAM bytecode via <code>EAF</code> (in the context of Elixir). It also contains useful links:</p> <p><a href="https://elixirforum.com/t/getting-each-stage-of-elixirs-compilation-all-the-way-to-the-beam-bytecode/1873">https://elixirforum.com/t/getting-each-stage-of-elixirs-compilation-all-the-way-to-the-beam-bytecode/1873</a></p> <p>Erlang <code>compile</code> module docs:</p> <p><a href="http://erlang.org/doc/man/compile.html#file-2">http://erlang.org/doc/man/compile.html#file-2</a></p> <p>BEAM file format:</p> <p><a href="http://rnyingma.synrc.com/publications/cat/Functional%20Languages/Erlang/BEAM.pdf">http://rnyingma.synrc.com/publications/cat/Functional%20Languages/Erlang/BEAM.pdf</a></p> <p>Erlang abstract format docs:</p> <p><a href="http://erlang.org/doc/apps/erts/absform.html">http://erlang.org/doc/apps/erts/absform.html</a></p> <p>The Erlang BEAM Virtual Machine Specification by one of its designers (Björn in the BEAM):</p> <p><a href="http://www.cs-lab.org/historical_beam_instruction_set.html">http://www.cs-lab.org/historical_beam_instruction_set.html</a></p> <p>A Peek Inside the Erlang Compiler:</p> <p><a href="http://prog21.dadgum.com/127.html">http://prog21.dadgum.com/127.html</a></p> <p>BEAM wisdoms:</p> <ul> <li>BEAM file format: <a href="http://beam-wisdoms.clau.se/en/latest/indepth-beam-file.html#">http://beam-wisdoms.clau.se/en/latest/indepth-beam-file.html#</a></li> <li>BEAM instructions: <a href="http://beam-wisdoms.clau.se/en/latest/indepth-beam-instructions.html#">http://beam-wisdoms.clau.se/en/latest/indepth-beam-instructions.html#</a></li> </ul> <p>Also a nice description of the BEAM instruction set: <a href="http://erlangonxen.org/more/beam">http://erlangonxen.org/more/beam</a></p> <p>Thesis which contains some useful information about the Erlang implementation: <a href="http://uu.diva-portal.org/smash/get/diva2:428121/FULLTEXT01.pdf">http://uu.diva-portal.org/smash/get/diva2:428121/FULLTEXT01.pdf</a></p> <p>Code of the BEAM loader which was described above: <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/beam_load.c">https://github.com/erlang/otp/blob/master/erts/emulator/beam/beam_load.c</a> You might be particularly interested in the <code>static int transform_engine(LoaderState* st)</code> function.</p> <p>Internal doc in the erts/emulator is a treasure trove of the underlying implementation details: <a href="https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc">https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc</a></p> <p>Will put reference to the Erlang garbage collection doc as a separate link: <a href="https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/GarbageCollection.md">https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/GarbageCollection.md</a></p> <p>Reductions:</p> <ul> <li> <p>Current max reductions is <a href="https://github.com/erlang/otp/blob/OTP-21.3.8.11/erts/emulator/beam/erl_vm.h#L39">4000</a>.</p> </li> <li> <p>Information regarding reduction from the <a href="https://github.com/happi/theBeamBook/blob/master/chapters/scheduling.asciidoc#reductions">BeamBook</a>.</p> </li> <li> <p>What counts as a VM reduction: As per Erlang VM maintainer <a href="https://stackoverflow.com/a/31805391/1744056">Lukas Larsson's comment</a>:</p> <pre><code>As a general rule of thumb; a function call (not the return) or anything that may take an unknown amount of time counts reductions. This includes bifs, nifs, gc, sending/receiving messages and probably more that I cannot think of right now. </code></pre> </li> </ul> <h4>Optimisations and monitoring</h4> <ol> <li> <p>Internal OTP doc which contains useful insights regaring the process managements, lock and run queues:<br> <a href="https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/ProcessManagementOptimizations.md">https://github.com/erlang/otp/blob/master/erts/emulator/internal_doc/ProcessManagementOptimizations.md</a></p> </li> <li> <p>Insights by one of the original designers of BEAM SMP support (Rickard Green) into the scheduler <a href="http://erlang.org/doc/man/erl.html#+scl">compaction of load mode</a> taken from this email thread: <a href="http://erlang.org/pipermail/erlang-questions/2012-October/069585.html">http://erlang.org/pipermail/erlang-questions/2012-October/069585.html</a></p> </li> </ol> <blockquote> <p>The runtime system tries to compact the load on as few schedulers as possible without getting run-queues that build up. The runtime system wont wake up new schedulers unless some overload has accumulated. This overload either show up as a quickly growing run-queue or a small run-queue over a longer time. The +swt flags sets the threshold that is used for determining when enough overload has accumulated to wake up another scheduler. This compaction of load onto fewer schedulers is there in order to reduce communication overhead when there aren't enough work to fully utilize all schedulers. The performance gain of this compaction depends on the hardware. We have gotten reports about problems with this functionality, but we have not found any bugs in this functionality. We have only found that it behaves as expected. That is, if more schedulers aren't woken this is<br> due to not enough accumulated overload. The +swt switch was introduced in order to give the user the possibility do define what is enough overload for his or her taste. The currently used wakeup strategy is very quick to forget about previously accumulated overload that has disappeared. Maybe even too quick for my taste when "+swt very_low" is used. I've therefore implemented an alternative strategy that most likely will be the default in R16. As of R15B02 you can try this strategy out by passing "+sws proposal" as a command line argument. In combination with "+swt very_low", the runtime system should be even more eager to wake up schedulers than when only using "+swt very_low".</p> </blockquote> <h2>ETS Table</h2> <p>When access an ETS table, there are wo locks that need to be taken:</p> <ol> <li>A lock to access the meta table, to convert the numeric table identifier to a pointer to the actual table.</li> <li>The lock for the table itself (either a read or write lock). read and write_concurrency: <a href="http://erlang.org/doc/man/ets.html#new_2_write_concurrency">http://erlang.org/doc/man/ets.html#new_2_write_concurrency</a></li> <li>The default locking is on table-level, allowing only one update operation at a time per table. Table option write_concurrency will enable locking on a more fine grained level, allowing concurrent update operations. In current implementation 16 locks per table is used, which result in a probability of 1/16 that two random keys will collide on the same lock. Option write_concurrency have no effect on ordered_set's.</li> </ol> <p>ETS Scalability paper: <a href="http://winsh.me/papers/erlang_workshop_2013.pdf">http://winsh.me/papers/erlang_workshop_2013.pdf</a><br> ETS meta table source code: <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_db.c#L324">https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_db.c#L324</a></p> <h3>External Term Format</h3> <p><a href="http://erlang.org/doc/apps/erts/erl_ext_dist.html">Official Erlang documentation</a> on this topic</p> <p><a href="http://erlang.org/doc/man/erlang.html#term_to_binary-2">term_to_binary/2</a> docs</p> <p><a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/external.c#L2061">C implementation</a> of the term_to_binary in ERTS.</p> <h3>Erlang GC</h3> <p>Garbage collect <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_gc.c#L664">C implementation</a>.</p> <h3>Erlang processes</h3> <p><a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_process.c#L11391">erl_create_process</a> function.</p> <h3>BIFs</h3> <p>We always use BIFs in Erlang such as <code>element</code> or <code>spawn_link</code> or whatnot in our code. <code>gen</code> and <code>gen_server</code> are based on the <code>erlang:spawn</code> BIF, hence I think it's important to know how BIFs are actually implemented: <a href="https://github.com/erlang/otp/blob/master/lib/stdlib/src/erl_internal.erl#L22">https://github.com/erlang/otp/blob/master/lib/stdlib/src/erl_internal.erl#L22</a> <code>stdlib/erl_internal</code> here is just a BIF-proxy because all the functions are implemented natively in C inside the ERTS/BEAM.</p> <p>Core C implementation of BIFs <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/bif.c#L69">resides here</a></p> <h3>Code Purger</h3> <p>The most important parts from the <a href="https://github.com/erlang/otp/blob/master/erts/preloaded/src/erts_code_purger.erl#L61">Erlang code purger</a> such as <code>purge</code>, <code>soft_purge</code>, etc.</p> <h3>Standard Library</h3> <p><a href="https://github.com/erlang/otp/blob/master/lib/stdlib/src/lists.erl#L20">Lists module</a> functionality. Some of the functions such as <code>keyfind/3</code>, <code>keymember/3</code>, etc. are implemented as BIFs and defined in a separate <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_bif_lists.c">C file inside of BEAM</a>.</p> <h3>Message Passing</h3> <p>Core functionality of the <a href="https://github.com/erlang/otp/blob/master/erts/emulator/beam/erl_message.c">Erlang message passing</a></p> <h3>Internal BEAM emulator docs:</h3> <p><a href="https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc">https://github.com/erlang/otp/tree/master/erts/emulator/internal_doc</a></p> https://tedneward.github.io/Research/vms/android/art/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/android/art/index.html <p><a href="https://source.android.com/devices/tech/dalvik/index.html">Website</a> (shared with Dalvik) | <a href="https://source.android.com/devices/tech/dalvik/art.html">Source</a> | <a href="https://en.wikipedia.org/wiki/Android_Runtime">Wikipedia</a></p> https://tedneward.github.io/Research/vms/aamachine/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/aamachine/index.html <p><a href="https://www.linusakesson.net/dialog/aamachine/index.php">Website</a></p> <p>It is designed for stories implemented in the Dialog programming language.</p> <p>In a sense, the Å-machine is to Dialog what Glulx is to Inform 7. It eliminates the tight restrictions on story size, and extends the basic functionality with a carefully balanced set of new features. But the Å-machine is designed to run the same stories on everything from 8-bit systems to modern web browsers. Data structures and encodings are economical, and the overall word size has not increased. Large stories are supported, but small stories still have a very compact binary representation.</p> <p>Compared to the Z-machine and Glulx, the Å-machine operates at a higher level of abstraction. This improves performance on vintage hardware, both by making story files smaller, which improves loading times, and by allowing larger chunks of computation to be implemented as native machine code. The downside is that the virtual machine is more tightly coupled to the idiosyncracies of a particular high-level language, in this case Dialog.</p> https://tedneward.github.io/Research/vms/air/index.html Mon, 13 Apr 2026 14:04:56 +0000 vms/air/index.html <p><a href="https://airsdk.harman.com/">Website</a> | <a href="https://github.com/airsdk">Source</a></p> https://tedneward.github.io/Research/tools/xcode/instruments/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xcode/instruments/index.html <h2>Articles</h2> <ul> <li> <p>[Instruments Tutorial with Swift: Getting Started](<a href="https://www.raywenderlich.com/16126261-instruments-tutorial-with-swift-ge">https://www.raywenderlich.com/16126261-instruments-tutorial-with-swift-ge</a>&gt; tting-started): In this Xcode tutorial, you’ll learn how to use Instruments to profile and debug performance, memory and reference issues in your iOS apps.</p> </li> <li> <p><a href="https://swiftwithmajid.com/2021/01/20/profiling-swiftui-app-using-instruments/">Profiling SwiftUI app using Instruments</a>: Xcode comes with a bunch of tools you need to build, debug and release your apps. One of these tools is the Instruments app. The Instruments app is a great tool for profiling your iOS apps. It provides many profiling templates for debugging Core Data, catching memory leaks, disk read/writes, and much more. This week we will learn how to profile SwiftUI apps using the SwiftUI template.</p> </li> </ul> https://tedneward.github.io/Research/tools/xml2rfc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xml2rfc/index.html <p><a href="https://author-tools.ietf.org/">Website</a></p> <p>This service allows you to convert an Internet-Draft from one format into another, including rendered outputs. In the background this service uses id2xml, kramdown-rfc, mmark, rst2rfcxml and xml2rfc, chaining them together as needed to deliver the requested conversion.</p> <p>The input must be a valid Internet-Draft in one of the following formats:</p> <ul> <li>XML as .xml (automatically recognises v3 as defined in RFC 7991 and v2 as defined in RFC 7749)</li> <li>Markdown as .md or .mkd (kramdown-rfc and mmark dialects are supported)</li> <li>reStructuredText (RST) as .rst (Author tools uses rst2rfcxml.)</li> <li>Plain text as .txt</li> </ul> https://tedneward.github.io/Research/tools/zenith/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/zenith/index.html <p><a href="https://github.com/bvaisvil/zenith">Source</a></p> https://tedneward.github.io/Research/tools/xcode/playgrounds/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xcode/playgrounds/index.html <h2><a href="https://stackoverflow.com/questions/39078307/is-it-possible-to-load-a-storyboard-in-an-ios-playground">Playgrounds loading Storyboards</a></h2> <ol> <li> <p>Compile the storyboard using</p> <pre><code>ibtool --compile MainMenu.storyboardc MainMenu.storyboard </code></pre> <p>(Note the "c" at the end of the .storyboard filename is NOT a typo.)</p> <p>May need to specify module name using --module flag when compiling. Otherwise, your outlets may cause a runtime error. For the playground book, the module name is Book_Sources. For your own playground, the module name is [Playground name]_Sources. Note I didn't find any of this in Apple's documentation, so it is subject to change. The final command looks like this:</p> <pre><code>ibtool --compile MainMenu.storyboardc MainMenu.storyboard --module Book_Sources </code></pre> <p>(When I tried this on 11 Apr 2022, it worked w/o the module flag.)</p> </li> <li> <p>Place the compiled storyboardc file in the Resources folder of the playground, not the Sources folder</p> </li> <li> <p>Instantiate it in the Playground and set the instantiated ViewController to be the one loaded by the Playground:</p> <pre><code>let storyboard = UIStoryboard(name: "Main", bundle: Bundle.main) PlaygroundPage.current.liveView = storyboard.instantiateInitialViewController() </code></pre> </li> </ol> https://tedneward.github.io/Research/tools/xtext/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xtext/index.html <p><a href="https://www.eclipse.org/Xtext/index.html">Website</a> | <a href="https://github.com/eclipse/xtext">Source</a></p> https://tedneward.github.io/Research/tools/zola/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/zola/index.html <p><a href="https://www.getzola.org/">Website</a> | <a href="https://github.com/getzola/zola">Source</a></p> https://tedneward.github.io/Research/tools/xcode/xcodebuild/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xcode/xcodebuild/index.html <ul> <li>xcbuild-debugging-tricks <ul> <li>Xcode new build system debugging tricks</li> <li><a href="https://gist.github.com/ddunbar/2dda0e836c855ea96759d1d05f086d69">https://gist.github.com/ddunbar/2dda0e836c855ea96759d1d05f086d69</a></li> </ul> </li> <li><a href="https://XcodeBuildSettings.com">https://XcodeBuildSettings.com</a></li> </ul> https://tedneward.github.io/Research/tools/yaak/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/yaak/index.html <p><a href="https://yaak.app/">Website</a> | <a href="https://github.com/yaakapp">Source</a></p> <p>Handy import/export to cURL, among other tools.</p> https://tedneward.github.io/Research/tools/xmake/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xmake/index.html <p><a href="https://xmake.io/">Website</a> | <a href="https://github.com/xmake-io/xmake/">Github</a></p> https://tedneward.github.io/Research/tools/yuml/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/yuml/index.html <p><a href="https://yuml.me/">Website</a> | Commercial? Has a Python package though? (<code>pip install https://github.com/wandernauta/yuml/zipball/master</code>)</p> https://tedneward.github.io/Research/tools/vscode/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/vscode/index/index.html <p><a href="https://github.com/Microsoft/vscode">Source</a></p> <p><a href="https://vscodium.com/">VSCodium</a>: a community-driven, freely-licensed binary distribution of Microsoft’s editor VS Code. "The VSCodium project exists so that you don’t have to download+build from source. This project includes special build scripts that clone Microsoft’s vscode repo, run the build commands, and upload the resulting binaries for you to GitHub releases. These binaries are licensed under the MIT license. Telemetry is disabled."</p> <h2>Resources</h2> <h3>Projects</h3> <p><a href="https://github.com/coder/code-server">code-server</a>: VS Code in the browser. Part of <a href="https://coder.com/">coder.com</a></p> https://tedneward.github.io/Research/tools/wikijs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/wikijs/index.html <p><a href="https://js.wiki/">Website</a> | <a href="https://github.com/Requarks/wiki">Source</a></p> https://tedneward.github.io/Research/tools/wireit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/wireit/index.html <p><a href="https://github.com/google/wireit">Source</a></p> https://tedneward.github.io/Research/tools/visualstudio/sapphire/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/visualstudio/sapphire/index.html <p><a href="http://www.sapphiresteel.com/Products/Ruby-In-Steel/Ruby-In-Steel-Developer-Overview.html">Website</a> | Commercial</p> <p>Mentions VS 2010, 2012, 2013, nothing later than that.</p> https://tedneward.github.io/Research/tools/wasm-tools/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/wasm-tools/index.html <p><a href="https://github.com/bytecodealliance/wasm-tools">Source</a></p> <h1>Installation</h1> <p>Installation can be confirmed with:</p> <pre><code>$ wasm-tools --version </code></pre> <p>Subcommands can be explored with:</p> <pre><code>$ wasm-tools help </code></pre> <h1>Examples</h1> <p>Basic validation/printing/parsing:</p> <pre><code class="language-sh"># Validate a WebAssembly file $ wasm-tools validate foo.wasm # Validate a WebAssembly module in the text format, automatically converting to # binary. $ wasm-tools validate foo.wat # Validate a WebAssembly file enabling an off-by-default feature $ wasm-tools validate foo.wasm --features=exception-handling # Validate a WebAssembly file with a default-enabled feature disabled $ wasm-tools validate foo.wasm --features=-simd # Print the text format of a module to stdout $ wasm-tools print foo.wasm # Convert a binary module to text $ wasm-tools print foo.wasm -o foo.wat # Convert a text module to binary $ wasm-tools parse foo.wat -o foo.wasm </code></pre> <p>Simple mutation as well as piping commands together:</p> <pre><code class="language-sh"># Mutate a WebAssembly module and print its text representation to stdout $ wasm-tools mutate foo.wasm -t # Mutate a WebAssembly module with a non-default seed and validate that the # output is a valid module. $ wasm-tools mutate foo.wasm --seed 192 | wasm-tools validate # Demangle Rust/C++ symbol names in the `name` section, strip all other custom # sections, and then print out what binary sections remain. $ wasm-tools demangle foo.wasm | wasm-tools strip | wasm-tools objdump </code></pre> <p>Working with components:</p> <pre><code class="language-sh"># Print the WIT interface of a component $ wasm-tools component wit component.wasm # Convert WIT text files to a binary-encoded WIT package, printing the result to # stdout $ wasm-tools component wit ./wit -t # Convert a WIT document to JSON $ wasm-tools component wit ./wit --json # Round trip WIT through the binary-encoded format to stdout. $ wasm-tools component wit ./wit --wasm | wasm-tools component wit # Convert a core WebAssembly binary into a component. Note that this requires # WIT metadata having previously been embedded in the core wasm module. $ wasm-tools component new my-core.wasm -o my-component.wasm # Convert a core WebAssembly binary which uses WASI to a component. $ wasm-tools component new my-core.wasm -o my-component.wasm --adapt wasi_snapshot_preview1.reactor.wasm </code></pre> <h3>CLI Conventions</h3> <p>There are a few conventions that all CLI commands adhere to:</p> <ul> <li>All subcommands print "short help" with <code>-h</code> and "long help" with <code>--help</code>.</li> <li>Input is by default read from stdin if no file input is specified (when applicable).</li> <li>Output is by default sent to stdout if a <code>-o</code> or <code>--output</code> flag is not provided. Binary WebAssembly is not printed to a tty by default, however.</li> <li>Commands which output WebAssembly binaries all support a <code>-t</code> or <code>--wat</code> flag to generate the WebAssembly text format instead.</li> <li>A <code>-v</code> or <code>--verbose</code> flag can be passed to enable log messages throughout the tooling. Verbosity can be turned up by passing the flag multiple times such as <code>-vvv</code>.</li> <li>Color in error messages and console output is enabled by default for TTY based outputs and can be configured with a <code>--color</code> argument.</li> </ul> <h1>Tools included</h1> <p>The <code>wasm-tools</code> binary internally contains a number of subcommands for working with wasm modules and component. Many subcommands also come with Rust crates that can be use programmatically as well:</p> <table> <thead> <tr> <th> CLI </th> <th> Rust Crate </th> <th> Playground </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>wasm-tools validate</code> </td> <td> <a href="https://crates.io/crates/wasmparser">wasmparser</a> </td> <td> </td> <td> Validate a WebAssembly file </td> </tr> <tr> <td> <code>wasm-tools parse</code> </td> <td> <a href="https://crates.io/crates/wat">wat</a> and <a href="https://crates.io/crates/wast">wast</a> </td> <td> <a href="https://bytecodealliance.github.io/wasm-tools/parse">parse</a> </td> <td> Translate the WebAssembly text format to binary </td> </tr> <tr> <td> <code>wasm-tools print</code> </td> <td> <a href="https://crates.io/crates/wasmprinter">wasmprinter</a> </td> <td> <a href="https://bytecodealliance.github.io/wasm-tools/print">print</a> </td> <td> Translate the WebAssembly binary format to text </td> </tr> <tr> <td> <code>wasm-tools smith</code> </td> <td> <a href="https://crates.io/crates/wasm-smith">wasm-smith</a> </td> <td> </td> <td> Generate a valid WebAssembly module from an input seed </td> </tr> <tr> <td> <code>wasm-tools mutate</code> </td> <td> <a href="https://crates.io/crates/wasm-mutate">wasm-mutate</a> </td> <td> </td> <td> Mutate an input wasm file into a new valid wasm file </td> </tr> <tr> <td> <code>wasm-tools shrink</code> </td> <td> <a href="https://crates.io/crates/wasm-shrink">wasm-shrink</a> </td> <td> </td> <td> Shrink a wasm file while preserving a predicate </td> </tr> <tr> <td> <code>wasm-tools dump</code> </td> <td> </td> <td> </td> <td> Print debugging information about the binary format </td> </tr> <tr> <td> <code>wasm-tools objdump</code> </td> <td> </td> <td> </td> <td> Print debugging information about section headers </td> </tr> <tr> <td> <code>wasm-tools strip</code> </td> <td> </td> <td> </td> <td> Remove custom sections from a WebAssembly file </td> </tr> <tr> <td> <code>wasm-tools demangle</code> </td> <td> </td> <td> </td> <td> Demangle Rust and C++ symbol names in the <code>name</code> section </td> </tr> <tr> <td> <code>wasm-tools compose</code> </td> <td> <a href="https://crates.io/crates/wasm-compose">wasm-compose</a> </td> <td> </td> <td> Compose wasm components together (<em>deprecated</em>) </td> </tr> <tr> <td> <code>wasm-tools component new</code> </td> <td> <a href="https://crates.io/crates/wit-component">wit-component</a> </td> <td> </td> <td> Create a component from a core wasm binary </td> </tr> <tr> <td> <code>wasm-tools component wit</code> </td> <td> </td> <td> </td> <td> Extract a <code>*.wit</code> interface from a component </td> </tr> <tr> <td> <code>wasm-tools component embed</code> </td> <td> </td> <td> </td> <td> Embed a <code>component-type</code> custom section in a core wasm binary </td> </tr> <tr> <td> <code>wasm-tools component unbundle</code> </td> <td> </td> <td> </td> <td> Extract core wasm modules from a component </td> </tr> <tr> <td> <code>wasm-tools metadata show</code> </td> <td> <a href="https://crates.io/crates/wasm-metadata">wasm-metadata</a> </td> <td> </td> <td> Show name and producer metadata in a component or module </td> </tr> <tr> <td> <code>wasm-tools metadata add</code> </td> <td> </td> <td> </td> <td> Add name or producer metadata to a component or module </td> </tr> <tr> <td> <code>wasm-tools addr2line</code> </td> <td> </td> <td> </td> <td> Translate wasm offsets to filename/line numbers with DWARF </td> </tr> <tr> <td> <code>wasm-tools completion</code> </td> <td> </td> <td> </td> <td> Generate shell completion scripts for <code>wasm-tools</code> </td> </tr> <tr> <td> <code>wasm-tools json-from-wast</code> </td> <td> </td> <td> </td> <td> Convert a <code>*.wast</code> file into JSON commands </td> </tr> <tr> <td> <code>wasm-tools wast</code> </td> <td> </td> <td> </td> <td> Validate the structure of a <code>*.wast</code> file </td> </tr> </tbody> </table> <p>The <code>wasm-tools</code> CLI contains useful tools for debugging WebAssembly modules and components. The various subcommands all have <code>--help</code> explainer texts to describe more about their functionality as well.</p> <h1>Language support</h1> <h2>JavaScript Tooling</h2> <p><a href="https://developer.mozilla.org/en-US/docs/WebAssembly">WebAssembly</a> was originally developed as a technology for running non-JavaScript workloads in the browser at near-native speed.</p> <p>JavaScript WebAssembly component model support is provided by a combination of tools:</p> <ul> <li><a href="https://github.com/bytecodealliance/StarlingMonkey">StarlingMonkey</a> a WebAssembly component aware Javascript engine</li> <li><a href="https://github.com/bytecodealliance/componentize-js"><code>componentize-js</code></a> a tool for building WebAssembly components from Javascript files</li> <li><a href="https://github.com/bytecodealliance/jco"><code>jco</code></a> a multi-tool for componentizing, type generation, and running components in NodeJS and browser contexts</li> </ul> <p>Note that <a href="https://typescriptlang.org">Typescript</a> can <em>also</em> be used, given that it is transpiled to JS first by relevant tooling (<code>tsc</code>). <code>jco</code> generates <a href="https://www.typescriptlang.org/docs/handbook/declaration-files/deep-dive.html#declaration-file-theory-a-deep-dive">type declaration files (<code>.d.ts</code>)</a> by default, and also has a <code>jco types</code> subcommand which generates typings that can be used with a Typescript codebase.</p> <blockquote> <p><a href="#">!WARNING</a><br> While popular projects like <a href="https://emscripten.org/"><code>emscripten</code></a> also build WebAssembly modules, those modules are not Component Model aware.</p> <p>Core WebAssembly modules do not contain the advanced features (rich types, structured language interoperation, composition) that the component model makes available.</p> </blockquote> <h2>Installing <code>jco</code></h2> <p>Installing <a href="https://github.com/bytecodealliance/jco"><code>jco</code></a> (which uses <a href="https://github.com/bytecodealliance/componentize-js"><code>componentize-js</code></a> can be done via standard NodeJS project tooling:</p> <pre><code class="language-console">npm install -g @bytecodealliance/jco </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> <code>jco</code> and <code>componentize-js</code> can be installed in a project-local manner with <code>npm install -D</code></p> </blockquote> <h2>Overview of Building a Component with JavaScript</h2> <p>Building a WebAssembly component with JavaScript often consists of:</p> <ol> <li>Determining which interface our functionality will target (i.e. a <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md">WebAssembly Interface Types ("WIT")</a> world)</li> <li>Writing JavaScript that satisfies the interface</li> <li>Compiling the interface-compliant JavaScript to WebAssembly</li> </ol> <h3>What is WIT?</h3> <p><a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md">WebAssembly Interface Types ("WIT")</a> is a featureful Interface Definition Language ("IDL") for defining functionality, but most of the time, you shouldn't need to write WIT from scratch. Often, it's sufficient to download a pre-existing interface that defines what your component should do.</p> <p>The <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/tutorial/wit/adder/world.wit"><code>adder</code> world</a> contains an interface with a single <code>add</code> function that sums two numbers:</p> <pre><code class="language-wit">{{#include ../../exmaples/tutorial/wit/adder/world.wit}} </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> <code>export</code>ing the <code>add</code> interface means that environments that interact with the resulting WebAssembly component<br> will be able to <em>call</em> the <code>add</code> function (fully qualified: <code>docs:adder/add.add@0.1.0</code>).</p> <p>To learn more about the WIT syntax, check out the <a href="https://component-model.bytecodealliance.org/design/wit.html">WIT explainer</a></p> </blockquote> <h2>Implementing a JS WebAssembly Component</h2> <p>To implement the <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/tutorial/wit/adder/world.wit"><code>adder</code> world</a>, we can write a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules">JavaScript ES module</a>:</p> <pre><code class="language-js">export const add = { add(x, y) { return x + y; } }; </code></pre> <blockquote> <p><a href="#">!WARNING</a><br> When building your JavaScript project, ensure to set the <code>"type":"module"</code> option in <code>package.json</code>, as <code>jco</code> works exclusively with JavaScript modules.</p> </blockquote> <p>In the code above:</p> <ul> <li>The <code>adder</code> world is analogous to the JavaScript module (file) itself</li> <li>The exported <code>add</code> object mirrors the <code>export</code>ed <code>add</code> interface in WIT</li> <li>The <code>add</code> function mirrors the <code>add</code> function inside the <code>add</code> interface</li> </ul> <p>With the WIT and JavaScript in place, we can use <a href="https://github.com/bytecodealliance/jco"><code>jco</code></a> to create a WebAssembly component from the JS module, using <code>jco componentize</code>.</p> <blockquote> <p><a href="#">!NOTE</a><br> You can also call <code>componentize-js</code> directly -- it supports both API driven usage and has a CLI.</p> </blockquote> <p>Our component is <em>so simple</em> (reminiscent of <a href="https://webassembly.github.io/spec/core/">Core WebAssembly</a>, which deals only in numeric values) that we're actually <em>not using</em> any of the <a href="https://wasi.dev/">WebAssembly System Interface</a> (access to files, network, etc). This means that we can <code>--disable</code> all unneeded WASI functionality when we invoke <code>jco componentize</code>:</p> <pre><code class="language-console">jco componentize \ --wit path/to/adder/world.wit \ --world-name example \ --out adder.wasm \ --disable all \ path/to/adder.js </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> If you're using <code>jco</code> as a project-local dependency, you can run <code>npx jco</code></p> </blockquote> <p>You should see output like the following:</p> <pre><code>OK Successfully written adder.wasm. </code></pre> <blockquote> <p><a href="#">!WARNING</a><br> By using <code>--disable all</code>, your component won't get access to any WASI interfaces that might be useful for debugging or logging.</p> <p>For example, you can't <code>console.log(...)</code> or <code>console.error(...)</code> without <code>stdio</code>; you can't use <code>Math.random()</code> without <code>random</code>; and you can't use <code>Date.now()</code> or <code>new Date()</code> without <code>clocks</code>.</p> <p>Please note that calls to <code>Math.random()</code> or <code>Date.now()</code> will return seemingly valid outputs, but without actual randomness or timestamp correctness.</p> </blockquote> <h3>Running the Component in the <code>example-host</code></h3> <p>To run the component we've built, we can use the <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/example-host"><code>example-host</code> project</a>:</p> <pre><code class="language-console">cd component-model/examples/example-host cargo run --release -- 1 2 ../path/to/adder.wasm 1 + 2 = 3 </code></pre> <blockquote> <p><a href="#">!WARNING</a><br> The <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/example-host"><code>example-host</code> Rust project</a> uses the <a href="https://www.rust-lang.org/tools/install">Rust toolchain</a>, in particular <a href="https://doc.rust-lang.org/cargo"><code>cargo</code></a>,<br> so to run the code in this section you may need to install some more dependencies (like the Rust toolchain).</p> </blockquote> <p>While the output isn't exciting, the code contained in <code>example-host</code> does a lot to make it happen:</p> <ul> <li>Loads the WebAssembly binary at the provided path (in the command above, <code>../path/to/adder.wasm</code>)</li> <li>Calls the <code>export</code>ed <code>add</code> function inside the <code>add</code> interface with arguments</li> <li>Prints the result</li> </ul> <p>The important Rust code looks something like this:</p> <pre><code class="language-rust">let component = Component::from_file(&amp;engine, path).context("Component file not found")?; let (instance, _) = Example::instantiate_async(&amp;mut store, &amp;component, &amp;linker) .await .context("Failed to instantiate the example world")?; instance .call_add(&amp;mut store, x, y) .await .context("Failed to call add function") </code></pre> <p>A quick reminder on the power and new capabilities afforded by WebAssembly -- we've written, loaded,<br> instantiated and executed JavaScript from Rust with a strict interface, without the need<br> for <a href="https://en.wikipedia.org/wiki/Foreign_function_interface">FFI</a>, subprocesses or a network call.</p> <h3>Running a Component from JavaScript Applications (including the Browser)</h3> <p>While JavaScript runtimes available in browsers can execute WebAssembly core modules, they cannot yet execute WebAssembly <em>components</em>, so WebAssembly components (JavaScript or otherwise) must be "transpiled" into a JavaScript wrapper and one or more <a href="https://webassembly.github.io/spec/core/binary/modules.html">WebAssembly core modules</a> which <em>can</em> be run by available WebAssembly engines.</p> <p>Given an existing WebAssembly component (e.g. <code>adder.wasm</code> which implements the <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/tutorial/wit/adder/world.wit"><code>adder</code> world</a>), we can "transpile" the WebAssembly component into runnable JavaScript by using <code>jco transpile</code>:</p> <pre><code class="language-console">jco transpile adder.wasm -o dist/transpiled </code></pre> <p>You should see output similar to the following:</p> <pre><code> Transpiled JS Component Files: - dist/transpiled/adder.core.wasm 10.1 MiB - dist/transpiled/adder.d.ts 0.1 KiB - dist/transpiled/adder.js 1.57 KiB </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> To follow along, see the <a href="https://github.com/bytecodealliance/jco/tree/main/examples/components/adder"><code>jco</code> example <code>adder</code> component</a>.</p> <p>With the project pulled locally, you also run <code>npm run transpile</code> which outputs to <code>dist/transpiled</code></p> </blockquote> <p>Thanks to <code>jco</code> transpilation, you can import the resulting <code>dist/transpiled/adder.js</code> file and run it from any JavaScript application using a runtime that supports the <a href="https://webassembly.github.io/spec/core/">core WebAssembly specification</a> as implemented for JavaScript.</p> <p>To use this component from <a href="https://nodejs.org/en">NodeJS</a>, you can write code like the following:</p> <pre><code class="language-mjs">import { add } from "./dist/transpiled/adder.js"; console.log("1 + 2 = " + add(1, 2)); </code></pre> <p>You can execute the JavaScript module with <code>node</code> directly:</p> <pre><code class="language-console">node run.js </code></pre> <p>You should see output like the following:</p> <pre><code>1 + 2 = 3 </code></pre> <p>This is directly comparable to the Rust host code mentioned in the previous section. Here, we are able to use NodeJS as a host for running WebAssembly, thanks to <code>jco</code>'s ability to transpile components.</p> <p>With <code>jco transpile</code> any WebAssembly binary (compiled from any language) can be run in JavaScript natively.</p> <h3>Building Reactor Components with <code>jco</code></h3> <p>Reactor components are WebAssembly components that are long running and meant to be called repeatedly over time. They're analogous to libraries of functionality rather than an executable (a "command" component).</p> <p>Components expose their interfaces via <a href="../design/wit.md">WebAssembly Interface Types</a>, hand-in-hand with the <a href="../design/why-component-model.md">Component Model</a> which enables components to use higher level types interchangeably.</p> <h4>Exporting WIT Interfaces with <code>jco</code></h4> <p>Packaging reusable functionality into WebAssembly components isn't useful if we have no way to <em>expose</em> that functionality.</p> <p>This section offers a slightly deeper dive into the usage of WIT in WebAssembly components that can use the Component Model.</p> <p>As in the previous example, <code>export</code>ing WIT interfaces for other components (or a WebAssembly host) to use is fundamental to developing WebAssembly programs.</p> <p>Let's examine a <a href="https://github.com/bytecodealliance/jco/tree/main/examples/components/string-reverse"><code>jco</code> example project called <code>string-reverse</code></a> that exposes functionality for reversing a string.</p> <p>To build a project like <code>string-reverse</code> from the ground up, first we'd start with a WIT like the following:</p> <pre><code class="language-wit">package example:string-reverse@0.1.0 @since(version = 0.1.0) interface reverse { reverse-string: func(s: string) -&gt; string; } world string-reverse { export reverse; } </code></pre> <p>As a slightly deeper crash course on <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md">WIT</a>, here's what the above code describes:</p> <ul> <li>We've defined a namespace called <code>example</code></li> <li>We've defined a package called <code>string-reverse</code> inside the <code>example</code> namespace</li> <li>This WIT file corresponds to version <code>0.1.0</code> of <code>example:string-reverse</code> package</li> <li>We've defined an interface called <code>reverse</code> which contains <em>one</em> function called <code>reverse-string</code></li> <li>We specify that the <code>reverse</code> interface has existed <em>since</em> the <code>0.1.0</code> version</li> <li>The <code>reverse-string</code> function (AKA. <code>example:reverse-string/reverse.reverse-string</code>) takes a string and returns a string</li> <li>We've defined a <code>world</code> called <code>string-reverse</code> which exports the functionality provided by the <code>reverse</code> interface</li> </ul> <blockquote> <p><a href="#">!WARNING</a><br> How do we <em>know</em> that <code>reverse</code> actually reverses a string?</p> <p>Unfortunately, that problem is not really solvable at this level -- this is between you and the writer of the component that implements the WIT interface.</p> <p>Of course, with WebAssembly, you <em>can</em> enforce static checks if you're so inclined, <em>before</em> you run any given binary.</p> </blockquote> <p>OK now let's see what the JS code looks like to <em>implement</em> the <code>component</code> world:</p> <pre><code class="language-js">/** * This module is the JS implementation of the `string-reverse` WIT world */ /** * This JavaScript will be interpreted by `jco` and turned into a * WebAssembly binary with a single export (this `reverse` function). */ function reverseString(s) { return s.reverse(); } /** * The JavaScript export below represents the export of the `reverse` interface, * which which contains `reverse-string` as it's primary exported function. */ export const reverse = { reverseString, }; </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> To view the full code listing along with instructions, see the <a href="https://github.com/bytecodealliance/jco/tree/main/examples/components/string-reverse"><code>examples/tutorials/jco/string-reverse</code> folder</a></p> </blockquote> <p>To use <code>jco</code> to compile this component, you can run the following from the <code>string-reverse</code> folder:</p> <pre><code class="language-console">npx jco componentize \ --wit wit/component.wit \ --world-name component \ --out string-reverse.wasm \ --disable all \ string-reverse.mjs </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> Like the previous example, we're not using any of the advanced <a href="https://wasi.dev/">WebAssembly System Interface</a> features, so we <code>--disable</code> all of them</p> <p>Rather than typing out the <code>jco componentize</code> command manually, you can also run<br> the build command with <a href="https://github.com/bytecodealliance/jco/blob/main/examples/components/string-reverse/package.json#L6"><code>npm run build</code> from the <code>string-reverse</code> folder</a>.</p> </blockquote> <p>You should see output like the following:</p> <pre><code>OK Successfully written string-reverse.wasm. </code></pre> <p>Now that we have a WebAssembly binary, we can <em>also</em> use <code>jco</code> to run it in a native JavaScript context by <em>transpiling</em> the WebAssembly binary (which could have come from anywhere!) to a JavaScript module.</p> <pre><code class="language-console">npx jco transpile string-reverse.wasm -o dist/transpiled </code></pre> <p>You should see the following output:</p> <pre><code> Transpiled JS Component Files: - dist/transpiled/interfaces/example-string-reverse-reverse.d.ts 0.1 KiB - dist/transpiled/string-reverse.core.wasm 10.1 MiB - dist/transpiled/string-reverse.d.ts 0.15 KiB - dist/transpiled/string-reverse.js 2.55 KiB </code></pre> <blockquote> <p><a href="#">!TIP</a><br> A gentle reminder that, transpilation <em>does</em> produce <a href="https://www.typescriptlang.org/docs/handbook/declaration-files/deep-dive.html#declaration-file-theory-a-deep-dive">Typescript declaration file</a>, for use in Typescript projects.</p> </blockquote> <p>Now that we have a transpiled module, we can run it from any JavaScript context that supports core WebAssembly (whether NodeJS or the browser).</p> <p>For NodeJS, we can use code like the following:</p> <pre><code class="language-mjs">// If this import listed below is missing, please run `npm run transpile` import { reverse } from "./dist/transpiled/string-reverse.mjs"; const reversed = reverse.reverseString("!dlroW olleH"); console.log(`reverseString('!dlroW olleH') = ${reversed}`); </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> In the <code>jco</code> example project, you can run <code>npm run transpiled-js</code> to build the existing code.</p> </blockquote> <p>Assuming you have the <code>dist/transpiled</code> folder populated (by running <code>jco transpile</code> in the previous step), you should see output like the following:</p> <pre><code>reverseString('!dlrow olleh') = hello world! </code></pre> <p>While it's somewhat redundant in this context, what we've done from NodeJS demonstrates the usefulness of WebAssembly and the <code>jco</code> toolchain. With the help of <code>jco</code>, we have:</p> <ul> <li>Compiled JavaScript to a WebAssembly module (<code>jco compile</code>), adhering to an interface defined via WIT</li> <li>Converted the compiled WebAssembly module (which could be from <em>any</em> language) to a module that can be used from any compliant JS runtime (<code>jco transpile</code>)</li> <li>Run the transpiled WebAssembly component from a JavaScript native runtime (NodeJS)</li> </ul> <h4>Advanced: Importing and Reusing WIT Interfaces via Composition</h4> <p>Just as <code>export</code>ing functionality is core to building useful WebAssembly components, and similarly <code>import</code>ing and reusing functionality is key to using the strengths of WebAssembly.</p> <p>Restated, <strong>WIT and the Component Model enable WebAssembly to <em>compose</em></strong>. This means we can build on top of functionality that already exists and <code>export</code> <em>new</em> functionality that depends on existing functionality.</p> <p>Let's say in addition to the reversing the string (in the previous example) we want to build shared functionality that <em>also</em> upper cases the text it receives.</p> <p>We can reuse the reversing functionality <em>and</em> export a new interface which enables us to reverse and upper-case.</p> <p>Let's examine a <a href="https://github.com/bytecodealliance/jco/tree/main/examples/components/string-reverse-upper"><code>jco</code> example project called <code>string-reverse-upper</code></a> that exposes functionality for reversing <em>and</em> upper-casing a string.</p> <p>Here's the WIT one might write to enable this functionality:</p> <pre><code class="language-wit">package example:string-reverse-upper@0.1.0; @since(version = 0.1.0) interface reversed-upper { reverse-and-uppercase: func(s: string) -&gt; string; } world revup { // // NOTE, the import below translates to: // &lt;namespace&gt;:&lt;package&gt;/&lt;interface&gt;@&lt;package version&gt; // import example:string-reverse/reverse@0.1.0; export reversed-upper; } </code></pre> <p>This time, the <code>world</code> named <code>revup</code> that we are building <em>relies</em> on the interface <code>reverse</code> in the package <code>string-reverse</code> from the namespace <code>example</code>.</p> <p>We can make use of <em>any</em> WebAssembly component that matches that interface, as long as we <em>compose</em> their functionality with the component that implements the <code>revup</code> world.</p> <p>The <code>revup</code> world <code>import</code>s (and makes use) of <code>reverse</code> in order to <code>export</code> (provide) the <code>reversed-upper</code> interface, which contains the <code>reverse-and-uppercase</code> function (in JS, <code>reverseAndUppercase</code>).</p> <blockquote> <p><a href="#">!NOTE</a><br> Functionality is imported via the <code>interface</code>, <em>not</em> the <code>world</code>. <code>world</code>s can be included/used, but the syntax is slightly different for that.</p> </blockquote> <p>The JavaScript to make this work (<a href="https://github.com/bytecodealliance/jco/blob/main/examples/components/string-reverse-upper/string-reverse-upper.mjs"><code>string-reverse-upper.mjs</code> in <code>jco/examples</code></a>) looks like this:</p> <pre><code class="language-js">/** * This module is the JS implementation of the `revup` WIT world */ /** * The import here is *virtual*. It refers to the `import`ed `reverse` interface in component.wit. * * These types *do not resolve* when the first `string-reverse-upper` component is built, * but the types are relevant for the resulting *composed* component. */ import { reverseString } from 'example:string-reverse/reverse@0.1.0'; /** * The JavaScript export below represents the export of the `reversed-upper` interface, * which which contains `revup` as it's primary exported function. */ export const reversedUpper = { /** * Represents the implementation of the `reverse-and-uppercase` function in the `reversed-upper` interface * * This function makes use of `reverse-string` which is *imported* from another WebAssembly binary. */ reverseAndUppercase() { return reverseString(s).toLocaleUpperCase(); }, }; </code></pre> <p>We can build the component with <code>jco componentize</code>:</p> <pre><code class="language-console">npx jco componentize \ string-reverse-upper.mjs \ --wit wit/ \ --world-name revup \ --out string-reverse-upper.incomplete.wasm \ --disable all </code></pre> <p>While we've successfully built a WebAssembly component, unlike the other examples, ours is <em>not yet complete</em>.</p> <p>We can see that if we print the WIT of the generated component by running <code>jco wit</code>:</p> <pre><code class="language-console">npx jco wit string-reverse-upper.incomplete.wasm </code></pre> <p>You should see output like the following:</p> <pre><code>package root:component; world root { import example:string-reverse/reverse@0.1.0; export example:string-reverse-upper/reversed-upper@0.1.0; } </code></pre> <p>This tells us that the component still has <em>unfulfilled <code>import</code>s</em> -- we <em>use</em> the <code>reverseString</code> function that's in <code>reverse</code> as if it exists, but it's not yet a real part of the WebAssembly component (hence we've named it <code>.incomplete.wasm</code>.</p> <p>To compose the two components (<code>string-reverse-upper/string-reverse-upper.incomplete.wasm</code> and <code>string-reverse/string-reverse.wasm</code> we built earlier), we'll need the <a href="https://github.com/bytecodealliance/wac">WebAssembly Composition tool (<code>wac</code>)</a>. We can use <code>wac plug</code>:</p> <pre><code class="language-console">wac plug \ -o string-reverse-upper.wasm \ --plug ../string-reverse/string-reverse.wasm \ string-reverse-upper.incomplete.wasm </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> You can also run this step with <code>npm run compose</code>.</p> </blockquote> <p>A new component <code>string-reverse-upper.wasm</code> should now be present, which is a "complete" component -- we can check the output of <code>jco wit</code> to ensure that all the imports are satisfied:</p> <pre><code class="language-wit">package root:component; world root { export example:string-reverse-upper/reversed-upper@0.1.0; } </code></pre> <p>It's as-if we never imported any functionality at all -- the functionality present in <code>string-reverse.wasm</code> has been <em>merged into</em> <code>string-reverse-upper.wasm</code>, and it now simply <code>export</code>s the advanced functionality.</p> <p>We can run this completed component with in any WebAssembly-capable native JavaScript environment by using a the transpiled result:</p> <pre><code class="language-console">npx jco transpile string-reverse-upper.wasm -o dist/transpiled </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> In the example project, you can run <code>npm run transpile</code> instead, which will also change the extension on <code>dist/transpiled/string-reverse-upper.js</code> to <code>.mjs</code></p> </blockquote> <p>You should see output like the following:</p> <pre><code> Transpiled JS Component Files: - dist/transpiled/interfaces/example-string-reverse-upper-reversed-upper.d.ts 0.12 KiB - dist/transpiled/string-reverse-upper.core.wasm 10.1 MiB - dist/transpiled/string-reverse-upper.core2.wasm 10.1 MiB - dist/transpiled/string-reverse-upper.d.ts 0.19 KiB - dist/transpiled/string-reverse-upper.js 6.13 KiB </code></pre> <blockquote> <p><a href="#">!TIP</a><br> Notice that there are <em>two</em> core WebAssembly files? That's because two core WebAssembly modules were involved<br> in creating the ultimate functionality we needed.</p> </blockquote> <p>To run the transpiled component, we can write code like the following:</p> <pre><code class="language-mjs">/** * If this import listed below is missing, please run * * ``` * npm run build &amp;&amp; npm run compose &amp;&amp; npm run transpile` * ``` */ import { reversedUpper } from "./dist/transpiled/string-reverse-upper.mjs"; const result = reversedUpper.reverseAndUppercase("!dlroW olleH"); console.log(`reverseAndUppercase('!dlroW olleH') = ${result}`); </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> In the <a href="https://github.com/bytecodealliance/jco/tree/main/examples/components/string-reverse-upper"><code>jco</code> example project</a>, you can run <code>npm run transpiled-js</code></p> </blockquote> <p>You should see output like the following:</p> <pre><code>reverseAndUppercase('!dlroW olleH') = HELLO WORLD! </code></pre> <h2>C/C++ Tooling</h2> <p>WebAssembly components can be built from C and C++ using <a href="https://clang.llvm.org/"><code>clang</code></a>, the C language family frontend for <a href="https://llvm.org/">LLVM</a>.</p> <p><a href="https://github.com/bytecodealliance/wit-bindgen"><code>wit-bindgen</code></a> is a tool to generate guest language bindings from a<br> given <code>.wit</code> file.</p> <p>Although <code>wit-bindgen</code> is a standalone tool (whereas some languages have more integrated toolchains like Rust's <a href="https://crates.io/crates/cargo-component"><code>cargo-component</code></a>),<br> <code>wit-bindgen</code> can generate source-level bindings for <code>Rust</code>, <code>C</code>, <code>Java (TeaVM)</code>, and <code>TinyGo</code>, with the ability for more<br> language generators to be added in the future.</p> <p><code>wit-bindgen</code> can be used to build C applications that can be compiled directly to Wasm modules using <a href="https://clang.llvm.org/"><code>clang</code></a> with a <a href="https://clang.llvm.org/docs/ClangCommandLineReference.html#webassembly"><code>wasm32-wasi</code></a> target.</p> <h3>1. Download dependencies</h3> <p>First, install the CLI for <a href="https://github.com/bytecodealliance/wit-bindgen#cli-installation"><code>wit-bindgen</code></a>, <a href="https://github.com/bytecodealliance/wasm-tools"><code>wasm-tools</code></a>, and the <a href="https://github.com/webassembly/wasi-sdk"><code>WASI SDK</code></a>.</p> <p>The WASI SDK will install a local version of <code>clang</code> configured with a wasi-sysroot. Follow <a href="https://github.com/WebAssembly/wasi-sdk#use">these instructions</a> to configure it for use. Note that you can also use your installed system or emscripten <code>clang</code> by building with <code>--target=wasm32-wasi</code> but you will need some artifacts from WASI SDK to enable and link that build target (more information is available in WASI SDK's docs).</p> <h3>2. Generate program skeleton from WIT</h3> <p>Start by generating a C skeleton from <code>wit-bindgen</code> using the <a href="https://github.com/bytecodealliance/component-docs/tree/main/examples/tutorial/wit/adder/world.wit">sample <code>adder/world.wit</code> file</a>:</p> <pre><code>&gt; wit-bindgen c path/to/adder/world.wit Generating "adder.c" Generating "adder.h" Generating "adder_component_type.o" </code></pre> <p>This has generated several files:</p> <p>1.<code>adder.h</code> (based on the <code>adder</code> world) with the prototype of the <code>add</code> function (prefixed by <code>exports_</code>) - <code>uint32_t exports_docs_adder_add_add(uint32_t x, uint32_t y);</code>.<br> 2. <code>adder.c</code> that interfaces with the component model ABI to call your function.<br> 3. <code>adder_component_type.o</code> which contains object code referenced in <code>adder.c</code> from an <code>extern</code> that must be linked via <code>clang</code>.</p> <h3>3. Write program code</h3> <p>Next, create an <code>component.c</code> that implements the <code>adder</code> world (i.e. the interface defined in <code>adder.h</code>):</p> <pre><code class="language-c">#include "adder.h" uint32_t exports_docs_adder_add_add(uint32_t x, uint32_t y) { return x + y; } </code></pre> <h3>4. Compile a WebAssembly module (P1) with <code>clang</code></h3> <p>Now, you can compile the function into a Wasm module via clang:</p> <pre><code class="language-console">clang component.c adder.c adder_component_type.o -o adder.wasm -mexec-model=reactor </code></pre> <blockquote> <p>Use the <code>clang</code> included in the WASI SDK installation, for example at <code>&lt;WASI_SDK_PATH&gt;/bin/clang</code>.</p> <p>Alternatively, you can also use the published <a href="https://github.com/WebAssembly/wasi-sdk/pkgs/container/wasi-sdk"><code>ghcr.io/webassembly/wasi-sdk</code> container images</a><br> for performing builds.</p> <p>For example, to enter a container with <code>wasi-sdk</code> installed:</p> <pre><code>docker run --rm -it --mount type=bind,src=path/to/app/src,dst=/app ghcr.io/webassembly/wasi-sdk:wasi-sdk-25 </code></pre> <p>See also: <a href="https://github.com/WebAssembly/wasi-sdk/blob/main/docker/Dockerfile"><code>Dockerfile</code> in <code>wasi-sdk</code></a></p> </blockquote> <h3>5. Convert the P1 component to a P2 component with <code>wasm-tools</code></h3> <p>Next, we need to transform the P1 component to a P2 component. To do this, we can use <code>wasm-tools component new</code>:</p> <pre><code class="language-console">wasm-tools component new ./adder.wasm -o adder.component.wasm </code></pre> <blockquote> <p><a href="#">!NOTE</a><br> The <code>.component.</code> extension has no special meaning -- <code>.wasm</code> files can be either modules or components.</p> </blockquote> <h3>6. (optional) Build a WASI-enabled WebAssembly (P2) component with <code>wasm-tools</code></h3> <p>Do note <code>wasm-tools component new</code> may fail if your code references any <a href="https://wasi.dev/">WASI</a> APIs that must be imported, for<br> example via standard library imports like <code>stdio.h</code>.</p> <p>Using WASI interfaces requires an additional step as the WASI SDK still references <code>wasi_snapshot_preview1</code> APIs that are not compatible directly with components.</p> <p>For example, modifying the above to reference <code>printf()</code> would compile:</p> <pre><code class="language-c">#include "adder.h" #include &lt;stdio.h&gt; uint32_t exports_docs_adder_add_add(uint32_t x, uint32_t y) { uint32_t result = x + y; printf("%d", result); return result; } </code></pre> <p>However, the module would fail to transform to a component:</p> <pre><code>&gt;wasm-tools component new ./adder.wasm -o adder.component.wasm error: failed to encode a component from module Caused by: 0: failed to decode world from module 1: module was not valid 2: module requires an import interface named `wasi_snapshot_preview1` </code></pre> <p>To build a P2 component that uses <a href="https://wasi.dev/">WASI</a> interfaces from a P1 component, we'll need to make use of adapter modules.</p> <p>Install the appropriate reactor adapter module <a href="https://github.com/bytecodealliance/wit-bindgen#creating-components-wasi">as documented here</a>.</p> <p>You can either get <a href="https://github.com/bytecodealliance/wasmtime/releases">the linked release</a> of <code>wasi_snapshot_preview1.reactor.wasm</code> and rename it to <code>wasi_snapshot_preview1.wasm</code>, or build it directly from source in <code>wasmtime</code> following the <a href="https://github.com/bytecodealliance/wasmtime/tree/main/crates/wasi-preview1-component-adapter">instructions here</a> (make sure you <code>git submodule update --init</code> first).</p> <p>Now, you can adapt preview1 to preview2 to build a component:</p> <pre><code class="language-console">wasm-tools component new adder.wasm --adapt wasi_snapshot_preview1.wasm -o adder.component.wasm </code></pre> <h3>7. Inspect the built component</h3> <p>Finally, you can inspect the embedded wit to see your component (including any WASI imports if necessary):</p> <pre><code>&gt;wasm-tools component wit adder.component.wasm package root:component; world root { import wasi:io/error@0.2.2; import wasi:io/streams@0.2.2; import wasi:cli/stdin@0.2.2; import wasi:cli/stdout@0.2.2; import wasi:cli/stderr@0.2.2; import wasi:cli/terminal-input@0.2.2; import wasi:cli/terminal-output@0.2.2; import wasi:cli/terminal-stdin@0.2.2; import wasi:cli/terminal-stdout@0.2.2; import wasi:cli/terminal-stderr@0.2.2; import wasi:clocks/wall-clock@0.2.2; import wasi:filesystem/types@0.2.2; import wasi:filesystem/preopens@0.2.2; export add: func(x: s32, y: s32) -&gt; s32; } ... </code></pre> <h3>8. Running the component from the example host</h3> <blockquote> <p><a href="#">!WARNING</a><br> You must be careful to use a version of the adapter (<code>wasi_snapshot_preview1.wasm</code>) that is compatible with the version of<br> <code>wasmtime</code> that will be used, to ensure that WASI interface versions (and relevant implementation) match.</p> </blockquote> <p>This repository contains an <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/example-host">example WebAssembly host</a> written in Rust that can run components that implement the <code>adder</code> world.</p> <blockquote> <p><a href="#">!NOTE</a><br> When hosts run components that use WASI interfaces, they must <em>explicitly</em> <a href="https://docs.wasmtime.dev/api/wasmtime_wasi/fn.add_to_linker_sync.html">add WASI to the linker</a> to run the built component.</p> </blockquote> <p>A successful run should show the following output:</p> <pre><code>cargo run --release -- 1 2 adder.component.wasm Compiling example-host v0.1.0 (/path/to/component-docs/component-model/examples/example-host) Finished `release` profile [optimized] target(s) in 7.85s Running `target/debug/example-host 1 2 /tmp/docs/c/adder.component.wasm` 1 + 2 = 3 </code></pre> <p>If <em>not</em> configured correctly, you may see errors like the following:</p> <pre><code>cargo run --release -- 1 2 adder.component.wasm Compiling example-host v0.1.0 (/path/to/component-docs/component-model/examples/example-host) Finished `release` profile [optimized] target(s) in 7.85s Running `target/release/example-host 1 2 adder.component.wasm` Error: Failed to instantiate the example world Caused by: 0: component imports instance `wasi:io/error@0.2.2`, but a matching implementation was not found in the linker 1: instance export `error` has the wrong type 2: resource implementation is missing </code></pre> <p>This kind of error normally indicates that the host in question does not contain satisfy WASI imports.</p> <h3>9. Running a Component from C/C++ Applications</h3> <p>It is not yet possible to run a WebAssembly Component using the C API of <code>wasmtime</code> <code>c-api</code>. See <a href="https://github.com/bytecodealliance/wasmtime/issues/6987"><code>wasmtime</code> issue #6987</a> for more details.</p> <p>The c-api is preferred over directly using the example host Rust crate in C++.</p> <p>However, C/C++ language guest components can be composed with components written in any other language and run by their toolchains, or even composed with a C language command component and run via the <code>wasmtime</code> CLI or any other host.</p> <p>See the <a href="../language-support/rust.md#running-a-component-from-rust-applications">Rust Tooling guide</a> for instructions on how to run this component from the Rust <code>example-host</code> (replacing the path to <code>add.wasm</code> with your <code>add-component</code> above).</p> <h2>C# Tooling</h2> <p>WebAssembly components in C# can be built with <a href="https://github.com/bytecodealliance/componentize-dotnet">componentize-dotnet</a>, a a NuGet package that can be used to create a fully AOT-compiled component, giving .NET developers a component experience comparable to those in Rust and TinyGo.</p> <h3>Building a Component with <code>componentize-dotnet</code></h3> <p><a href="https://github.com/bytecodealliance/componentize-dotnet"><code>componentize-dotnet</code></a> serves as a one-stop shop, wrapping several tools into one:</p> <ul> <li><a href="https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-LLVM">NativeAOT-LLVM</a> (compilation)</li> <li><a href="https://github.com/bytecodealliance/wit-bindgen">wit-bindgen</a> (WIT imports and exports)</li> <li><a href="https://github.com/bytecodealliance/wasm-tools">wasm-tools</a> (component conversion)</li> <li><a href="https://github.com/WebAssembly/wasi-sdk">WASI SDK</a> (SDK used by NativeAOT-LLVM)</li> <li><a href="https://github.com/bytecodealliance/wac">Wac</a> (used to compose components)</li> </ul> <p>First, install the .NET SDK. For this walkthrough, we’ll use the <a href="https://dotnet.microsoft.com/en-us/download/dotnet/10.0">.NET 10 SDK preview</a>.<br> You should also have <a href="https://wasmtime.dev/">wasmtime</a> installed so you can run the binary that you produce.</p> <h3>1. Create a new project</h3> <p>Once you have the .NET SDK installed, create a new project:</p> <pre><code class="language-sh">dotnet new install BytecodeAlliance.Componentize.DotNet.Templates dotnet new componentize.wasi.cli -o adder cd adder </code></pre> <h3>2. Create or download your WIT world</h3> <p>Next, create or download the WIT world you would like to target.</p> <p>For this example we will use the <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/tutorial/wit/adder/world.wit"><code>adder</code> world</a>, with an <code>add</code> function (e.g. to <code>wit/component.wit</code>):</p> <pre><code class="language-wit">package docs:adder@0.1.0; interface add { add: func(x: u32, y: u32) -&gt; u32; } world adder { export add; } </code></pre> <p>In the <code>adder.csproj</code> project file, add a new <code>&lt;ItemGroup&gt;</code>:</p> <pre><code class="language-xml">&lt;ItemGroup&gt; &lt;Wit Update="wit/component.wit" World="adder" /&gt; &lt;/ItemGroup&gt; </code></pre> <p>Since this component will only export a function dotnet considers this a library project.<br> Let's update the <code>&lt;OutputType&gt;</code> to be a library in the <code>adder.csproj</code>:</p> <pre><code class="language-diff">- &lt;OutputType&gt;Exe&lt;/OutputType&gt; + &lt;OutputType&gt;Library&lt;/OutputType&gt; </code></pre> <p>And remove the automatically generated <code>Program.cs</code> file:</p> <pre><code class="language-bash">rm Program.cs </code></pre> <h3>3. Write the implementation for the <code>adder</code> world</h3> <p>If you try to build the project with <code>dotnet build</code>, you'll get an error like the following:</p> <pre><code>➜ dotnet build Restore complete (8.6s) You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy adder failed with 1 error(s) (25.6s) /path/to/adder/obj/Debug/net10.0/wasi-wasm/wit_bindgen/AdderWorld.wit.exports.docs.adder.v0_1_0.AddInterop.cs(15,19): error CS0103: The name 'AddImpl' does not exist in the current context Build failed with 1 error(s) in 34.6s </code></pre> <p>This is because we've promised an implementation, but haven't yet written one for the <code>adder</code> world.</p> <p>To fix this, add the following code to your in a file called <code>Component.cs</code>:</p> <pre><code class="language-csharp">namespace AdderWorld; public class AddImpl : IAdderWorld { public static uint Add(uint x, uint y) { return x + y; } } </code></pre> <p>Then, we can build our component:</p> <pre><code class="language-sh">dotnet build </code></pre> <p>The component will be available at <code>bin/Debug/net10.0/wasi-wasm/native/adder.wasm</code>.</p> <h3>4. (optional) the component from the example host</h3> <blockquote> <p><a href="#">!WARNING</a><br> You must be careful to use a version of the adapter (<code>wasi_snapshot_preview1.wasm</code>) that is compatible with the version of<br> <code>wasmtime</code> that will be used, to ensure that WASI interface versions (and relevant implementation) match.</p> </blockquote> <p>This repository contains an <a href="https://github.com/bytecodealliance/component-docs/tree/main/component-model/examples/example-host">example WebAssembly host</a> written in Rust that can run components that implement the <code>adder</code> world.</p> <blockquote> <p><a href="#">!NOTE</a><br> When hosts run components that use WASI interfaces, they must <em>explicitly</em> <a href="https://docs.wasmtime.dev/api/wasmtime_wasi/fn.add_to_linker_sync.html">add WASI to the linker</a> to run the built component.</p> </blockquote> <p>A successful run should show the following output:</p> <pre><code>cargo run --release -- 1 2 adder.component.wasm Compiling example-host v0.1.0 (/path/to/component-docs/component-model/examples/example-host) Finished `release` profile [optimized] target(s) in 7.85s Running `target/debug/example-host 1 2 /tmp/docs/c/adder.component.wasm` 1 + 2 = 3 </code></pre> <p>If <em>not</em> configured correctly, you may see errors like the following:</p> <pre><code>cargo run --release -- 1 2 adder.component.wasm Compiling example-host v0.1.0 (/path/to/component-docs/component-model/examples/example-host) Finished `release` profile [optimized] target(s) in 7.85s Running `target/release/example-host 1 2 adder.component.wasm` Error: Failed to instantiate the example world Caused by: 0: component imports instance `wasi:io/error@0.2.2`, but a matching implementation was not found in the linker 1: instance export `error` has the wrong type 2: resource implementation is missing </code></pre> <p>This kind of error normally indicates that the host in question does not contain satisfy WASI imports.</p> <h3>Building a component that exports an interface</h3> <p>The previous example uses a WIT file that exports a function. However, you'll often prefer to export an interface, either to comply with an existing specification or to capture a set of functions and types that tend to go together. Let's expand our <code>example</code> world to export an interface rather than directly export the function. We are also adding the <code>hostapp</code> world to our WIT file which we will implement in <a href="#building-a-component-that-imports-an-interface">the next section</a> to demonstrate how to build a component that <em>imports</em> an interface.</p> <pre><code class="language-wit">// adder/world.wit package example:component; interface add { add: func(x: u32, y: u32) -&gt; u32; } world example { export add; } world hostapp { import add; } </code></pre> <p>If you peek at the bindings, you'll notice that we now implement a class for the <code>add</code> interface rather than for the <code>example</code> world -- this is a consistent pattern. As you export more interfaces from your world, you implement more classes.</p> <p>Our <code>Component.cs</code> example gets the slight update of:</p> <pre><code class="language-csharp">namespace ExampleWorld.wit.exports.example.component; public class AddImpl : IAdd { public static uint Add(uint x, uint y) { return x + y; } } </code></pre> <p>Once again, compile an application to a Wasm component using <code>dotnet build</code>:</p> <pre><code class="language-sh">$ dotnet build Restore complete (0.4s) You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy adder succeeded (1.1s) → bin/Debug/net10.0/wasi-wasm/adder.dll Build succeeded in 2.5s </code></pre> <p>The component will be available at <code>bin/Debug/net10.0/wasi-wasm/native/adder.wasm</code>.</p> <h3>Building a component that imports an interface</h3> <p>So far, we've been dealing with library components. Now we will be creating a command component that implements the <code>hostapp</code> world. This component will import the <code>add</code> interface that is exported from our <code>adder</code> component and call the <code>add</code> function. We will later compose this command component with the <code>adder</code> library component we just built.</p> <p>Now we will be taking the <code>adder</code> component and executing it from another WebAssembly component.</p> <p><code>dotnet new componentize.wasi.cli</code> creates a new project that creates an executable.</p> <p>Back out of the current project and create a new one:</p> <pre><code class="language-sh">cd .. dotnet new componentize.wasi.cli -o host-app cd host-app </code></pre> <p>Copy the same WIT file as before into your project:</p> <pre><code class="language-wit">// adder/world.wit package example:component; interface add { add: func(x: u32, y: u32) -&gt; u32; } world example { export add; } world hostapp { import add; } </code></pre> <p>Add it to your <code>host-app.csproj</code> project file as a new <code>ItemGroup</code>:</p> <pre><code class="language-xml">&lt;ItemGroup&gt; &lt;Wit Update="adder/add.wit" World="hostapp" /&gt; &lt;/ItemGroup&gt; </code></pre> <p>Notice how the <code>World</code> changed from <code>example</code> to <code>hostapp</code>. The previous examples focused on implementing the class library for this WIT file - the <code>export</code> functions. Now we'll be focusing on the executable side of the application - the <code>hostapp</code> world.</p> <p>Modify <code>Program.cs</code> to look like this:</p> <pre><code class="language-csharp">// Pull in all imports of the `hostapp` world, namely the `add` interface. // example.component refers to the package name defined in the WIT file. using HostappWorld.wit.imports.example.component; uint left = 1; uint right = 2; var result = AddInterop.Add(left, right); Console.WriteLine($"{left} + {right} = {result}"); </code></pre> <p>Once again, compile your component with <code>dotnet build</code>:</p> <pre><code class="language-sh">$ dotnet build Restore complete (0.4s) You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy host-app succeeded (1.1s) → bin/Debug/net10.0/wasi-wasm/host-app.dll Build succeeded in 2.5s </code></pre> <p>At this point, you'll have two Webassembly components:</p> <ol> <li>A component that implements the <code>example</code> world.</li> <li>A component that implements the <code>hostapp</code> world.</li> </ol> <p>Since the <code>host-app</code> component depends on the <code>add</code> function which is defined in the <code>example</code> world, it needs to be composed the first component. You can compose your <code>host-app</code> component with your <code>adder</code> component by running <a href="https://github.com/bytecodealliance/wac"><code>wac plug</code></a>:</p> <pre><code class="language-sh">wac plug \ bin/Debug/net10.0/wasi-wasm/native/host-app.wasm \ --plug ../adder/bin/Debug/net10.0/wasi-wasm/native/adder.wasm \ -o main.wasm </code></pre> <p>You can also automate the process by adding the following to your <code>host-app.csproj</code>:</p> <pre><code class="language-xml">&lt;Target Name="ComposeWasmComponent" AfterTargets="Publish"&gt; &lt;PropertyGroup&gt; &lt;EntrypointComponent&gt;bin/$(Configuration)/$(TargetFramework)/wasi-wasm/native/host-app.wasm&lt;/EntrypointComponent&gt; &lt;DependencyComponent&gt;../adder/bin/$(Configuration)/$(TargetFramework)/wasi-wasm/native/adder.wasm&lt;/DependencyComponent&gt; &lt;/PropertyGroup&gt; &lt;MakeDir Directories="dist" /&gt; &lt;Exec Command="$(WacExe) plug $(EntrypointComponent) --plug $(DependencyComponent) -o dist/main.wasm" /&gt; &lt;/Target&gt; </code></pre> <p>Run <code>dotnet build</code> again you will have a composed component in <code>./dist/main.wasm</code></p> <p>Then you can run the composed component:</p> <pre><code class="language-sh">wasmtime run ./dist/main.wasm 1 + 2 = 3 </code></pre> <p>Check out the <a href="https://github.com/bytecodealliance/componentize-dotnet">componentize-dotnet docs</a> for more configurations options.</p> <hr> <h1>Building a Component with <code>wasm-tools</code></h1> <p><code>wasm-tools</code> can be used to create a component from WebAssembly Text (WAT). This walks through creating a component from WAT that implements the <a href="https://github.com/bytecodealliance/component-docs/blob/main/component-model/examples/tutorial/wit/adder/world.wit"><code>adder</code> world</a> and simply adds two numbers.</p> <ol> <li>Install</li> <li>The <code>add</code> function is defined inside the following <code>world</code> world:</li> </ol> <pre><code class="language-wit">package docs:adder@0.1.0; interface add { add: func(x: u32, y: u32) -&gt; u32; } world adder { export add; } </code></pre> <ol> <li>Define an <code>add</code> core module in WAT that exports an <code>add</code> function that adds two parameters:</li> </ol> <pre><code class="language-wat">(module (func $add (param $lhs i32) (param $rhs i32) (result i32) local.get $lhs local.get $rhs i32.add) (export "docs:adder/add@0.1.0" (func $add)) ) </code></pre> <ol> <li>Use <code>wasm-tools</code> to create a component from the core module, first embedding component metadata<br> inside the core module and then encoding the WAT to a Wasm binary.</li> </ol> <pre><code class="language-sh">$ wasm-tools component embed adder/world.wit add.wat -o add.wasm $ wasm-tools component new add.wasm -o add.component.wasm </code></pre> <h3>Running a Component with Wasmtime</h3> <p>You can "run" a component by calling one of its exports. Hosts and runtimes often only support running components with certain exports. The <a href="https://github.com/bytecodealliance/wasmtime"><code>wasmtime</code></a> CLI can only run "command" components, so in order to run the <code>add</code> function above, it first must be composed with a primary "command" component that calls it. See <a href="./creating-and-consuming/running.md">documentation on running components</a> for more details.</p> <h2>Useful links</h2> <p>The following references are helpful in understanding the Component Model and related ecosystem/projects.</p> <ul> <li><a href="https://github.com/bytecodealliance/wac">WebAssembly Composition tool (<code>wac</code>)</a></li> <li><a href="https://github.com/bytecodealliance/wasm-pkg-tools">WebAssembly package tools (notably <code>wkg</code>)</a></li> <li><a href="https://github.com/WebAssembly/WASI/tree/main/wasip2">WASI Preview 2</a></li> <li><a href="https://github.com/WebAssembly/component-model">Component Model internals</a></li> <li><a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md">Component Model AST</a></li> <li><a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md">Canonical ABI</a></li> </ul> https://tedneward.github.io/Research/tools/wikmd/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/wikmd/index.html <p><a href="https://github.com/Linbreux/wikmd">Source</a></p> https://tedneward.github.io/Research/tools/witchcraft/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/witchcraft/index.html <p><a href="https://github.com/endrazine/wcc/wiki">User manual</a></p> <p>wld: The Witchcraft Linker<br> - wld takes an ELF executable as an input and modifies it to create a shared library.</p> <p>wcc: The Witchcraft Compiler<br> - The wcc compiler takes binaries (ELF, PE, ...) as an input and creates valid ELF binaries as an output. It can be used to create relocatable object files from executables or shared libraries.</p> <p>wsh: The Witchcraft shell<br> - The witchcraft shell accepts ELF shared libraries, ELF ET_DYN executables and Witchcraft Shell Scripts written in Punk-C as an input. It loads all the executables in its own address space and makes their API available for programming in its embedded interpreter. This provides for binaries functionalities similar to those provided via reflection on languages like Java.</p> <p>wldd: print shared libraries compilation flags</p> <p>wcch: generate C headers from binaries</p> <p><a href="https://github.com/endrazine/wcc">Github</a></p> https://tedneward.github.io/Research/tools/vmlens/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/vmlens/index.html <p><a href="https://vmlens.com/">Website</a> | <a href="https://github.com/vmlens/vmlens">Source</a></p> https://tedneward.github.io/Research/tools/watc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/watc/index.html <p><a href="https://flak.tedunangst.com/post/watc">Original Post</a> | <a href="https://humungus.tedunangst.com/r/watc">Source</a> (Go)</p> <p><a href="https://nullprogram.com/blog/2022/05/22/">Another take</a> (in C) | <a href="https://github.com/skeeto/scratch/blob/master/misc/watc.c">Source</a></p> https://tedneward.github.io/Research/tools/winboat/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/winboat/index.html <p><a href="https://www.winboat.app/">Website</a></p> https://tedneward.github.io/Research/tools/x11/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/x11/index.html <p><a href="http://cyber.dabamos.de/unix/x11/">Cool, but obscure, X11 tools</a>:</p> <ul> <li> <p><a href="http://www.newbreedsoftware.com/3dpong/index2.php3">3D Pong</a> is three-dimensional version of the first arcade game, “Pong”, for the X Window System.</p> <pre><code> # pkg install games/3dpong </code></pre> <p>To play against the computer, run:</p> <pre><code> $ 3dpong :0 --computer --sound </code></pre> </li> <li> <p>Angband: A roguelike game derived from Moria, based loosely on the books of J. R. R. Tolkien. The ultimate aim of the game is to advance in skill and strength, collecting better and more powerful magical items until you are ready to face the Master of the dungeon: Morgoth himself! Start the X11 mode with <code>angband -mx11</code>.</p> <pre><code># pkg install games/angband </code></pre> </li> <li> <p>AppRes: Prints the resources seen by an application. It can be used to determine which resources a particular program will load.</p> <pre><code># pkg install x11/appres </code></pre> </li> <li> <p>ASClock: AfterStep clock with some language extensions.</p> <pre><code># pkg install x11-clocks/asclock </code></pre> </li> <li> <p>Battalion: Battalion is a 3D game from 1994, originally written for the Silicon Graphics Indy UNIX workstation. The player controls a monster to blow up a city. The author dedicated a website to the game.</p> <p>Depending on the perspective, the controls are a little awkward. You may want to change the camera with key 1, 2, 3, or 4. Use the mouse to rotate and to move forward/backward. The arrow keys only work if the mouse cursor is placed in the center of the screen. Shoot with the left control key or the left mouse button, and tilt the head with A and Z to target helicopters and jets. The camera can be rotated with I, J, K, and L. Alter the rendering details with D. Press G to grab the mouse cursor and P to pause the game.</p> <p>I made a quick &amp; dirty FreeBSD port based on the Linux version (no sound). Build and run Battalion with:</p> <pre><code>$ tar xfvz battalion1.4b-freebsd.tar.gz $ cd battalion1.4b-freebsd/ $ gmake battalion CC=gcc $ ./battalion </code></pre> </li> <li> <p>Free42: Simulation of the HP-42S scientific calculator and HP-82240 printer.</p> <pre><code># pkg install misc/free42 </code></pre> </li> <li> <p>FSV2: A 3D filesystem visualiser, like fsn for SGI IRIX. Prone to crashing (perhaps, UTF-8 issues).</p> <pre><code># pkg install x11-fm/fsv2 </code></pre> </li> <li> <p>GLXGears: A simple OpenGL tool for testing your graphics adapter.</p> <pre><code># pkg install graphics/mesa-demos </code></pre> </li> <li> <p>GMixer: Audio mixer for X11, based on now deprecated GTK+ 1.2.</p> <pre><code># pkg install audio/gmixer </code></pre> </li> <li> <p>GVim: Improved version of the vi editor. The graphical front-end can be compiled with either X11, Motif, Xaw, or GTK+ 2/3.</p> <pre><code># pkg install editors/vim </code></pre> </li> <li> <p>HP-15c: Emulator of the HP-15C programmable calculator, written in Tcl/Tk. The source-code is available on the <a href="https://hp-15c.homepage.t-online.de/">official website</a>.</p> <pre><code>$ unzip HP-15C_4.1.00_Source.zip $ wish8.6 ./HP-15C.tcl </code></pre> </li> <li> <p>Ico: Displays wire-framed rotating polyhedrons.</p> <pre><code># pkg install x11/ico </code></pre> </li> <li> <p>Mgdiff: Motif-based front-end to the Unix diff command. The appearance of mgdiff is based upon a program called gdiff, which runs only on Silicon Graphics workstations and for which source code is not provided.</p> <pre><code># pkg install textproc/mgdiff </code></pre> </li> <li> <p>Mxico: Maho’s eXtended Ico is a simple demo for X, rotating polyhedra in the window. See <a href="https://people.freebsd.org/~maho/mxico/Zalgaller.html">this table</a> for reference and start MxIco with, for example, <code>mxico -sleep 1 -obj n65</code>.</p> <pre><code># pkg install x11/mxico </code></pre> </li> <li> <p>Micropolis: Open-source version of the famous computer game SimCity. Available on <a href="https://github.com/interkosmos/micropolis">GitHub</a>.</p> <pre><code>$ git clone https://github.com/interkosmos/micropolis </code></pre> </li> <li> <p>NEdit: X11/Motif GUI text editor for programs and plain text files.</p> <pre><code># pkg install editors/nedit </code></pre> </li> <li> <p>Plan: Plan is a schedule planner based on X/Motif.</p> <pre><code># pkg install deskutils/plan </code></pre> </li> <li> <p>Sunclock: A fancy clock for the X Window system, providing local time (legal time and solar time), sunrise, sunset and various geographical data through a point and click interface.</p> <pre><code># pkg install astro/sunclock </code></pre> </li> <li> <p>Ted: A simple RTF editor for X11. Can either be compiled with Motif or GTK+ 2. The binary starts with an upper-case letter (Ted). Prone to crashing.</p> <pre><code># pkg install editors/ted </code></pre> </li> <li> <p>TIEmu: Emulator for Texas Instruments TI-89/TI-92 (Plus)/Voyage 200 graphical calculators. ROM files not included. TIGCC is a compatible C compiler, based on GCC, and available as a FreeBSD port as well.</p> <pre><code># pkg install emulators/tiemu3 </code></pre> </li> <li> <p>WordNet: WordNet is a lexical reference system that combines aspects of dictionaries and thesauri with psycholinguistic theories of human lexical memory. It is developed by the Cognitive Science Laboratory at Princeton University.</p> <p>The graphical WordNet browser wnb depends on Tcl/Tk:</p> <pre><code># pkg install x11-toolkits/tk86 textproc/wordnet </code></pre> </li> <li> <p>X026: X026 is an keypunch emulator, simulating the IBM 026. Perfect for writing FORTRAN code on Hollerith punch cards and getting a clearer picture of how laborious data input back then was. The default character set is BCD-H.</p> <pre><code># pkg install deskutils/x026 </code></pre> </li> <li> <p>X48: Emulator of the HP 48GX graphing calculator. ROM image is included.</p> <pre><code># pkg install emulators/x48 </code></pre> </li> <li> <p>XAbacus: Simply an abacus for the X Window System.</p> <pre><code># pkg install games/xabacus </code></pre> </li> <li> <p>XAntfarm: Ant hill simulation on the X11 root window.</p> <pre><code># pkg install x11/xantfarm </code></pre> </li> <li> <p>XArchiver: XArchiver is a GTK+ 2 front-end to various archiving tools (tar, zip, 7z, …).</p> <pre><code># pkg install archivers/xarchiver </code></pre> </li> <li> <p>XASCII: XASCII displays the ASCII table in hex, decimal, and octal.</p> <pre><code># pkg install x11/xascii </code></pre> </li> <li> <p>XBiff: XBiff shows a small mailbox with its flag raised when new mail arrives in your inbox. It simply monitors the size of a given file. For instance, the cache of the Sylpheed e-mail client can be watched with:</p> <pre><code>$ xbiff -file $HOME/.sylpheed-2.0/imapcache/&lt;server&gt;/&lt;address&gt;/INBOX/.sylpheed_cache </code></pre> <p>You can change the pixmap in ~/.Xdefaults:</p> <pre><code>xbiff*flip: false xbiff*fullPixmap: mailfull xbiff*emptyPixmap: mailempty xbiff*fullPixmapMask: mailfullmsk xbiff*emptyPixmapMask: mailemptymsk xbiff*shapeWindow: true </code></pre> <p>Install it with:</p> <pre><code># pkg install x11/xbiff </code></pre> </li> <li> <p>XBill: XBill is an arcade game from 1994 in which the player has to prevent an evil computer hacker named “Bill” from installing a popular operating system on various machines. The OS can spread over networks and purges existing systems (BSD, Solaris, SGI IRIX, …). A contemporary modification of the game is known as XLennart.</p> <pre><code># pkg install games/xbill </code></pre> </li> <li> <p>XBoard: XBoard is a graphical front-end for chess engines like GNU Chess. Start it with xboard -fcp 'gnuchess --xboard'.</p> <pre><code># pkg install games/xboard </code></pre> </li> <li> <p>XCalc: A simple and clean scientific calculator for X11. Additional colour styles imitating the Texas Instruments TI-30 and the Hewlett-Packard 10C are listed in the file /usr/local/share/X11/app-defaults/XCalc-color. Just copy one of the styles into your ~/.Xdefaults.</p> <pre><code># pkg install x11/xcalc </code></pre> </li> <li> <p>XCalendar: A calendar with an integrated notebook.</p> <pre><code># pkg install deskutils/xcalendar </code></pre> </li> <li> <p>XCHM: Microsoft Windows help file viewer for Unix.</p> <pre><code># pkg install deskutils/xchm </code></pre> </li> <li> <p>XChomp: Clone of the famous arcade game Pac-Man for X. See also the unofficial homepage.</p> <pre><code># pkg install games/xchomp </code></pre> </li> <li> <p>XClipboard: A client for clipboard management in X.</p> <pre><code># pkg install x11/xclipboard </code></pre> </li> <li> <p>XClock: An analog and digital clock for X.</p> <pre><code># pkg install x11-clocks/xclock </code></pre> </li> <li> <p>XClock/Cat Clock: A Motif-based variant of MIT xclock from 1990 that features a “cat” mode. Available on GitHub.</p> <pre><code># git clone https://github.com/BarkyTheDog/catclock </code></pre> </li> <li> <p>XColorSel: Grabs the colours of X windows.</p> <pre><code># pkg install x11/xcolorsel </code></pre> </li> <li> <p>XConsole: XConsole shows system console messages.</p> <pre><code># pkg install x11/xconsole </code></pre> </li> <li> <p>XCrySDen: XCrySDen is a crystalline and molecular structure visualisation program aiming at display of isosurfaces and contours, which can be superimposed on crystalline structures and interactively rotated and manipulated.</p> <pre><code># pkg install science/xcrysden </code></pre> </li> <li> <p>XDiary: XDiary will help you keep track of your meetings, appointments and plan your time.</p> <pre><code># pkg install deskutils/xmdiary </code></pre> </li> <li> <p>XDigger: Boulderdash-like game for the X Window System. Originally written for the East-German KC 85/3 and KC 85/4 computers in 1988 (see also the author’s website). Unfortunately, the game suffers from a major bug that prevents the rocks from falling.</p> <pre><code># pkg install games/xdigger </code></pre> </li> <li> <p>XEarth: Renders the earth in 3D on the root window, making itself an animated backdrop. To show it inside a window, run:</p> <pre><code>$ xearth -geometry 800x600 -shade -nolabel -wait 1 -timewarp 50 -noroot -twopix </code></pre> <p>Similiar tools are XWorld, XGlobe and XPlanet, which also texture the earth.</p> <pre><code># pkg install astro/xearth </code></pre> </li> <li> <p>XEdit: A simple text editor for X. The automated backup function for edited files can be disabled by adding xedit*enableBackups: off to ~/.Xdefaults.</p> <pre><code># pkg install x11/xedit </code></pre> </li> <li> <p>Xev: Prints contents of X events to the terminal.</p> <pre><code># pkg install x11/xev </code></pre> </li> <li> <p>XEyes: Eyes following the mouse cursor.</p> <pre><code># pkg install x11/xeyes </code></pre> </li> <li> <p>XFishtank: Make fish swim in the background of your screen to waste CPU cycles.</p> <pre><code># pkg install x11/xfishtank </code></pre> <p>Currently, the port is unfetchable. I patched the Linux version manually to run on FreeBSD:</p> <pre><code>$ tar xfvz xfishtank-2.2.tar.gz $ cd xfishtank2.0/ $ gmake CC=gcc $ ./xfishtank -d </code></pre> </li> <li> <p>XFontSel: Lists installed fonts and available font options.</p> <pre><code># pkg install x11-fonts/xfontsel </code></pre> </li> <li> <p>XGalaga: Clone of the classic game Galaga for X11.</p> <pre><code># pkg install games/xgalaga </code></pre> </li> <li> <p>Xgc: X11 program that demonstrates various features of the X graphics primitives.</p> <pre><code># pkg install x11/xgc </code></pre> </li> <li> <p>XInfocom: Infocom game interpreter for X11.</p> <pre><code># pkg install games/xinfocom </code></pre> </li> <li> <p>XInvaders 3D: A 3D vector-graphics Space Invaders clone for X11.</p> <pre><code># pkg install games/xinvaders3d </code></pre> </li> <li> <p>XKill: Command-line tool to kill X applications.</p> <pre><code># pkg install x11/xkill </code></pre> </li> <li> <p>XLaby: Daemonic X Labyrinth played directly with the mouse pointer.</p> <pre><code># pkg install games/xlaby </code></pre> </li> <li> <p>XLennart: XLennart is a modification of the arcade game XBill. An evil and unpopular computer hacker named “Lennart” tries to install his malicious init system on various BSD and Linux systems. Like in XBill, the player has to hit him and restore infected machines.</p> <pre><code># pkg install games/xlennart </code></pre> </li> <li> <p>XLess: XLess is less for X.</p> <pre><code># pkg install misc/xless </code></pre> </li> <li> <p>XLife: Cellular automaton for X11 simulating Conway’s Game of Life.</p> <pre><code># pkg install games/xlife </code></pre> </li> <li> <p>XLoad: XLoad shows the current CPU load.</p> <pre><code># pkg install x11/xload </code></pre> </li> <li> <p>XLock: XLock locks the screen and optionally shows a screen saver. It can be combined with XAutoLock to activate it after a user defined time of inactivity.</p> <p>A more simple approach is used by XTransparentLock. Jamie Zawinski, the author of XScreenSaver, has listed some reason why you should use XScreenSaver instead of XLock.</p> <pre><code># pkg install x11/xlockmore </code></pre> </li> <li> <p>XLogo: XLogo simply draws the logo of the X Window System. The background and foreground colours can be altered in ~/.Xdefaults.</p> <pre><code># pkg install x11/xlogo </code></pre> </li> <li> <p>XMahjongg: Classic Mahjongg game for X11. Several tile sets are available.</p> <pre><code># pkg install games/xmahjongg </code></pre> </li> <li> <p>XMan: XMan displays system man pages.</p> <pre><code># pkg install x11/xman </code></pre> </li> <li> <p>XMascot: XMascot displays a moving mascot on your X11 screen. It can talk, set an alarm, or check your mail.</p> <pre><code># pkg install x11/xmascot </code></pre> </li> <li> <p>XMaxima: XMaxima is a Tcl/Tk front-end to Maxima, a Computer Algebra System written in Common Lisp.</p> <pre><code># pkg install math/xmaxima </code></pre> </li> <li> <p>XmBibTeX: XmBibTeX is a Motif reference manager based on the BibTeX file format. It allows to add, delete, and edit references. The references can be saved in the BibTeX file format and also written on a LaTeX file that can be printed using LaTeX and BibTeX. No Unicode support.</p> <pre><code># pkg install print/xmbibtex </code></pre> </li> <li> <p>XMessage: XMessage displays arbitrary text messages.</p> <pre><code># pkg install x11/xmessage </code></pre> </li> <li> <p>XmGrace: A WYSIWYG 2D plotting tool for the X Window System and Motif. Features polynomial regression, splines, running averages, DFT/FFT, and cross/auto-correlation.</p> <pre><code># pkg install math/grace </code></pre> </li> <li> <p>XMH: Graphical user interface to the MH Message Handling System.</p> <pre><code># pkg install x11/xmh </code></pre> </li> <li> <p>XMixer: Audio mixer for X11.</p> <pre><code># pkg install audio/xmixer </code></pre> </li> <li> <p>XmMix: Motif-based audio mixer for X11.</p> <pre><code># pkg install audio/xmmix </code></pre> </li> <li> <p>XMore: XMore is more for X.</p> <pre><code># pkg install x11/xmore </code></pre> </li> <li> <p>XMosaic: NCSA Mosaic, one of the first graphical web browsers, for X. The source code of version 2.7b is available on GitHub.</p> <pre><code># git clone https://github.com/alandipert/ncsa-mosaic </code></pre> </li> <li> <p>XMOTD: XMOTD shows the message of the day. Run it with xmotd -always /etc/motd. The appearance can be altered in ~/.Xdefaults.</p> <pre><code># pkg install x11/xmotd </code></pre> </li> <li> <p>XMountains: XMountains renders a mountain scene into the root window. Also available as an XScreenSaver module.</p> <pre><code># pkg install graphics/xmountains </code></pre> </li> <li> <p>XNeko:Neko (猫) is a cat chasing the mouse cursor all over the screen. Oneko is a modified version of xneko for BSD and Linux. In the port games/oneko-sakura several other creatures can be choosen, like tora-neko, Sakura Kinomoto, Tomoyo Daidouji, the BSD daemon, or a dog.</p> <pre><code># pkg install games/xneko </code></pre> <p>Or:</p> <pre><code># pkg install games/oneko </code></pre> <p>Or:</p> <pre><code># pkg install games/oneko-sakura </code></pre> </li> <li> <p>XOdometer: XOdometer tracks the total distance of your pointing device. The source code of the version from January 1996 is available on GitHub</p> <pre><code># git clone https://github.com/interkosmos/xodo </code></pre> </li> <li> <p>XOJ: XOJ lets the O. J. Simpson chase relive on your X root window.</p> <pre><code># pkg install games/xoj </code></pre> </li> <li> <p>XOSView: XOSView is a visual system monitor for X11.</p> <pre><code># pkg install sysutils/xosview </code></pre> </li> <li> <p>Xplore: Yet another Motif-based file manager. Prone to crashing (probably, encoding issues).</p> <pre><code># pkg install x11-fm/xplore </code></pre> </li> <li> <p>XPostIt: The most ugly note taking application on earth.</p> <pre><code># pkg install deskutils/xpostit </code></pre> </li> <li> <p>XRoach: XRoach displays disgusting cockroaches on your root window. These creepy crawlies scamper around until they find a window to hide under. Whenever you move or iconify a window, the exposed beetles again scamper for cover.</p> <pre><code># pkg install games/xroach </code></pre> </li> <li> <p>XScreenSaver: Jamie Zawinski’s famous screen saver for X.</p> <pre><code># pkg install x11/xscreensaver </code></pre> </li> <li> <p>XSnow: Let it snow on your desktop. Shows snow flakes, trees, and santa.</p> <pre><code># pkg install x11/xsnow </code></pre> </li> <li> <p>XSol: Motif-based solitaire game from 1998, similar to Klondike. Another implementation with the same name from 1987 had been ported to FreeBSD, but was later removed due to dependencies to the deprecated X10 headers. I haven’t found a port of this XSol, but the game can still be compiled from source:</p> <pre><code>$ tar xfvz xsol-0.3.1.tar.gz $ cd xsol/ $ make $ ./xsol </code></pre> </li> <li> <p>XSpread: XSpread is a simple spreadsheet application based on math/sc. It can be started without X11 by using the parameter -X.</p> <pre><code># pkg install math/xspread </code></pre> </li> <li> <p>XStarRoll: XStarRoll is a simple demonstration program for X11. Letters and pixmaps fly far, far, far away, optionally on the root window. Start with, for example: ls -al | xstarroll -font "-<em>-courier-medium-r-</em>-<em>-20-</em>-<em>-</em>-<em>-</em>-<em>-</em>" -bg "black" -fg "yellow" -geometry 640x400 -noblur.</p> <pre><code># pkg install x11/xstarroll </code></pre> </li> <li> <p>Xtacy: A trippy colour-cycling toy for X11. Displays various graphical effects and fractals, like bouncing quadrilaterals, plasma clouds, or phased sine waves. May run too fast on modern computers. See also the official website.</p> <pre><code># pkg install x11/xtacy </code></pre> </li> <li> <p>XTar: XTar is a Motif-based tool for viewing and manipulating (compressed) tar archives.</p> <pre><code># pkg install misc/xtar </code></pre> </li> <li> <p>XTerm: Terminal emulator for the X Window System.</p> <pre><code># pkg install x11/xterm </code></pre> </li> <li> <p>XTide: Harmonic tide clock and tide predictor for X.</p> <pre><code># pkg install astro/xtide </code></pre> </li> <li> <p>XTrojka: A Tetris-like game where the player has to control and place falling blocks to create three patterns or colors in a row, horizontally or diagonally.</p> <pre><code># pkg install games/xtrojka </code></pre> </li> <li> <p>Xv: Image viewer that displays various formats.</p> <pre><code># pkg install graphics/xv </code></pre> </li> <li> <p>Xvkbd: Virtual keyboard for X applications.</p> <pre><code># pkg install x11/xvkbd </code></pre> </li> <li> <p>XWorld: Earth as seen from the direction of the sun.</p> <pre><code># pkg install astro/xworld </code></pre> </li> <li> <p>XWPE: A Borland-like integrated development environment (IDE).</p> <pre><code># pkg install devel/xwpe </code></pre> </li> <li> <p>XZoom: Tool to magnify, rotate, and mirror a section of the X screen.</p> <pre><code># pkg install x11/xzoom </code></pre> </li> </ul> https://tedneward.github.io/Research/tools/voiceliner/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/voiceliner/index.html <p><a href="https://a9.io/voiceliner/">Website</a> | <a href="https://github.com/maxkrieger/voiceliner">Source</a></p> https://tedneward.github.io/Research/tools/whamm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/whamm/index.html <p><a href="https://github.com/ejrgilbert/whamm">Source</a> | <a href="https://ejrgilbert.github.io/whamm/intro.html">Book</a></p> https://tedneward.github.io/Research/tools/winstall/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/winstall/index.html <p><a href="https://winstall.app/">Website</a> | <a href="https://github.com/MehediH/winstall">Source</a></p> https://tedneward.github.io/Research/tools/xcode/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/xcode/index/index.html <p>See also <a href="../xcodebuild">xcodebuild</a>, <a href="/languages/swift">Swift</a>, <a href="/languages/objc">Objective-C</a></p> <p><a href="https://github.com/RobotsAndPencils/xcodes">xcodes</a>: Command-line utility to manage multiple XCode installations (multiple versions, locations, etc); install with <code>brew install robotsandpencils/made/xcodes</code></p> <h2>Tips and Tricks</h2> <p>Find a list of <a href="https://xcode-tips.github.io">useful tips here</a>.</p> <h2>Xcode build process</h2> <ul> <li> <p><a href="https://xcodebuildsettings.com">Xcode Build Settings</a></p> </li> <li> <p><a href="https://www.objc.io/issues/6-build-tools/build-process/">Xcode Build Process</a></p> </li> <li> <p><a href="https://www.avanderlee.com/optimization/analysing-build-performance-xcode/">Build performance analysis for speeding up Xcode builds</a></p> </li> <li> <p><a href="https://www.onswiftwings.com/posts/build-time-optimization-part1/">Xcode Build Time Optimization</a></p> </li> <li> <p><a href="https://github.com/pfandrade/Xconfig">Xconfig: Simple Mac app to display build settings for the currently open Xcode projects</a></p> </li> <li> <p><a href="https://asifmohd.github.io/ios/2021/03/11/xcbuild-debug-info.html">Digging deeper into xcbuild: Rules and Tasks</a></p> </li> <li> <p><a href="https://gist.github.com/ddunbar/2dda0e836c855ea96759d1d05f086d69">New Build System Tricks</a>, by Daniel Dunbar</p> </li> </ul> <h2>Automatically setting Xcode project build numbers</h2> <ul> <li> <p><a href="https://mokacoding.com/blog/automatic-xcode-versioning-with-git/">Automated Xcode version and build numbering via Git</a></p> </li> <li> <p><a href="https://useyourloaf.com/blog/setting-xcode-build-version-to-git-commit-name/">Setting Xcode build version to Git commit name</a></p> </li> <li> <p><a href="https://useyourloaf.com/blog/setting-iphone-application-build-versions/">Setting iPhone Application Build Versions</a></p> </li> </ul> <h2>Static vs Dynamic libraries and frameworks</h2> <ul> <li> <p><a href="https://pewpewthespells.com/blog/static_and_dynamic_libraries.html">Static and Dynamic Libraries, by pewpewthespells</a></p> </li> <li> <p><a href="https://www.bignerdranch.com/blog/it-looks-like-you-are-trying-to-use-a-framework/">It Looks Like You Are Trying to Use a Framework</a></p> </li> <li> <p><a href="https://www.runtastic.com/blog/en/frameworks-ios/">Basic Overview Of Static And Dynamic Frameworks On iOS</a></p> </li> <li> <p><a href="http://blog.cocoapods.org/CocoaPods-1.9.0-beta/">CocoaPods 1.9 Beta</a>, <code>use_frameworks! :linkage =&gt; :static</code></p> </li> </ul> <h2>Cross-platform frameworks in Xcode</h2> <ul> <li><a href="https://promisekit.org/news/2016/08/Multiplatform-Single-Scheme-Xcode-Projects/">Multiplatform, Single-scheme Xcode Projects</a></li> <li><a href="http://ilya.puchka.me/xcode-cross-platform-frameworks/">Xcode &amp; cross-platform frameworks</a></li> </ul> <h2>Documentation Packages ("DocC")</h2> <p>XCode Developer Documentation: <code>Developer Tools/DocC</code></p> <p>Introduced (?) in XCode 13. DocC doc compiler converst Markdown-based text into "rich documentation" for Swift projects, accessible within XCode.</p> <ul> <li>Add a doc catalog to an existing project: <ul> <li>select project/package in Project Navigator</li> <li>File &gt; New &gt; File &gt; Documentation Catalog template. Place the doc catalog in the same folder as the source files for DocC to associate the catalog with the library.</li> <li>"Root" markdown file is for the package-level documentation</li> </ul> </li> <li>Source files use <code>///</code> comments (with Markdown tags embedded within them) to provide context documentation</li> <li>Product &gt; Build Documentation generates the docs</li> <li>Docs appear in Window &gt; Developer Documentation &gt; Workspace Documentation (at the top)</li> </ul> <p><code>Developer Tools/DocC/Documentation Types/API Documentation</code> has more on the format of the Markdown</p> https://tedneward.github.io/Research/tools/uv/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/uv/index.html <p><a href="https://docs.astral.sh/uv/">Website</a> | <a href="https://github.com/astral-sh/uv">Source</a></p> <ul> <li>A single tool to replace pip, pip-tools, pipx, poetry, pyenv, twine, virtualenv, and more.</li> <li>10-100x faster than pip.</li> <li>Provides comprehensive project management, with a universal lockfile.</li> <li>Runs scripts, with support for inline dependency metadata.</li> <li>Installs and manages Python versions.</li> <li>Runs and installs tools published as Python packages.</li> <li>Includes a pip-compatible interface for a performance boost with a familiar CLI.</li> <li>Supports Cargo-style workspaces for scalable projects.</li> <li>Disk-space efficient, with a global cache for dependency deduplication.</li> <li>Installable without Rust or Python via curl or pip.</li> <li>Supports macOS, Linux, and Windows.</li> </ul> https://tedneward.github.io/Research/tools/verasco/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/verasco/index.html <ul> <li>static analyzer for the CompCert subset of ISO C 1999 that establishes the absence of run-time errors in analyzed programs</li> <li>entirely specified and proved sound using the Coq proof assistant</li> <li><a href="http://compcert.inria.fr/verasco/">http://compcert.inria.fr/verasco/</a></li> </ul> https://tedneward.github.io/Research/tools/visualstudio/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/visualstudio/index/index.html <ul> <li> <p>Visual C++ documentation</p> <ul> <li><a href="https://docs.microsoft.com/en-us/cpp/">https://docs.microsoft.com/en-us/cpp/</a></li> <li><a href="https://github.com/Microsoft/cpp-docs">https://github.com/Microsoft/cpp-docs</a></li> <li><a href="https://msdn.microsoft.com/en-us/library/60k1461a.aspx">https://msdn.microsoft.com/en-us/library/60k1461a.aspx</a></li> </ul> </li> <li> <p>/analyze (Code analysis)</p> <ul> <li><a href="https://docs.microsoft.com/en-us/cpp/build/reference/analyze-code-analysis">https://docs.microsoft.com/en-us/cpp/build/reference/analyze-code-analysis</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/vagrant/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/vagrant/index.html <p><a href="https://www.vagrantup.com/">Website</a> | <a href="https://www.github.com/hashicorp/vagrant">Source</a></p> https://tedneward.github.io/Research/tools/vim/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/vim/index.html <h3>Vim</h3> <ul> <li><a href="http://www.swaroopch.com/notes/vim/">A Byte of Vim</a></li> <li><a href="https://github.com/iggredible/Learn-Vim">Learn Vim (the Smart Way)</a> (HTML) (:construction: <em>in process</em>)</li> <li><a href="https://danielmiessler.com/study/vim/">Learn Vim For the Last Time</a> - Daniel Miessler</li> <li><a href="http://yannesposito.com/Scratch/en/blog/Learn-Vim-Progressively/">Learn Vim Progressively</a></li> <li><a href="http://learnvimscriptthehardway.stevelosh.com">Learn Vimscript the Hard Way</a> - Steve Losh</li> <li><a href="https://leanpub.com/VimLikeAPro">Use Vim Like A Pro</a> - Tim Ottinger <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://www.truth.sk/vim/vimbook-OPL.pdf">Vi Improved -- Vim</a> - Steve Oualline (PDF)</li> <li><a href="https://github.com/mhinz/vim-galore#readme">VIM-GALORE - All things Vim!</a> (HTML)</li> <li><a href="https://web.archive.org/web/20130302172911/http://vim.runpaint.org/vim-recipes.pdf">Vim Recipes</a> (PDF)</li> <li><a href="http://vimregex.com">Vim Regular Expressions 101</a></li> </ul> https://tedneward.github.io/Research/tools/umlet/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/umlet/index.html <p><a href="https://www.umlet.com/">Website</a></p> <p>UMLet is a UML tool aimed at providing a fast way of creating UML diagrams. UML elements are modified using text input instead of pop-up dialogs. Elements can be modified and used as templates; this way, users can easily tailor UMLet to their modeling needs. UMLet supports a variety of UML diagram types: class diagrams, use case diagrams, sequence diagrams, state diagrams, deployment diagrams, activity diagrams -- see some examples.</p> <p>UMLet also allows users to create their own custom UML elements. An element's look can be modified at run-time by changing a few lines of Java code; UMLet then compiles the new element's code on the fly. Without leaving UMLet, users can thus create and add new element types to their diagrams.</p> https://tedneward.github.io/Research/tools/valgrind/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/valgrind/index.html <p>There are Valgrind tools that can automatically detect many memory management and threading bugs, and profile your programs in detail. You can also use Valgrind to build new tools.</p> <p><a href="http://valgrind.org/">Website</a></p> <p><a href="http://valgrind.org/docs/pubs.html">Valgrind Research Papers</a></p> <ul> <li>Overload Journal; Paul Floyd <ul> <li>Part 1 – Introduction - <a href="https://accu.org/index.php/journals/1930">https://accu.org/index.php/journals/1930</a></li> <li>Part 2 – Basic memcheck - <a href="https://accu.org/index.php/journals/1913">https://accu.org/index.php/journals/1913</a></li> <li>Part 3 – Advanced memcheck - <a href="https://accu.org/index.php/journals/1905">https://accu.org/index.php/journals/1905</a></li> <li>Part 4 – Cachegrind and Callgrind - <a href="https://accu.org/index.php/journals/1886">https://accu.org/index.php/journals/1886</a></li> <li>Part 5 – Massif - <a href="https://accu.org/index.php/journals/1884">https://accu.org/index.php/journals/1884</a></li> <li>Part 6 – Helgrind and DRD - <a href="https://accu.org/index.php/journals/1867">https://accu.org/index.php/journals/1867</a></li> </ul> </li> </ul> <p><a href="https://www.youtube.com/watch?v=3l0BQs2ThTo">C++ Weekly - Episode 86</a></p> <p><a href="https://fau.re/blog/20140330_vgdb.html">gdb + valgrind</a></p> <p><a href="http://heeris.id.au/2016/valgrind-gdb/">Valgrind and GDB: Tame the Wild C</a></p> <p><a href="https://kristerw.blogspot.com/2020/02/watching-for-software-inefficiencies.html">Watching for software inefficiencies with Valgrind</a></p> <p><a href="https://github.com/kristerw/deadstores">deadstores: A Valgrind tool for finding redundant loads/stores</a></p> <p><a href="https://github.com/rimsa/CFGgrind">CFGGrind: Dynamic Control Flow Graph Reconstruction</a><br> - <a href="https://compilers.iecc.com/comparch/article/19-11-004">https://compilers.iecc.com/comparch/article/19-11-004</a></p> <p><a href="https://github.com/wmkhoo/taintgrind">Taintgrind</a>: A taint-tracking plugin for the Valgrind memory checking tool</p> https://tedneward.github.io/Research/tools/visidata/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/visidata/index.html <p><a href="https://www.visidata.org/">Website</a> | <a href="https://github.com/saulpw/visidata">Source</a></p> <p>VisiData is an interactive multitool for tabular data. It combines the clarity of a spreadsheet, the efficiency of the terminal, and the power of Python, into a lightweight utility which can handle millions of rows with ease.</p> <p>Dang, this thing is Excel or Access in a terminal with a ton of different data sources.</p> https://tedneward.github.io/Research/tools/unison/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/unison/index.html <p><a href="https://github.com/bcpierce00/unison">Source</a></p> <p>It allows two replicas of a collection of files and directories to be stored on different hosts (or different disks on the same host), modified separately, and then brought up to date by propagating the changes in each replica to the other.</p> <p>Unison has been in use for over 20 years and many people use it to synchronize data they care about.</p> <p>Features:</p> <ul> <li> <p>Unison works across platforms, allowing you to synchronize a Windows laptop with a Unix server, for example.</p> </li> <li> <p>Unlike simple mirroring or backup utilities, Unison can deal with updates to both replicas of a distributed directory structure. Updates that do not conflict can be propagated automatically. Conflicting updates are detected and displayed.</p> </li> <li> <p>Unlike many network filesystems, Unison copies data so that already-synchronized data can be read and written while offline.</p> </li> <li> <p>Unlike most distributed filesystems, Unison is a user-level program that simply uses normal systems calls: there is no need to modify the kernel, to have superuser privileges on either host, or to have a FUSE implementation.</p> </li> <li> <p>Unison works between any pair of machines connected to the internet, typically communicating over ssh, but also directly over TCP. It is careful with network bandwidth, and runs well over slow links. Transfers of small updates to large files are optimized using a compression protocol similar to rsync.</p> </li> <li> <p>Unison is resilient to failure. It is careful to leave the replicas and its own private structures in a sensible state at all times, even in case of abnormal termination or communication failures.</p> </li> <li> <p>Unison can run in "repeat" mode with a filesystem monitor, so that changes are synchronized soon after they happen.</p> </li> <li> <p>Unison has a clear and precise specification.</p> </li> </ul> https://tedneward.github.io/Research/tools/ventoy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ventoy/index.html <p><a href="https://www.ventoy.net/en/index.html">Website</a></p> <h2>Reading</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://www.xda-developers.com/how-to-turn-an-old-usb-drive-into-the-ultimate-recovery-tool-for-almost-any-pc/">Turn your old USB drive into a powerful recovery tool</a></li> <li><a href="https://www.xda-developers.com/operating-systems-to-keep-on-a-ventoy-usb-drive/">I keep these operating systems on a Ventoy USB drive</a> (Windows, Ubuntu, MX Linux, ZorinOS, Clonezilla, Memtest86+, GParted)</li> </ul> https://tedneward.github.io/Research/tools/visualpython/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/visualpython/index.html <p><a href="https://visualpython.ai/">Website</a> | <a href="https://github.com/visualpython/visualpython">Source</a></p> https://tedneward.github.io/Research/tools/spoon/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spoon/index.html <p><a href="https://spoon.gforge.inria.fr/">Website</a> | <a href="https://github.com/INRIA/spoon">Source</a></p> <p>It parses source files to build a well-designed AST with powerful analysis and transformation API. It supports modern Java versions up to Java 20. Spoon is an official Inria open-source project, and member of the <a href="https://www.ow2.org/">OW2</a> open-source consortium.</p> <h2>Documentation</h2> <p>The latest official documentation is available at <a href="http://spoon.gforge.inria.fr/">http://spoon.gforge.inria.fr/</a>.</p> <h2>Getting started in 2 seconds</h2> <blockquote> <p><strong>Java version:</strong> Spoon version 10 and up requires Java 11 or later. Spoon 9.1.0 is the final Spoon release compatible with Java 8, and we do not plan to backport any bug fixes or features to Spoon 9. Note that Spoon can of course still consume source code for older versions of Java, but it needs JDK 11+ to run.</p> </blockquote> <p>Get latest stable version with Maven, see <a href="https://search.maven.org/artifact/fr.inria.gforge.spoon/spoon-core">https://search.maven.org/artifact/fr.inria.gforge.spoon/spoon-core</a></p> <p>And start using it:</p> <pre><code>CtClass l = Launcher.parseClass("class A { void m() { System.out.println(\"yeah\");} }"); </code></pre> <p>Documentation:</p> <ul> <li>Reference documentation: <a href="http://spoon.gforge.inria.fr/">http://spoon.gforge.inria.fr/</a> (contains the content of the <a href="https://github.com/INRIA/spoon/tree/master/doc">doc folder</a>)</li> <li>Code examples: <a href="https://github.com/SpoonLabs/spoon-examples">https://github.com/SpoonLabs/spoon-examples</a></li> <li>Videos: <a href="https://www.youtube.com/watch?v=ZZzdVTIu-OY">Spoon: Getting Started - Simon Urli @ OW2Con’18 (Paris)</a>, <a href="https://www.youtube.com/watch?v=JcCIbjnkfD4">Generate Test Assertion with Spoon - Benjamin Danglot @ OW2Con’17 (Paris)</a></li> </ul> <h2>Contributing in 2 seconds</h2> <p>Create your first pull request to improve the documentation, see <a href="https://github.com/INRIA/spoon/tree/master/doc">doc</a>! Proceed with your first bug fix! The community is open-minded, respectful and patient. All external contributions are welcome.</p> <h2>Design Philosophy</h2> <p>R1) The Spoon metamodel is as close as possible to the language concepts.</p> <p>R2) The Spoon model of a program is complete and sound.</p> <p>R3) The text version of a Spoon model is well-formed and semantically equivalent to the original program.</p> <p>R4) The analysis and transformation API is intuitive and regular.</p> <p>R5) Transformation operators are designed to warn as fast as possible about invalid programs. This is done either with static type checking or with dynamic checks when the operators are used.</p> <p>R6) When feasible, the text version of a Spoon model is close to the original one.</p> <h3>Compiling</h3> <p>To compile Spoon, you need a Java Development Kit (JDK) and Maven:</p> <pre><code>git clone https://github.com/INRIA/spoon cd spoon mvn compile </code></pre> <p>To run the tests:</p> <pre><code>mvn test </code></pre> <h3>Download</h3> <p>Latest version: <a href="https://search.maven.org/remote_content?g=fr.inria.gforge.spoon&amp;a=spoon-core&amp;v=LATEST&amp;c=jar-with-dependencies">https://search.maven.org/remote_content?g=fr.inria.gforge.spoon&amp;a=spoon-core&amp;v=LATEST&amp;c=jar-with-dependencies</a> - <a href="http://spoon.gforge.inria.fr/mvnsites/spoon-core/apidocs/index.html">Javadoc</a></p> <p>Maven:</p> <pre><code>&lt;dependency&gt; &lt;groupId&gt;fr.inria.gforge.spoon&lt;/groupId&gt; &lt;artifactId&gt;spoon-core&lt;/artifactId&gt; &lt;!-- See rendered release value at http://spoon.gforge.inria.fr/ --&gt; &lt;version&gt;&lt;/version&gt; &lt;/dependency&gt; </code></pre> https://tedneward.github.io/Research/tools/statiq/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/statiq/index.html <p><a href="https://statiq.dev/">Website</a> | <a href="https://github.com/statiqdev">Source</a></p> https://tedneward.github.io/Research/tools/surfsense/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/surfsense/index.html <p><a href="https://www.surfsense.com/">Website</a> | <a href="https://github.com/MODSetter/SurfSense">Source</a> | <a href="https://www.surfsense.com/docs">Docs</a></p> <p>Connect any LLM to your internal knowledge sources and chat with it in real time alongside your team. OSS alternative to NotebookLM, Perplexity, and Glean.</p> <p>SurfSense is a highly customizable AI research agent, connected to external sources such as Search Engines (SearxNG, Tavily, LinkUp), Google Drive, Slack, Microsoft Teams, Linear, Jira, ClickUp, Confluence, BookStack, Gmail, Notion, YouTube, GitHub, Discord, Airtable, Google Calendar, Luma, Circleback, Elasticsearch, Obsidian and more to come.</p> <h2>Podcast Sample</h2> <p><a href="https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7">https://github.com/user-attachments/assets/a0a16566-6967-4374-ac51-9b3e07fbecd7</a></p> <h2>How to Use SurfSense</h2> <h3>Self Hosted</h3> <p>Run SurfSense on your own infrastructure for full data control and privacy.</p> <p><strong>Prerequisites:</strong> <a href="https://www.docker.com/products/docker-desktop/">Docker Desktop</a> must be installed and running.</p> <h4>For Linux/MacOS users:</h4> <pre><code class="language-bash">curl -fsSL https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.sh | bash </code></pre> <h4>For Windows users:</h4> <pre><code class="language-bash">irm https://raw.githubusercontent.com/MODSetter/SurfSense/main/docker/scripts/install.ps1 | iex </code></pre> <p>The install script sets up <a href="https://github.com/nicholas-fedor/watchtower">Watchtower</a> automatically for daily auto-updates. To skip it, add the <code>--no-watchtower</code> flag.</p> <p>For Docker Compose, manual installation, and other deployment options, see the <a href="https://www.surfsense.com/docs/">docs</a>.</p> <h3>How to Realtime Collaborate (Beta)</h3> <ol> <li>Go to Manage Members page and create an invite.</li> </ol> <p align="center"><img src="https://github.com/user-attachments/assets/40ed7683-5aa6-48a0-a3df-00575528c392" alt="Invite Members"></p> <ol> <li>Teammate joins and that SearchSpace becomes shared.</li> </ol> <p align="center"><img src="https://github.com/user-attachments/assets/ea4e1057-4d2b-4fd2-9ca0-cd19286a285e" alt="Invite Join Flow"></p> <ol> <li>Make chat shared.</li> </ol> <p align="center"><img src="https://github.com/user-attachments/assets/17b93904-0888-4c3a-ac12-51a24a8ea26a" alt="Make Chat Shared"></p> <ol> <li>Your team can now chat in realtime.</li> </ol> <p align="center"><img src="https://github.com/user-attachments/assets/83803ac2-fbce-4d93-aae3-85eb85a3053a" alt="Realtime Chat"></p> <ol> <li>Add comment to tag teammates.</li> </ol> <p align="center"><img src="https://github.com/user-attachments/assets/3b04477d-8f42-4baa-be95-867c1eaeba87" alt="Realtime Comments"></p> <h2>Key Features</h2> <table> <thead> <tr> <th> Feature </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> OSS Alternative </td> <td> Drop in replacement for NotebookLM, Perplexity, and Glean with real time team collaboration </td> </tr> <tr> <td> 50+ File Formats </td> <td> Upload documents, images, videos via LlamaCloud, Unstructured, or Docling (local) </td> </tr> <tr> <td> Hybrid Search </td> <td> Semantic + Full Text Search with Hierarchical Indices and Reciprocal Rank Fusion </td> </tr> <tr> <td> Cited Answers </td> <td> Chat with your knowledge base and get Perplexity style cited responses </td> </tr> <tr> <td> Deep Agent Architecture </td> <td> Powered by <a href="https://docs.langchain.com/oss/python/deepagents/overview">LangChain Deep Agents</a> planning, subagents, and file system access </td> </tr> <tr> <td> Universal LLM Support </td> <td> 100+ LLMs, 6000+ embedding models, all major rerankers via OpenAI spec &amp; LiteLLM </td> </tr> <tr> <td> Privacy First </td> <td> Full local LLM support (vLLM, Ollama) your data stays yours </td> </tr> <tr> <td> Team Collaboration </td> <td> RBAC with Owner / Admin / Editor / Viewer roles, real time chat &amp; comment threads </td> </tr> <tr> <td> Podcast Generation </td> <td> 3 min podcast in under 20 seconds; multiple TTS providers (OpenAI, Azure, Kokoro) </td> </tr> <tr> <td> Browser Extension </td> <td> Cross browser extension to save any webpage, including auth protected pages </td> </tr> <tr> <td> 25+ Connectors </td> <td> Search Engines, Google Drive, Slack, Teams, Jira, Notion, GitHub, Discord &amp; <a href="#external-sources">more</a> </td> </tr> <tr> <td> Self Hostable </td> <td> Open source, Docker one liner or full Docker Compose for production </td> </tr> </tbody> </table> <details> <summary><b>Full list of External Sources</b></summary> <a id="external-sources"></a> <p>Search Engines (Tavily, LinkUp) · SearxNG · Google Drive · Slack · Microsoft Teams · Linear · Jira · ClickUp · Confluence · BookStack · Notion · Gmail · YouTube Videos · GitHub · Discord · Airtable · Google Calendar · Luma · Circleback · Elasticsearch · Obsidian, and more to come.</p> </details> https://tedneward.github.io/Research/tools/syncthing/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/syncthing/index.html <p><a href="http://syncthing.net/">Website</a> | <a href="https://github.com/syncthing/syncthing">Source</a> | <a href="https://docs.syncthing.net/intro/getting-started.html">Getting Started</a></p> <p>Syncthing is a continuous file synchronization program. It synchronizes files between two or more computers. We strive to fulfill the goals below. The goals are listed in order of importance, the most important ones first.</p> <p>Syncthing should be:</p> <ol> <li>Safe From Data Loss: Protecting the user's data is paramount. We take every reasonable precaution to avoid corrupting the user's files.</li> <li>Secure Against Attackers: Again, protecting the user's data is paramount. Regardless of our other goals, we must never allow the user's data to be susceptible to eavesdropping or modification by unauthorized parties.</li> <li>Easy to Use: Syncthing should be approachable, understandable, and inclusive.</li> <li>Automatic: User interaction should be required only when absolutely necessary.</li> <li>Universally Available: Syncthing should run on every common computer. We are mindful that the latest technology is not always available to every individual.</li> <li>For Individuals: Syncthing is primarily about empowering the individual user with safe, secure, and easy to use file synchronization.</li> <li>Everything Else: There are many things we care about that don't make it on to the list. It is fine to optimize for these values, as long as they are not in conflict with the stated goals above.</li> </ol> https://tedneward.github.io/Research/tools/terraform/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/terraform/index.html <p><a href="https://www.terraform.io/">Website</a> | <a href="https://github.com/hashicorp/terraform">Source</a> | <a href="https://developer.hashicorp.com/terraform/tutorials?product_intent=terraform">Tutorials</a></p> <p>Officially joins IBM as of 28 Feb 2025.</p> <p><a href="https://woodruff.dev/?s=terraform&amp;e_search_props=e533c5b-883">Woody's blogs</a>:</p> <ul> <li>101: <a href="https://woodruff.dev/terraform-101-your-first-steps-into-infrastructure-as-code/">https://woodruff.dev/terraform-101-your-first-steps-into-infrastructure-as-code/</a></li> <li>Crash course in HCL: <a href="https://woodruff.dev/speaking-terraform-a-crash-course-in-hcl/">https://woodruff.dev/speaking-terraform-a-crash-course-in-hcl/</a></li> </ul> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://dev.to/arbythecoder/my-first-week-with-terraform-the-struggles-nobody-talks-about-and-how-i-overcame-them-4m32">My First Week with Terraform: The Struggles Nobody Talks About (And How I Overcame Them)</a></li> </ul> <h3>Books</h3> <ul> <li><a href="https://learning.oreilly.com/library/view/terraform-up-and/9781098116736/">Terraform: Up &amp; Running</a></li> </ul> <h2>Tools</h2> <ul> <li><a href="https://github.com/GoogleCloudPlatform/terraformer">terraformer</a>: CLI tool to generate terraform files from existing infrastructure (reverse Terraform); Infrastructure to Code <ul> <li>Article: <a href="https://blog.cloudtechner.com/terraformer-write-back-your-infrastructure-to-code-9b86ca930457">"Terraformer: Write back your infrastructure to code"</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/tldx/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tldx/index.html <p><a href="https://brandonyoung.dev/blog/introducing-tldx/">Blog post</a> | <a href="https://github.com/brandonyoungdev/tldx">Source</a></p> <p>You give it some keywords, optionally add prefixes, suffixes, or TLDs. Then it checks what’s available.</p> <pre><code>$ tldx google ❌ google.com is not available $ tldx openai -p get,use -s ly,hub -t com,io,ai --only-available ✔️ getopenaily.com is available ✔️ useopenaihub.io is available ... </code></pre> <p><strong>Features</strong></p> <ul> <li>Keyword-based domain permutations (prefixes, suffixes, TLDs)</li> <li>Fast and concurrent RDAP availability checks</li> <li>Streams results as they’re found</li> <li>Optional filtering by domain length</li> </ul> <h2>Quick Start</h2> <p><code>brew install brandonyoungdev/tldx/tldx</code></p> <p>or</p> <p><code>brew tap brandonyoungdev/tldx; brew install tldx</code></p> https://tedneward.github.io/Research/tools/treesitter/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/treesitter/index.html <p><a href="https://tree-sitter.github.io/tree-sitter/">Website</a> | <a href="https://github.com/tree-sitter/tree-sitter">Source</a> | <a href="https://tree-sitter.github.io/tree-sitter/playground">Playground</a></p> <p>Tree-sitter is a parser generator tool and an incremental parsing library. It can build a concrete syntax tree for a source file and efficiently update the syntax tree as the source file is edited. Tree-sitter aims to be:</p> <ul> <li>General enough to parse any programming language</li> <li>Fast enough to parse on every keystroke in a text editor</li> <li>Robust enough to provide useful results even in the presence of syntax errors</li> <li>Dependency-free so that the runtime library (which is written in pure C) can be embedded in any application</li> </ul> <p>Language Bindings: C#, Go, Guile, Haskell, Java, Java (Android), JavaScript (Node.js), JavaScript (Wasm), Kotlin, Lua, OCaml, Odin, Perl, Python, Ruby, Rust, Swift</p> https://tedneward.github.io/Research/tools/tuple/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tuple/index.html <p><a href="https://tuple.app/">Website</a></p> https://tedneward.github.io/Research/tools/spotbugs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spotbugs/index.html <p><a href="https://spotbugs.github.io/">Website</a> | <a href="https://github.com/spotbugs/spotbugs">Source</a></p> <p><a href="https://spotbugs.github.io/">SpotBugs</a> is the spiritual successor of <a href="https://github.com/findbugsproject/findbugs">FindBugs</a>, carrying on from the point where it left off with support of its community.</p> <p>Bug descriptions found by SpotBugs can be found <a href="https://spotbugs.readthedocs.io/en/latest/bugDescriptions.html">here</a>.</p> https://tedneward.github.io/Research/tools/structurizr/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/structurizr/index.html <p><a href="https://structurizr.com/help/code">Website</a> | <a href="https://github.com/structurizr">Source</a></p> <p><a href="https://structurizr.com/help/lite">Structurizr Lite</a>:</p> <p>From a <a href="https://dev.to/simonbrown/getting-started-with-structurizr-lite-27d0">Dev.to post</a>:</p> <ol> <li> <p><strong>Create a new directory:</strong> First we need to create a directory somewhere to store our workspace.</p> </li> <li> <p><strong>Create a workspace.dsl file:</strong> Next we need to create a file called workspace.dsl inside that directory, with the following content.</p> </li> </ol> <pre><code>workspace { model { user = person "User" softwareSystem = softwareSystem "Software System" user -&gt; softwareSystem "Uses" } views { systemContext softwareSystem "Diagram1" { include * autoLayout } theme default } } </code></pre> <pre><code> and  You can find more about the Structurizr DSL at https://github.com/structurizr/dsl, but essentially this DSL file says: * Create a model with a user and a software system, where the user uses the software system. * Create a system context view for the software system, adding the default set of elements, using auto-layout. * Use the default theme for styling elements and relationships. </code></pre> <ol> <li><strong>Start Structurizr Lite:</strong> Assuming that you have Docker installed, you can now start Structurizr Lite with the following commands:</li> </ol> <pre><code>docker pull structurizr/lite docker run -it --rm -p 8080:8080 -v PATH:/usr/local/structurizr structurizr/lite </code></pre> <pre><code>Be sure to replace PATH with the full path to the directory created in step 1. For example, if the directory is located at /Users/alice/structurizr, the commands would be: </code></pre> <pre><code>docker pull structurizr/lite docker run -it --rm -p 8080:8080 -v /Users/alice/structurizr:/usr/local/structurizr structurizr/lite </code></pre> <ol> <li> <p><strong>View the diagram:</strong> Open the workspace in a web browser by heading to <a href="http://localhost:8080">http://localhost:8080</a> and you should see your diagram. If you make a change to the DSL file and save it, you should see that change immediately reflected when you refresh your web browser.</p> </li> <li> <p><strong>Add some documentation:</strong> Let's add some documentation to describe our software system. To do this, create a subdirectory named docs and create a file named 01-context.md inside that subdirectory, with the following content:</p> </li> </ol> <pre><code>## Context Here is a description of my software system...  </code></pre> <pre><code>Now change the content of the workspace.dsl file to be as follows (notice the addition of the !docs line): </code></pre> <pre><code>workspace { model { user = person "User" softwareSystem = softwareSystem "Software System" { !docs docs } user -&gt; softwareSystem "Uses" } views { systemContext softwareSystem "Diagram1" { include * autoLayout } theme default } } </code></pre> <pre><code> Save all files, refresh your web browser, and click on the top-left button with the icon that looks like a book. You should now see the Markdown documentation page that we just created, which itself is embedding our existing system context diagram. </code></pre> https://tedneward.github.io/Research/tools/swagger_to-uml/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/swagger_to-uml/index.html <p><a href="https://github.com/nlohmann/swagger_to_uml">Source</a> <em>(Note: last update circa 2020)</em></p> https://tedneward.github.io/Research/tools/systemrescue/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/systemrescue/index.html <p><a href="https://www.system-rescue.org/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <p><a href="https://www.makeuseof.com/tools-always-keep-on-windows-rescue-usb-saved-countless-pcs/">https://www.makeuseof.com/tools-always-keep-on-windows-rescue-usb-saved-countless-pcs/</a></p> https://tedneward.github.io/Research/tools/timekeeper/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/timekeeper/index.html <p><a href="https://keeper.sh/">Website</a> | <a href="https://github.com/ridafkih/keeper.sh">Source</a></p> https://tedneward.github.io/Research/tools/toad/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/toad/index.html <p><a href="https://www.batrachian.ai/">Website</a> | <a href="https://github.com/batrachianai/toad">Source</a></p> <h2>Resources</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://willmcgugan.github.io/toad-released/">Toad is a unified experience for AI in the terminal</a></li> </ul> https://tedneward.github.io/Research/tools/tundra/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tundra/index.html <p><a href="https://github.com/deplinenoise/tundra">Github</a></p> https://tedneward.github.io/Research/tools/turbosharp/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/turbosharp/index.html <p><a href="https://github.com/Open-Inventions/TurboSharp">Source</a></p> <p>Inspired by Borland's Turbo Pascal/Turbo C IDEs.</p> https://tedneward.github.io/Research/tools/sqlc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sqlc/index.html <p><a href="https://sqlc.dev/?utm_source=tldrwebdev">Website</a> | <a href="https://github.com/sqlc-dev/sqlc">Source</a></p> https://tedneward.github.io/Research/tools/subversion/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/subversion/index.html <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://bobbrowning.me.uk/2025/12/23/why-svn-still-makes-sense-for-most-development-teams/">Why SVN Still Makes Sense for Most Development Teams</a></li> </ul> https://tedneward.github.io/Research/tools/symmetricds/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/symmetricds/index.html <p><a href="https://symmetricds.org">Website</a></p> https://tedneward.github.io/Research/tools/tap/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tap/index.html <p><a href="https://www.tatatap.com/">Website</a></p> https://tedneward.github.io/Research/tools/tldraw/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tldraw/index.html <p><a href="https://tldraw.dev/">Website</a> | <a href="https://github.com/tldraw/tldraw">Source</a></p> https://tedneward.github.io/Research/tools/tracer/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tracer/index.html <p><a href="https://github.com/SideChannelMarvels/Tracer">Source</a></p> <p><a href="https://insinuator.net/2016/03/how-to-crack-a-white-box-without-much-effort/">https://insinuator.net/2016/03/how-to-crack-a-white-box-without-much-effort/</a><br> - TracerPIN: TracerPin is a Intel PIN tool for generating execution traces of a running process.<br> - TracerGrind: TracerGrind is a Valgrind tool for generating execution traces of a running process.<br> - TraceGraph: TraceGraph is a GUI for visualizing execution traces produced by TracerGrind and TracerPin.</p> https://tedneward.github.io/Research/tools/tunneling/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tunneling/index.html <h1>Recommendations</h1> <p><em>(according to <a href="https://github.com/anderspitman/awesome-tunneling">https://github.com/anderspitman/awesome-tunneling</a>)</em></p> <ul> <li>For most people, I currently recommend <a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/">CloudFlare Tunnel</a>. Although it's closed source, this is the production-quality service that gets the closest to achieving the dream. It's also a loss-leader for CloudFlare's other products which means they can offer it for free.</li> <li>If you want to self-host, there are many options. For something production ready <a href="https://github.com/fatedier/frp">frp</a> is probably what you want. If you're a developer, I'd recommend starting with my own <a href="https://github.com/anderspitman/SirTunnel">SirTunnel</a> project and modifying it for your needs. For non-developers and those wanting more of a GUI experience, I created <a href="https://boringproxy.io/">boringproxy</a>. It's my take on a comprehensive tunnel proxy solution. It's in beta but currently solves almost everything I want. Once the server is running this is a very easy tool to use and has some nice features.</li> </ul> <h1>Open source (at least with a reasonably permissive license)</h1> <ul> <li><a href="https://github.com/robbie-cahill/tunnelmole-client/">Tunnelmole</a> - Open source and optionally self hostable. The client and server are both written in TypeScript.</li> <li><a href="https://telebit.cloud/">Telebit</a> - Written in JS. <a href="https://git.coolaj86.com/coolaj86/telebit.js">Code</a>.</li> <li><a href="https://tunnel.pyjam.as/">tunnel.pyjam.as</a> - No custom client; uses WireGuard directly instead. Written in Python. <a href="https://gitlab.com/pyjam.as/tunnel">source code</a></li> <li><a href="https://bitbucket.org/ValdikSS/dropbear-sshj/">SSH-J.com</a> - Public SSH Jump &amp; Port Forwarding server. No software, no registration, just an anonymous SSH server for forwarding. Users are encouraged to use it for SSH exposure only, to preserve end-to-end encryption. No public ports, only in-SSH connectivity. Run <code>ssh ssh-j.com</code> and it will display usage information.</li> <li><a href="https://github.com/fatedier/frp">frp</a> <a href="https://github.com/fatedier/frp/stargazers"><img src="https://img.shields.io/github/stars/fatedier/frp?style=flat" alt="frp github stars badge"></a> - Comprehensive open alternative to ngrok. Supports UDP, and has a P2P mode. Supports multiplexing over TCP (single connection or pool), QUIC, and KCP.</li> <li><a href="https://github.com/inconshreveable/ngrok">ngrok 1.0</a> <a href="https://github.com/inconshreveable/ngrok/stargazers"><img src="https://img.shields.io/github/stars/inconshreveable/ngrok?style=flat" alt="ngrok 1.0 github stars badge"></a> - Original version of ngrok. No longer developed in favor of the commercial 2.0 version.</li> <li><a href="https://github.com/localtunnel">localtunnel</a> <a href="https://github.com/localtunnel/localtunnel/stargazers"><img src="https://img.shields.io/github/stars/localtunnel/localtunnel?style=flat" alt="localtunnel github stars badge"></a> - Written in node. Popular suggestion.</li> <li><a href="https://github.com/sshuttle/sshuttle">sshuttle</a> <a href="https://github.com/sshuttle/sshuttle/stargazers"><img src="https://img.shields.io/github/stars/sshuttle/sshuttle?style=flat" alt="sshuttle github stars badge"></a> - Open source project originally from one of the founders of Tailscale. Server doesn't require root; client does. Explicitly designed to avoid TCP-over-TCP issues.</li> <li><a href="https://github.com/jpillora/chisel">chisel</a> <a href="https://github.com/jpillora/chisel/stargazers"><img src="https://img.shields.io/github/stars/jpillora/chisel?style=flat" alt="chisel github stars badge"></a> - SSH under the hood, but still uses a custom client binary. Supports auto certs from LetsEncrypt. Written in Go.</li> <li><a href="https://github.com/ekzhang/bore">bore</a> <a href="https://github.com/ekzhang/bore/stargazers"><img src="https://img.shields.io/github/stars/ekzhang/bore?style=flat" alt="bore github stars badge"></a> - Minimal tunneling solution. MIT Licensed. Written in Rust.</li> <li><a href="https://github.com/rapiz1/rathole">rathole</a> <a href="https://github.com/rapiz1/rathole/stargazers"><img src="https://img.shields.io/github/stars/rapiz1/rathole?style=flat" alt="rathole github stars badge"></a> - Similar to frp, including the config format, but with improved performance. Low resource consumption. Hot reload. Written in Rust.</li> <li><a href="https://github.com/beyondcode/expose">expose</a> <a href="https://github.com/beyondcode/expose/stargazers"><img src="https://img.shields.io/github/stars/beyondcode/expose?style=flat" alt="expose github stars badge"></a> - ngrok alternative written in PHP.</li> <li><a href="https://github.com/antoniomika/sish">sish</a> <a href="https://github.com/antoniomika/sish/stargazers"><img src="https://img.shields.io/github/stars/antoniomika/sish?style=flat" alt="sish github stars badge"></a> - Open source ngrok/serveo alternative. SSH-based but uses a custom server written in Go. Supports WebSocket tunneling.</li> <li><a href="https://github.com/mmatczuk/go-http-tunnel">go-http-tunnel</a> <a href="https://github.com/mmatczuk/go-http-tunnel/stargazers"><img src="https://img.shields.io/github/stars/mmatczuk/go-http-tunnel?style=flat" alt="go-http-tunnel github stars badge"></a> - Uses a single HTTP/2 connection for muxing. Need to manually generate certs for server and clients.</li> <li><a href="https://github.com/pgrok/pgrok">pgrok/pgrok</a> <a href="https://github.com/pgrok/pgrok/stargazers"><img src="https://img.shields.io/github/stars/pgrok/pgrok?style=flat" alt="pgrok github stars badge"></a> - A multi-tenant HTTP reverse tunnel solution through SSH remote port forwarding.</li> <li><a href="https://tunnelto.dev/">tunnelto</a> <a href="https://github.com/agrinman/tunnelto/stargazers"><img src="https://img.shields.io/github/stars/agrinman/tunnelto?style=flat" alt="tunnelto github stars badge"></a> - Open source (MIT). Written in Rust.</li> <li><a href="https://github.com/erebe/wstunnel">wstunnel</a> <a href="https://github.com/erebe/wstunnel/stargazers"><img src="https://img.shields.io/github/stars/erebe/wstunnel?style=flat" alt="wstunnel github stars badge"></a> - Proxies over WebSockets. Focus on proxying from behind networks that block certain protocols. Written in Haskell with executables provided.</li> <li><a href="https://zrok.io/">zrok</a> <a href="https://github.com/openziti/zrok/stargazers"><img src="https://img.shields.io/github/stars/openziti/zrok?style=flat" alt="zrok github stars badge"></a> - Aims for effortless sharing both publicly and privately. Supports multiple types of resources, including HTTP endpoints and files. Built on OpenZiti (see overlay section below). Apache 2 License. Written in Go.</li> <li><a href="https://boringproxy.io/">boringproxy</a> <a href="https://github.com/boringproxy/boringproxy/stargazers"><img src="https://img.shields.io/github/stars/boringproxy/boringproxy?style=flat" alt="boringproxy github stars badge"></a> - Designed to be very easy to use. No config files. Clients can be remote-controlled through a simple WebUI and/or REST API on the server.</li> <li><a href="https://github.com/anderspitman/SirTunnel">SirTunnel</a> <a href="https://github.com/anderspitman/SirTunnel/stargazers"><img src="https://img.shields.io/github/stars/anderspitman/SirTunnel?style=flat" alt="SirTunnel github stars badge"></a> - Minimal, self-hosted, 0-config alternative to ngrok. Similar to sish but leverages Caddy+OpenSSH rather than custom server code.</li> <li><a href="https://github.com/azimjohn/jprq">jprq</a> <a href="https://github.com/azimjohn/jprq/stargazers"><img src="https://img.shields.io/github/stars/azimjohn/jprq?style=flat" alt="jprq github stars badge"></a> - Proxies over WebSockets. Written in Python.</li> <li><a href="https://pagekite.net/">PageKite</a> <a href="https://github.com/pagekite/PyPagekite/stargazers"><img src="https://img.shields.io/github/stars/pagekite/PyPagekite?style=flat" alt="pagekite github stars badge"></a> - Comprehensive open source solution with hosted options.</li> <li><a href="https://github.com/sandialabs/wiretap">Wiretap</a> <a href="https://github.com/sandialabs/wiretap/stargazers"><img src="https://img.shields.io/github/stars/sandialabs/wiretap?style=flat" alt="wiretap github stars badge"></a> - Transparent tunneling over WireGuard (UDP) using userspace network stack. Root not required on server. Supports multiple clients and servers. Written in Go.</li> <li><a href="https://github.com/q3k/crowbar">Crowbar</a> <a href="https://github.com/q3k/crowbar/stargazers"><img src="https://img.shields.io/github/stars/q3k/crowbar?style=flat" alt="crowbar github stars badge"></a> - Tunnels TCP connections over HTTP GET and POST requests.</li> <li><a href="https://github.com/skx/tunneller">tunneller</a> <a href="https://github.com/skx/tunneller/stargazers"><img src="https://img.shields.io/github/stars/skx/tunneller?style=flat" alt="tunneller github stars badge"></a> - Open source. Written in Go.</li> <li><a href="https://github.com/cmars/onionpipe">onionpipe</a> <a href="https://github.com/cmars/onionpipe/stargazers"><img src="https://img.shields.io/github/stars/cmars/onionpipe?style=flat" alt="onionpipe github stars badge"></a> - Onion addresses for anything. <code>onionpipe</code> forwards ports on the local host to remote Onion addresses as Tor hidden services and vice-versa. Written in Go.</li> <li><a href="https://github.com/koding/tunnel">tunnel</a> <a href="https://github.com/koding/tunnel/stargazers"><img src="https://img.shields.io/github/stars/koding/tunnel?style=flat" alt="tunnel github stars badge"></a> - This one is a Golang library, not a program you can just run. However, it looks easy to use for creating custom solutions. Uses a single TCP socket, and <a href="https://github.com/hashicorp/yamux">yamux</a> for multiplexing.</li> <li><a href="https://www.proxy.jetzt/">jerson/pgrok</a> <a href="https://github.com/jerson/pgrok/stargazers"><img src="https://img.shields.io/github/stars/jerson/pgrok?style=flat" alt="pgrok github stars badge"></a> - Fork of ngrok 1.0, with more recent commits. Archived.</li> <li><a href="https://github.com/vitobotta/docker-tunnel">docker-tunnel</a> <a href="https://github.com/vitobotta/docker-tunnel/stargazers"><img src="https://img.shields.io/github/stars/vitobotta/docker-tunnel?style=flat" alt="docker-tunnel github stars badge"></a> - Simple Docker-based nginx+SSH solution.</li> <li><a href="https://github.com/fasmide/remotemoe">remotemoe</a> <a href="https://github.com/fasmide/remotemoe/stargazers"><img src="https://img.shields.io/github/stars/fasmide/remotemoe?style=flat" alt="remotemoe github stars badge"></a> - SSH-based, with custom golang server. Does some cool unique things. Instead of just plain tunnels, it drops you into a basic CLI UI that offers several useful commands interactively, such as adding a custom hostname. Also allows end-to-end encryption for both HTTPS and upstream SSH. Doesn't appear to offer non-e2e HTTPS, ie no auto Let's Encrypt support.</li> <li><a href="https://github.com/berstend/hypertunnel">hypertunnel</a> <a href="https://github.com//berstend/hypertunnel/stargazers"><img src="https://img.shields.io/github/stars/berstend/hypertunnel?style=flat" alt="frp github stars badge"></a> - Public server appears to be down. MIT Licensed. Written in JavaScript.</li> <li><a href="https://github.com/ntnj/tunwg">tunwg</a> <a href="https://github.com/ntnj/tunwg/stargazers"><img src="https://img.shields.io/github/stars/ntnj/tunwg?style=flat" alt="tunwg github stars badge"></a> - Wireguard in userspace based. Offers end to end encrypted TLS with LetsEncrypt certificates generated automatically by clients, with support for custom domains. Server can be self-hosted and doesn't require storing any data.</li> <li><a href="https://github.com/ao-space/gt">gt</a><a href="https://github.com/ao-space/gt/stargazers"><img src="https://img.shields.io/github/stars/ao-space/gt?style=flat" alt="gt github stars badge"></a> - Supports peer-to-peer direct connection (P2P) and Internet relay. Focus on performance. Written in Go.</li> <li><a href="https://github.com/CypherpunkArmory/holepunch">holepunch</a> <a href="https://github.com/CypherpunkArmory/holepunch/stargazers"><img src="https://img.shields.io/github/stars/CypherpunkArmory/holepunch?style=flat" alt="holepunch github stars badge"></a> - Has nice hosted solution. Uses SSH for muxing.</li> <li><a href="https://tunnel.staqlab.com/">StaqLab Tunnel</a> <a href="https://github.com/abhishekq61/tunnel-client/stargazers"><img src="https://img.shields.io/github/stars/abhishekq61/tunnel-client?style=flat" alt="staqlab github stars badge"></a> - SSH-based. Client is open source. Server doesn't appear to be.</li> <li><a href="https://github.com/LiljebergXYZ/tnnlink">tnnlink</a> <a href="https://github.com/LiljebergXYZ/tnnlink/stargazers"><img src="https://img.shields.io/github/stars/LiljebergXYZ/tnnlink?style=flat" alt="tnnlink github stars badge"></a> - SSH-based. Golang. Not maintained.</li> <li><a href="https://github.com/DigitallyRefined/docker-wireguard-tunnel">docker-wireguard-tunnel</a> <a href="https://github.com/DigitallyRefined/docker-wireguard-tunnel/stargazers"><img src="https://img.shields.io/github/stars/DigitallyRefined/docker-wireguard-tunnel?style=flat" alt="ngtor github stars badge"></a> - Connect two or more Docker servers together sharing container ports between them via a WireGuard tunnel.</li> <li><a href="https://github.com/theborakompanioni/ngtor">ngtor</a> <a href="https://github.com/theborakompanioni/ngtor/stargazers"><img src="https://img.shields.io/github/stars/theborakompanioni/ngtor?style=flat" alt="ngtor github stars badge"></a> - Easily expose local services via Tor. Written in Java.</li> <li><a href="https://github.com/Degola/punchmole/">Punchmole</a> <a href="https://github.com/Degola/punchmole/stargazers"><img src="https://img.shields.io/github/stars/Degola/punchmole?style=flat" alt="punchmole github stars badge"></a> - Can be integrated directly into an existing Node.js project. Written in JavaScript.</li> </ul> <h1>Commercial/Closed source</h1> <ul> <li><a href="https://ngrok.com/">ngrok 2.0</a> - Probably the gold standard and most popular. Closed source. Lots of features, including TLS and TCP tunnels. Doesn't require root to run client.</li> <li><a href="https://developers.cloudflare.com/cloudflare-one/connections/connect-apps/install-and-setup">CloudFlare Tunnel</a> - Excellent free option. Nicely integrates tunneling with the rest of Cloudflare's products, which include DNS and auto HTTPS. Client <a href="https://github.com/cloudflare/cloudflared">source code</a> is Apache 2.0 licensed and written in Golang.</li> <li><a href="https://beeceptor.com/">Beeceptor</a> - Goes beyond tunneling. Rest API mocking and intercepting tool. You can view the live requests and send mocked response. Written in JavaScript.</li> <li><a href="https://pinggy.io/">Pinggy</a> - SSH based single command HTTPS / TCP / TLS tunnels, no downloads required. Rich terminal interface and a web debugger. Free tier - 60 min timeout. Paid tier allows custom domains with built-in Let's Encrypt certificates.</li> <li><a href="https://loophole.cloud/">Loophole</a> - Offers end-to-end TLS encryption with the client automatically getting certs from Let's Encrypt. QR codes for URL sharing. Client is open source. Can serve a local directory over WebDAV. MIT License. Written in Go.</li> <li><a href="https://localhost.run/">localhost.run</a> - Simple hosted SSH option. Supports custom domains for a cost.</li> <li><a href="https://packetriot.com">Packetriot</a> - Comprehensive alternative to ngrok. HTTP Inspector, Let's Encrypt integration, doesn't require root and Linux repos for apt, yum and dnf. Enterprise licenses and self-hosted option.</li> <li><a href="https://hoppy.network/">Hoppy</a> - WireGuard-based. Provides static IPv4 and IPv6 addresses for your machines, which is a simple and useful level of abstraction. Targeted towards self-hosters and people behind NATs.</li> <li><a href="https://gw.run/">gw.run</a> - Specifically focusing on securely exposing internal web apps to a group of people; not for publicly facing apps. Share access via email address then allow users to log in with common login providers like Google.</li> <li><a href="https://sshreach.me/">SSHReach.me</a> - Paid SSH-based option. Uses a simple python script.</li> <li><a href="https://kubesail.com/">KubeSail</a> - Company offering tunneling, dynamic DNS, and other services for self-hosting with Kubernetes.</li> <li><a href="https://inlets.dev/">inlets</a> - Used to be <a href="https://github.com/inlets/inlets-archived">open source</a>; now focused on a polished commercial offering. Designed to work well with Kubernetes.</li> <li><a href="https://localtonet.com/">LocalToNet</a> - Supports UDP. Free for a single tunnel. Paid supports custom domains.</li> <li><a href="https://localxpose.io">LocalXpose</a> - Looks like a solid paid option, with a limited free tier.</li> <li><a href="https://tabserve.dev">Tabserve.dev</a> - Web UI that runs entirely in the browser and uses a Cloudflare Worker for https.</li> <li><a href="https://serveo.net">Serveo</a> - SSH-based, signup optional, offering HTTP(S) and TCP tunneling and SSH jump host forwarding capabilities.</li> </ul> <h1>Overlay networks and other advanced tools</h1> <ul> <li><a href="https://github.com/juanfont/headscale">headscale</a> <a href="https://github.com/juanfont/headscale/stargazers"><img src="https://img.shields.io/github/stars/juanfont/headscale?style=flat" alt="headscale github stars badge"></a> - Open source implementation of Tailscale control server. Can be used with Tailscale's official open source client. Written in Go.</li> <li><a href="https://www.tailscale.com/">Tailscale</a> <a href="https://github.com/tailscale/tailscale/stargazers"><img src="https://img.shields.io/github/stars/tailscale/tailscale?style=flat" alt="tailscale github stars badge"></a> - Built on WireGuard. Easy to use. Control server is closed source. Client <a href="https://github.com/tailscale">code</a> available with a BSD3 license + separate patents file.</li> <li><a href="https://goteleport.com/">Teleport</a> <a href="https://github.com/gravitational/teleport"><img src="https://img.shields.io/github/stars/gravitational/teleport?style=flat" alt="teleport github stars badge"></a> - Comprehensive control plane tool, but also supports <a href="https://goteleport.com/docs/application-access/introduction/">accessing apps</a> behind NATs. Written in Go.</li> <li><a href="https://github.com/slackhq/nebula">Nebula</a> - <a href="https://github.com/zerotier/slackhq/nebula"><img src="https://img.shields.io/github/stars/slackhq/nebula?style=flat" alt="nebula github stars badge"></a> Peer-to-peer overlay network. Developed and used internally by Slack. Similar to Tailscale but completely open source. Doesn't use WireGuard. Written in Go.</li> <li><a href="https://www.zerotier.com/">ZeroTier</a> - <a href="https://github.com/zerotier/ZeroTierOne/stargazers"><img src="https://img.shields.io/github/stars/zerotier/ZeroTierOne?style=flat" alt="zerotier github stars badge"></a> Layer 2 overlay network. They take decentralization seriously, and like to say "decentralize until it hurts, then centralize until it works." Written in C++.</li> <li><a href="https://github.com/gravitl/netmaker">Netmaker</a> <a href="https://github.com/gravitl/netmaker/stargazers"><img src="https://img.shields.io/github/stars/gravitl/netmaker?style=flat" alt="netmaker github stars badge"></a> - Layer 3 peer-to-peer overlay network and private DNS. Similar to Tailscale, but with a self-hosted server/admin UI. Runs kernel WireGuard so very fast. Not FOSS, but source is available. Written in Go.</li> <li><a href="https://github.com/netbirdio/netbird">NetBird</a> <a href="https://github.com/netbirdio/netbird/stargazers"><img src="https://img.shields.io/github/stars/netbirdio/netbird?style=flat" alt="netbird github stars badge"></a> - NetBird is an open-source VPN management platform built on top of WireGuard® making it easy to create secure private networks for your organization or home.</li> <li><a href="https://www.firezone.dev/">Firezone</a> <a href="https://github.com/firezone/firezone"><img src="https://img.shields.io/github/stars/firezone/firezone?style=flat" alt="firezone github stars badge"></a> - Layer 3/4 overlay network. Runs on kernel WireGuard® and supports SSO using generic OIDC/SAML connectors. Distributed under apache 2.0 license and written in Elixir/Rust.</li> <li><a href="https://github.com/tonarino/innernet">innernet</a> <a href="https://github.com/tonarino/innernet/stargazers"><img src="https://img.shields.io/github/stars/tonarino/innernet?style=flat" alt="innernet github stars badge"></a> - Similar to Netmaker, nebula, and Tailscale. Takes advantage of existing networking concepts like CIDRs and the security properties of WireGuard to turn your computer's basic IP networking into more powerful ACL primitives. Written in Rust.</li> <li><a href="https://pritunl.com/">Pritunl</a> <a href="https://github.com/pritunl/pritunl/stargazers"><img src="https://img.shields.io/github/stars/pritunl/pritunl?style=flat" alt="pritunl github stars badge"></a> - Seems quite comprehensive and complicated. OpenVPN, WireGuard, and IPSec support.</li> <li><a href="https://github.com/gsliepen/tinc">Tinc</a> <a href="https://github.com/gsliepen/tinc/stargazers"><img src="https://img.shields.io/github/stars/gsliepen/tinc?style=flat" alt="tinc github stars badge"></a> - Tinc is a peer-to-peer VPN daemon that supports VPNs with an arbitrary number of nodes. Instead of configuring tunnels, you give tinc the location and public key of a few nodes in the VPN. After making the initial connections to those nodes, tinc will learn about all other nodes on the VPN, and will make connections automatically. When direct connections are not possible, data will be forwarded by intermediate nodes. Written in C.</li> <li><a href="https://openziti.github.io">OpenZiti</a> - <a href="https://github.com/openziti/ziti/stargazers"><img src="https://img.shields.io/github/stars/openziti/ziti?style=flat" alt="OpenZiti github stars badge"></a> - Overlay network. The goal of OpenZiti is to extend zero trust all the way into your application, not just to your network. Apache 2.0 license. Written in Go.</li> <li><a href="https://github.com/zufardhiyaulhaq/ngrok-operator">Ngrok-operator</a> <a href="https://github.com/zufardhiyaulhaq/ngrok-operator/stargazers"><img src="https://img.shields.io/github/stars/zufardhiyaulhaq/ngrok-operator?style=flat" alt="ngrok operator github stars badge"></a> - Ngrok but integrated with Kubernetes, allows developers on private kubernetes to easily access their services via Ngrok.</li> <li><a href="https://github.com/FyraLabs/chisel-operator/">chisel-operator</a> <a href="https://github.com/FyraLabs/chisel-operator/stargazers"><img src="https://img.shields.io/github/stars/FyraLabs/chisel-operator?style=flat" alt="chisel operator github stars badge"></a> - Kubernetes integration for Chisel. Similar functionality to inlets. MIT License. Written in Rust.</li> </ul> <h1>Reference</h1> <ul> <li><a href="https://jerrington.me/posts/2019-01-29-self-hosted-ngrok.html">Roll your own Ngrok with Nginx, Letsencrypt, and SSH reverse tunnelling</a></li> <li><a href="https://dev.to/k4ml/poor-man-ngrok-with-tcp-proxy-and-ssh-reverse-tunnel-1fm">Poor man's ngrok with tcp proxy and ssh reverse tunnel</a></li> <li><a href="https://dev.to/azimjohn/how-i-built-ngrok-alternative-3n0g">How I built Ngrok Alternative (jprq)</a></li> <li><a href="https://stackoverflow.com/a/52614266/943814">Great SO answer by AJ ONeal about how these things work</a></li> <li><a href="https://youtu.be/E1Q2MWGCADo">Talk by AJ ONeal about tunneling tech</a></li> <li><a href="https://morph027.gitlab.io/blog/localtunnel-ngrok/">ngrok alternative: localtunnel + Caddy + Lets Encrypt</a></li> </ul> https://tedneward.github.io/Research/tools/spoofax/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spoofax/index.html <p><a href="http://www.metaborg.org/en/latest">Website</a> | <a href="https://github.com/metaborg">Source</a></p> <p>The platform provides the following ingredients:</p> <ul> <li>Meta-languages for high-level declarative language definition</li> <li>An interactive environment for developing languages using these meta-languages</li> <li>Code generators that produces parsers, type checkers, compilers, interpreters, and other tools from language definitions</li> <li>Generation of full-featured Eclipse editor plugins from language definitions</li> <li>Generation of full-featured IntelliJ editor plugins from language definitions (experimental)</li> <li>An API for programmatically combining the components of a language implementation</li> </ul> <p>With Spoofax you can focus on the essence of language definition and ignore irrelevant implementation details.</p> <p>Spoofax is the integration of many different tools, compilers, (meta-)languages, (meta-)libraries, and runtime components. This integration is made concrete in the spoofax-releng Git repository on GitHub. This repository contains all components via Git submodules, which are updated by our build farm that builds Spoofax whenever one of its components in a submodule changes.</p> <p>Spoofax currently contains the following subcomponents as submodules:</p> <ul> <li>releng - release engineering scripts for managing and building the spoofax-releng repostory.</li> <li>Java libraries and runtimes <ul> <li>mb-rep - libraries for program representation such as abstract terms</li> <li>mb-exec - Stratego interpreter and utilities</li> <li>jsglr - JSGLR parser</li> <li>spoofax - Spoofax Core, a cross platform API to Spoofax languages</li> <li>spoofax-maven - Maven integration for Spoofax Core</li> <li>spoofax-sunshine - Command-line integration for Spoofax Core</li> <li>spoofax-eclipse - Eclipse plugin for Spoofax Core</li> <li>spoofax-intellij - IntelliJ plugin for Spoofax Core</li> </ul> </li> <li>Meta-languages and libraries <ul> <li>esv - Editor service language</li> <li>sdf - <a href="/formats/sdf">Syntax Definition Formalisms</a>, containing the SDF2 and SDF 3 languages</li> <li><a href="http://strategoxt.org/Stratego/StrategoLanguage.html">stratego and strategoxt</a> - Stratego compiler, runtime, and editor - "Stratego is a small and efficient domain-specific language for program transformation. It is based on the paradigm of programmable rewriting strategies, which makes it extremely well-suited for traversing and transforming all kinds of tree structures. Its library and supporting tools are geared towards processing computer programs in the form of concrete or abstract syntax trees, as produced by text parsers."</li> <li>nabl - Name binding languages, containing the NaBL and NaBL2 languages, and support libraries for NaBL2</li> <li>ts - Type system language</li> <li>dynsem - Dynamic semantics language</li> <li>metaborg-coq - Coq signatures and syntax definition</li> <li>spt - Spoofax testing language</li> <li>runtime-libraries - NaBL support libraries, incremental task engine for incremental name and type analysis</li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/sqlplusnet/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sqlplusnet/index.html <p><a href="https://sqlplusweb.azurewebsites.net/">Website</a> | Commercial, no source</p> <p>Write SQL, generate .NET</p> https://tedneward.github.io/Research/tools/superagent/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/superagent/index.html <p><a href="https://superagent.com/">Website</a> | Commercial?</p> https://tedneward.github.io/Research/tools/symtool/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/symtool/index.html <p><a href="https://github.com/calebzulawski/symtool">Github</a></p> <p>Changing symbol visibility. Renaming symbols. Actions are performed in-place, leaving the rest of the binary untouched.</p> https://tedneward.github.io/Research/tools/termsvg/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/termsvg/index.html <p><a href="https://github.com/MrMarble/termsvg">Source</a></p> <p>TermSVG is an all in one cli tool to record, replay and export your terminal session to svg. It uses the same format as asciinema so you can convert asciicast files to SVG or use the asciinema player with a TermSVG recording.</p> <h2>Installation</h2> <h3>Manually</h3> <p>You can download a pre compiled binary directly from the <a href="https://github.com/mrmarble/termsvg/releases">releases</a> for your OS/Architecture.</p> <h3>Go cli</h3> <pre><code class="language-sh">go install github.com/mrmarble/termsvg/cmd/termsvg@latest # or target a specific version @v0.6.0 </code></pre> <h3>Install script</h3> <pre><code class="language-sh">curl -sL https://raw.githubusercontent.com/MrMarble/termsvg/master/scripts/install-termsvg.sh | sudo -E bash - # or with wget wget -O - https://raw.githubusercontent.com/MrMarble/termsvg/master/scripts/install-termsvg.sh | sudo -E bash - </code></pre> <h2>Usage</h2> <p>termsvg is composed of multiple commands, similar to <code>git</code>, <code>docker</code> or <code>asciinema</code>.</p> <p>When you run <code>termsvg</code> with no arguments help message is displayed, listing all available commands with their options.</p> <h3><code>rec &lt;filename&gt;</code></h3> <p><strong>Record terminal session.</strong></p> <p>By running <code>termsvg rec &lt;filename&gt;</code> you start a new recording session. The command (process) that is recorded can be specified with <code>-c</code> option (see below), and defaults to <code>$SHELL</code> which is what you want in most cases.</p> <p>You can temporarily pause recording of terminal by pressing <kbd>Ctrl+P</kbd>. This is useful when you want to execute some commands during the recording session that should not be captured (e.g. pasting secrets). Resume by pressing <kbd>Ctrl+P</kbd> again.</p> <p>Recording finishes when you exit the shell (hit <kbd>Ctrl+D</kbd> or type <code>exit</code>). If the recorded process is not a shell then recording finishes when the process exits.</p> <p>The resulting recording (called <a href="doc/asciicast-v2.md">asciicast</a>) is saved to a local file. It can later be replayed with <code>termsvg play &lt;filename&gt;</code> and/or exported to svg with <code>termsvg export -i &lt;filename&gt;</code>.</p> <p>Available options:</p> <ul> <li><code>-c, --command=&lt;command&gt;</code> - Specify command to record, defaults to $SHELL</li> </ul> <h3><code>play &lt;filename&gt;</code></h3> <p><strong>Replay recorded asciicast in a terminal.</strong></p> <p>This command replays given asciicast (as recorded by <code>rec</code> command) directly in your terminal.</p> <p>Playing from a local file:</p> <pre><code class="language-sh">termsvg play /path/to/asciicast.cast </code></pre> <p>Available options:</p> <ul> <li><code>-i, --idle-time-limit=&lt;sec&gt;</code> - Limit replayed terminal inactivity to max <code>&lt;sec&gt;</code> seconds</li> <li><code>-s, --speed=&lt;factor&gt;</code> - Playback speed (can be fractional)</li> </ul> <blockquote> <p>For the best playback experience it is recommended to run <code>termsvg play</code> in<br> a terminal of dimensions not smaller than the one used for recording, as<br> there's no "transcoding" of control sequences for new terminal size.</p> </blockquote> <h3><code>export &lt;filename&gt;</code></h3> <p><strong>Export recorded asciicast to svg.</strong></p> <p>This command exports given asciicast (as recorded by <code>rec</code> command) to svg.</p> <p>Exporting from a local file:</p> <pre><code class="language-sh">termsvg export /path/to/asciicast.cast </code></pre> <p>Available options:</p> <ul> <li><code>-o, --output=&lt;file&gt;</code> - Output svg to be created. Defaults to [input].svg</li> <li><code>-m, --minify</code> - Minify svg using <a href="https://github.com/tdewolff/minify">Minify</a></li> </ul> https://tedneward.github.io/Research/tools/tldw/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tldw/index.html <p><a href="https://tldwproject.com/">Website</a> | <a href="https://github.com/the-crypt-keeper/tldw">Source</a> | <a href="https://github.com/rmusser01/tldw_server">Source (Forked, ahead)</a></p> https://tedneward.github.io/Research/tools/treesheets/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/treesheets/index.html <p><a href="https://strlen.com/treesheets/">Website</a> | <a href="https://github.com/aardappel/treesheets">Source</a></p> <p>A "hierarchical spreadsheet" that is a great replacement for spreadsheets, mind mappers, outliners, PIMs, text editors and small databases.</p> <p>Suitable for any kind of data organization, such as todo lists, calendars, project management, brainstorming, organizing ideas, planning, requirements gathering, presentation of information, etc.</p> <p>It's like a spreadsheet, immediately familiar, but much more suitable for complex data because it's hierarchical.<br> It's like a mind mapper, but more organized and compact.<br> It's like an outliner, but in more than one dimension.<br> It's like a text editor, but with structure.</p> <p>Have a quick look at what the application looks like on the <a href="https://strlen.com/treesheets/docs/screenshots.html">screenshots</a> page, see how easy it is to use in the <a href="https://strlen.com/treesheets/docs/tutorial.html">tutorial</a>, then give it a download (above). A <a href="https://www.youtube.com/watch?v=UB-saQZfrsw">video</a> someone made (on Linux)</p> <p>TreeSheets is exceptionally small &amp; fast, so can sit in your system tray at all times: with several documents loaded representing the equivalent of almost 100 pages of text, it uses only 5MB of memory on Windows 7 (!)</p> https://tedneward.github.io/Research/tools/tup/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/tup/index.html <p><a href="http://gittup.org/tup/">Website</a> | <a href="https://github.com/gittup/tup">Github</a></p> <p><a href="http://hdl.handle.net/2078.1/223053">Lessons and Pitfalls in Building Firefox with Tup</a> by Guillaume Maudoux and Kim Mens (12th Seminar on Advanced Techniques &amp; Tools for Software Evolution (SATToSE) 2019)</p> https://tedneward.github.io/Research/tools/rumdl/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rumdl/index.html <p><a href="https://github.com/rvben/rumdl">Source</a></p> <p>rumdl is a high-performance Markdown linter and formatter that helps ensure consistency and best practices in your Markdown files. Inspired by <a href="https://github.com/astral-sh/ruff">ruff</a> 's approach to Python linting, rumdl brings similar speed and developer experience improvements to the Markdown ecosystem.</p> <p>It offers:</p> <ul> <li>⚡️ <strong>Built for speed</strong> with Rust - significantly faster than alternatives</li> <li>🔍 <strong>54 lint rules</strong> covering common Markdown issues</li> <li>🛠️ <strong>Automatic formatting</strong> with <code>--fix</code> for files and stdin/stdout</li> <li>📦 <strong>Zero dependencies</strong> - single binary with no runtime requirements</li> <li>🔧 <strong>Highly configurable</strong> with TOML-based config files</li> <li>🌐 <strong>Multiple installation options</strong> - Rust, Python, standalone binaries</li> <li>🐍 <strong>Installable via pip</strong> for Python users</li> <li>📏 <strong>Modern CLI</strong> with detailed error reporting</li> <li>🔄 <strong>CI/CD friendly</strong> with non-zero exit code on errors</li> </ul> https://tedneward.github.io/Research/tools/sbt/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sbt/index.html <p><a href="https://www.scala-sbt.org/">Website</a> | <a href="https://github.com/sbt/sbt">Source</a></p> https://tedneward.github.io/Research/tools/sed/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sed/index.html <h3>Reading</h3> <ul> <li><a href="https://learnbyexample.github.io/learn_gnused/">GNU sed</a> - Sundeep Agarwal</li> <li><a href="https://www.grymoire.com/Unix/Sed.html">Sed - An Introduction and Tutorial</a> - Bruce Barnett</li> </ul> https://tedneward.github.io/Research/tools/sherlock/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sherlock/index.html <p><a href="https://sherlockproject.xyz/">Website</a> | <a href="https://github.com/sherlock-project/sherlock">Source</a></p> https://tedneward.github.io/Research/tools/sophiascript/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sophiascript/index.html <p><a href="https://github.com/farag2/Sophia-Script-for-Windows">Source</a></p> https://tedneward.github.io/Research/tools/spec-kit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spec-kit/index.html <p><a href="https://github.com/github/spec-kit">Source</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/">Spec-driven development with AI: Get started with a new open source toolkit</a></li> <li><a href="https://medium.com/@abhinav.dobhal/revolutionizing-ai-powered-development-a-complete-guide-to-githubs-speckit-a85a39f0e2ee">Revolutionizing AI-Powered Development: A Complete Guide to GitHub’s SpecKit</a></li> </ul> https://tedneward.github.io/Research/tools/ropper/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ropper/index.html <p><a href="https://scoding.de/ropper/">Website</a> | <a href="https://github.com/sashs/ropper">Github</a></p> https://tedneward.github.io/Research/tools/runway/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/runway/index.html <p><a href="https://www.runway.team/">Website</a> | Commercial, cloud</p> https://tedneward.github.io/Research/tools/scc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/scc/index.html <p><a href="https://github.com/boyter/scc">Source</a></p> <p>A tool similar to cloc, sloccount and tokei. For counting the lines of code, blank lines, comment lines, and physical lines of source code in many programming languages.</p> <p>Goal is to be the fastest code counter possible, but also perform COCOMO calculation like sloccount, estimate code complexity similar to cyclomatic complexity calculators and produce unique lines of code or DRYness metrics. In short one tool to rule them all.</p> <p>Why use <code>scc</code>?</p> <ul> <li>It is very fast and gets faster the more CPU you throw at it</li> <li>Accurate</li> <li>Works very well across multiple platforms without slowdown (Windows, Linux, macOS)</li> <li>Large language support</li> <li>Can ignore duplicate files</li> <li>Has complexity estimations</li> <li>You need to tell the difference between Coq and Verilog in the same directory</li> <li>cloc yaml output support so potentially a drop in replacement for some users</li> <li>Can identify or ignore minified files</li> <li>Able to identify many #! files ADVANCED! <a href="https://github.com/boyter/scc/issues/115">https://github.com/boyter/scc/issues/115</a></li> <li>Can ignore large files by lines or bytes</li> <li>Can calculate the ULOC or unique lines of code by file, language or project</li> <li>Supports multiple output formats for integration, CSV, SQL, JSON, HTML and more</li> </ul> <p>Why not use <code>scc</code>?</p> <ul> <li>You don't like Go for some reason</li> <li>It cannot count D source with different nested multi-line comments correctly <a href="https://github.com/boyter/scc/issues/27">https://github.com/boyter/scc/issues/27</a></li> </ul> <h3>Differences</h3> <p>There are some important differences between <code>scc</code> and other tools that are out there. Here are a few important ones for you to consider.</p> <p>Blank lines inside comments are counted as comments. While the line is technically blank the decision was made that<br> once in a comment everything there should be considered a comment until that comment is ended. As such the following,</p> <pre><code>/* blank lines follow */ </code></pre> <p>Would be counted as 4 lines of comments. This is noticeable when comparing scc's output to other tools on large<br> repositories.</p> <p><code>scc</code> is able to count verbatim strings correctly. For example in C# the following,</p> <pre><code>private const string BasePath = @"a:\"; // The below is returned to the user as a version private const string Version = "1.0.0"; </code></pre> <p>Because of the prefixed @ this string ends at the trailing " by ignoring the escape character \ and as such should be<br> counted as 2 code lines and 1 comment. Some tools are unable to<br> deal with this and instead count up to the "1.0.0" as a string which can cause the middle comment to be counted as<br> code rather than a comment.</p> <p><code>scc</code> will also tell you the number of bytes it has processed (for most output formats) allowing you to estimate the<br> cost of running some static analysis tools.</p> <h3>Usage</h3> <p>Command line usage of <code>scc</code> is designed to be as simple as possible.<br> Full details can be found in <code>scc --help</code> or <code>scc -h</code>. Note that the below reflects the state of master not a release, as such<br> features listed below may be missing from your installation.</p> <pre><code>Sloc, Cloc and Code. Count lines of code in a directory with complexity estimation. Version 3.3.4 Ben Boyter &lt;ben@boyter.org&gt; + Contributors Usage: scc [flags] [files or directories] Flags: --avg-wage int average wage value used for basic COCOMO calculation (default 56286) --binary disable binary file detection --by-file display output for every file -m, --character calculate max and mean characters per line --ci enable CI output settings where stdout is ASCII --cocomo-project-type string change COCOMO model type [organic, semi-detached, embedded, "custom,1,1,1,1"] (default "organic") --count-as string count extension as language [e.g. jsp:htm,chead:"C Header" maps extension jsp to html and chead to C Header] --count-ignore set to allow .gitignore and .ignore files to be counted --currency-symbol string set currency symbol (default "$") --debug enable debug output -a, --dryness calculate the DRYness of the project (implies --uloc) --eaf float the effort adjustment factor derived from the cost drivers (1.0 if rated nominal) (default 1) --exclude-dir strings directories to exclude (default [.git,.hg,.svn]) -x, --exclude-ext strings ignore file extensions (overrides include-ext) [comma separated list: e.g. go,java,js] -n, --exclude-file strings ignore files with matching names (default [package-lock.json,Cargo.lock,yarn.lock,pubspec.lock,Podfile.lock,pnpm-lock.yaml]) --file-gc-count int number of files to parse before turning the GC on (default 10000) -f, --format string set output format [tabular, wide, json, json2, csv, csv-stream, cloc-yaml, html, html-table, sql, sql-insert, openmetrics] (default "tabular") --format-multi string have multiple format output overriding --format [e.g. tabular:stdout,csv:file.csv,json:file.json] --gen identify generated files --generated-markers strings string markers in head of generated files (default [do not edit,&lt;auto-generated /&gt;]) -h, --help help for scc -i, --include-ext strings limit to file extensions [comma separated list: e.g. go,java,js] --include-symlinks if set will count symlink files -l, --languages print supported languages and extensions --large-byte-count int number of bytes a file can contain before being removed from output (default 1000000) --large-line-count int number of lines a file can contain before being removed from output (default 40000) --min identify minified files -z, --min-gen identify minified or generated files --min-gen-line-length int number of bytes per average line for file to be considered minified or generated (default 255) --no-cocomo remove COCOMO calculation output -c, --no-complexity skip calculation of code complexity -d, --no-duplicates remove duplicate files from stats and output --no-gen ignore generated files in output (implies --gen) --no-gitignore disables .gitignore file logic --no-ignore disables .ignore file logic --no-large ignore files over certain byte and line size set by max-line-count and max-byte-count --no-min ignore minified files in output (implies --min) --no-min-gen ignore minified or generated files in output (implies --min-gen) --no-size remove size calculation output -M, --not-match stringArray ignore files and directories matching regular expression -o, --output string output filename (default stdout) --overhead float set the overhead multiplier for corporate overhead (facilities, equipment, accounting, etc.) (default 2.4) -p, --percent include percentage values in output --remap-all string inspect every file and remap by checking for a string and remapping the language [e.g. "-*- C++ -*-":"C Header"] --remap-unknown string inspect files of unknown type and remap by checking for a string and remapping the language [e.g. "-*- C++ -*-":"C Header"] --size-unit string set size unit [si, binary, mixed, xkcd-kb, xkcd-kelly, xkcd-imaginary, xkcd-intel, xkcd-drive, xkcd-bakers] (default "si") --sloccount-format print a more SLOCCount like COCOMO calculation -s, --sort string column to sort by [files, name, lines, blanks, code, comments, complexity] (default "files") --sql-project string use supplied name as the project identifier for the current run. Only valid with the --format sql or sql-insert option -t, --trace enable trace output (not recommended when processing multiple files) -u, --uloc calculate the number of unique lines of code (ULOC) for the project -v, --verbose verbose output --version version for scc -w, --wide wider output with additional statistics (implies --complexity) </code></pre> <p>Output should look something like the below for the redis project</p> <pre><code>$ scc redis ─────────────────────────────────────────────────────────────────────────────── Language Files Lines Blanks Comments Code Complexity ─────────────────────────────────────────────────────────────────────────────── C 296 180267 20367 31679 128221 32548 C Header 215 32362 3624 6968 21770 1636 TCL 143 28959 3130 1784 24045 2340 Shell 44 1658 222 326 1110 187 Autoconf 22 10871 1038 1326 8507 953 Lua 20 525 68 70 387 65 Markdown 16 2595 683 0 1912 0 Makefile 11 1363 262 125 976 59 Ruby 10 795 78 78 639 116 gitignore 10 162 16 0 146 0 YAML 6 711 46 8 657 0 HTML 5 9658 2928 12 6718 0 C++ 4 286 48 14 224 31 License 4 100 20 0 80 0 Plain Text 3 185 26 0 159 0 CMake 2 214 43 3 168 4 CSS 2 107 16 0 91 0 Python 2 219 12 6 201 34 Systemd 2 80 6 0 74 0 BASH 1 118 14 5 99 31 Batch 1 28 2 0 26 3 C++ Header 1 9 1 3 5 0 Extensible Styleshe… 1 10 0 0 10 0 Smarty Template 1 44 1 0 43 5 m4 1 562 116 53 393 0 ─────────────────────────────────────────────────────────────────────────────── Total 823 271888 32767 42460 196661 38012 ─────────────────────────────────────────────────────────────────────────────── Estimated Cost to Develop (organic) $6,918,301 Estimated Schedule Effort (organic) 28.682292 months Estimated People Required (organic) 21.428982 ─────────────────────────────────────────────────────────────────────────────── Processed 9425137 bytes, 9.425 megabytes (SI) ─────────────────────────────────────────────────────────────────────────────── </code></pre> <p>Note that you don't have to specify the directory you want to run against. Running <code>scc</code> will assume you want to run against the current directory.</p> <p>You can also run against multiple files or directories <code>scc directory1 directory2 file1 file2</code> with the results aggregated in the output.</p> <h3>Ignore Files</h3> <p><code>scc</code> mostly supports .ignore files inside directories that it scans. This is similar to how ripgrep, ag and tokei work. .ignore files are 100% the same as .gitignore files with the same syntax, and as such <code>scc</code> will ignore files and directories listed in them. You can add .ignore files to ignore things like vendored dependency checked in files and such. The idea is allowing you to add a file or folder to git and have ignored in the count.</p> <h3>Interesting Use Cases</h3> <p>Used inside Intel Nemu Hypervisor to track code changes between revisions <a href="https://github.com/intel/nemu/blob/topic/virt-x86/tools/cloc-change.sh#L9">https://github.com/intel/nemu/blob/topic/virt-x86/tools/cloc-change.sh#L9</a><br> Appears to also be used inside both <a href="http://codescoop.com/">http://codescoop.com/</a> <a href="https://pinpoint.com/">https://pinpoint.com/</a> <a href="https://github.com/chaoss/grimoirelab-graal">https://github.com/chaoss/grimoirelab-graal</a></p> <p>It also is used to count code and guess language types in <a href="https://searchcode.com/">https://searchcode.com/</a> which makes it one of the most frequently run code counters in the world.</p> <p>You can also hook scc into your gitlab pipeline <a href="https://gitlab.com/guided-explorations/ci-cd-plugin-extensions/ci-cd-plugin-extension-scc">https://gitlab.com/guided-explorations/ci-cd-plugin-extensions/ci-cd-plugin-extension-scc</a></p> <p>Also used by CodeQL <a href="https://github.com/boyter/scc/pull/317">https://github.com/boyter/scc/pull/317</a> and Scaleway <a href="https://twitter.com/Scaleway/status/1488087029476995074?s=20&amp;t=N2-z6O-ISDdDzULg4o4uVQ">https://twitter.com/Scaleway/status/1488087029476995074?s=20&amp;t=N2-z6O-ISDdDzULg4o4uVQ</a></p> <ul> <li><a href="https://docs.linuxfoundation.org/lfx/insights/v3-beta-version-current/getting-started/landing-page/cocomo-cost-estimation-simplified">https://docs.linuxfoundation.org/lfx/insights/v3-beta-version-current/getting-started/landing-page/cocomo-cost-estimation-simplified</a></li> <li><a href="https://openems.io/">https://openems.io/</a></li> </ul> <h3>Features</h3> <p><code>scc</code> uses a small state machine in order to determine what state the code is when it reaches a newline <code>\n</code>. As such it is aware of and able to count</p> <ul> <li>Single Line Comments</li> <li>Multi Line Comments</li> <li>Strings</li> <li>Multi Line Strings</li> <li>Blank lines</li> </ul> <p>Because of this it is able to accurately determine if a comment is in a string or is actually a comment.</p> <p>It also attempts to count the complexity of code. This is done by checking for branching operations in the code. For example, each of the following <code>for if switch while else || &amp;&amp; != ==</code> if encountered in Java would increment that files complexity by one.</p> <h3>Complexity Estimates</h3> <p>Let's take a minute to discuss the complexity estimate itself.</p> <p>The complexity estimate is really just a number that is only comparable to files in the same language. It should not be used to compare languages directly without weighting them. The reason for this is that its calculated by looking for branch and loop statements in the code and incrementing a counter for that file.</p> <p>Because some languages don't have loops and instead use recursion they can have a lower complexity count. Does this mean they are less complex? Probably not, but the tool cannot see this because it does not build an AST of the code as it only scans through it.</p> <p>Generally though the complexity there is to help estimate between projects written in the same language, or for finding the most complex file in a project <code>scc --by-file -s complexity</code> which can be useful when you are estimating on how hard something is to maintain, or when looking for those files that should probably be refactored.</p> https://tedneward.github.io/Research/tools/semmleql/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/semmleql/index.html <p><a href="https://github.com/Semmle/ql">Source</a></p> <p>Vulnerability hunting with Semmle QL, <a href="https://blogs.technet.microsoft.com/srd/2018/08/16/vulnerability-hunting-with-semmle-ql-part-1/">part 1</a></p> https://tedneward.github.io/Research/tools/simplenote/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/simplenote/index.html <p><a href="https://simplenote.com/">Website</a> | Source: <a href="https://github.com/Automattic/simplenote-electron">simplenote-electron</a> is the official Simplenote desktop app for Windows and Linux.</p> <p>For other platforms, see:</p> <ul> <li><a href="https://github.com/Automattic/simplenote-macos">simplenote-macos</a></li> <li><a href="https://github.com/Automattic/simplenote-ios">simplenote-ios</a></li> <li><a href="https://github.com/Automattic/simplenote-android">simplenote-android</a></li> </ul> https://tedneward.github.io/Research/tools/souffle/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/souffle/index.html <p><a href="https://souffle-lang.github.io/">Website</a> | <a href="https://github.com/souffle-lang/souffle">Source</a> | <a href="https://souffle-lang.github.io/tutorial">Tutorial</a></p> <h2>Examples/Intro</h2> <p>Say we have a Datalog file example.dl, whose contents are as shown.</p> <pre><code>.decl edge(x:number, y:number) .input edge .decl path(x:number, y:number) .output path path(x, y) :- edge(x, y). path(x, y) :- path(x, z), edge(z, y). </code></pre> <p>We see that edge is a .input relation, and so will be read from disk. Also, path is a .output relation, and so will be written to disk.</p> <p>The last two lines say that 1) “there is a path from x to y if there is an edge from x to y”, and 2) “there is a path from x to y if there is a path from x to some z, and there is an edge from that z to y”.</p> <p>So if the input edge relation is pairs of vertices in a graph, by these two rules the output path relation will give us all pairs of vertices x and y for which a path exists in that graph from x to y.</p> https://tedneward.github.io/Research/tools/sphinx/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sphinx/index.html <p><a href="https://www.sphinx-doc.org/en/master/">Website</a></p> <p>Uses <a href="/formats/restructuredtext">reStructuredText</a> for content. Not sure of the benefit above/beyond Markdown.</p> https://tedneward.github.io/Research/tools/rubberduck/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rubberduck/index.html <p><a href="https://rubberduckvba.com/">Website</a> | <a href="https://github.com/rubberduck-vba">Source</a></p> https://tedneward.github.io/Research/tools/rv/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rv/index.html <p><a href="https://github.com/spinel-coop/rv">Source</a></p> <h2>Quickstart</h2> <pre><code>brew install rv # zsh echo 'eval "$(rv shell init zsh)"' &gt;&gt; ~/.zshrc eval "$(rv shell init zsh)" # bash echo 'eval "$(rv shell init bash)"' &gt;&gt; ~/.bashrc eval "$(rv shell init bash)" # fish echo 'rv shell init fish | source' &gt;&gt; ~/.config/fish/config.fish rv shell init fish | source </code></pre> https://tedneward.github.io/Research/tools/schemacrawler/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/schemacrawler/index.html <p><a href="https://www.schemacrawler.com/">Website</a> | <a href="https://github.com/schemacrawler/SchemaCrawler">Source</a></p> <blockquote> <p>You can search for database schema objects using regular expressions, and output the schema and data in a readable text format. The output serves for database documentation, and is designed to be diff-ed against other database schemas. SchemaCrawler also generates schema diagrams. You can execute scripts in any standard scripting language against your database. You can find potential schema design issues with lint.</p> </blockquote> https://tedneward.github.io/Research/tools/semtools/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/semtools/index.html <p><a href="https://github.com/run-llama/semtools">Source</a></p> <p>By default, <code>parse</code> uses LlamaParse as a backend. Get your API key today for free at <a href="https://cloud.llamaindex.ai">https://cloud.llamaindex.ai</a>. <code>search</code> remains local-only.</p> https://tedneward.github.io/Research/tools/slashbase/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/slashbase/index.html <p><a href="https://slashbase.com/">Website</a> | <a href="https://github.com/slashbaseide/slashbase">Source</a></p> https://tedneward.github.io/Research/tools/spall/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spall/index.html <p><a href="https://gravitymoth.com/spall/">Website</a></p> https://tedneward.github.io/Research/tools/spin/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/spin/index.html <p><a href="https://www.fermyon.com/spin">Website</a> | <a href="https://github.com/spinframework/spin">Source</a></p> <p>Spin is an open source framework for building and running fast, secure, and composable cloud microservices with WebAssembly. It aims to be the easiest way to get started with WebAssembly microservices, and takes advantage of the latest developments in the WebAssembly component model and Wasmtime runtime.</p> <p>Spin offers a simple CLI that helps you create, distribute, and execute applications, and in the next sections we will learn more about Spin applications and how to get started.</p> https://tedneward.github.io/Research/tools/ruff/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ruff/index.html <p><a href="https://docs.astral.sh/ruff/">Website</a> | <a href="https://github.com/astral-sh/ruff">Source</a></p> <ul> <li>10-100x faster than existing linters (like Flake8) and formatters (like Black)</li> <li>Installable via pip</li> <li>pyproject.toml support</li> <li>Python 3.13 compatibility</li> <li>Drop-in parity with Flake8, isort, and Black</li> <li>Built-in caching, to avoid re-analyzing unchanged files</li> <li>Fix support, for automatic error correction (e.g., automatically remove unused imports)</li> <li>Over 800 built-in rules, with native re-implementations of popular Flake8 plugins, like flake8-bugbear</li> <li>First-party editor integrations for VS Code and more</li> <li>Monorepo-friendly, with hierarchical and cascading configuration</li> </ul> https://tedneward.github.io/Research/tools/sasm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sasm/index.html <p><a href="https://dman95.github.io/SASM/english.html">Website</a> | <a href="https://github.com/Dman95/SASM">Source</a></p> https://tedneward.github.io/Research/tools/scigen/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/scigen/index.html <p><a href="https://pdos.csail.mit.edu/archive/scigen/">Website</a> | <a href="https://github.com/strib/scigen">Source</a></p> <p>It uses a hand-written <strong>context-free grammar</strong> to form all elements of the papers. Our aim here is to maximize amusement, rather than coherence.</p> <p>One useful purpose for such a program is to auto-generate submissions to conferences that you suspect might have very low submission standards. A prime example, which you may recognize from spam in your inbox, is SCI/IIIS and its dozens of co-located conferences (check out the very broad conference description on the <a href="http://www.iiisci.org/sci2005/">WMSCI 2005</a> website). There's also a list of <a href="http://anthony.liekens.net/index.php/Misc/FakeConferences">known bogus conferences</a>. Using SCIgen to generate submissions for conferences like this gives us pleasure to no end. In fact, one of our papers was accepted to SCI 2005! See <a href="https://pdos.csail.mit.edu/archive/scigen/#examples">Examples</a> for more details.</p> <p>We went to WMSCI 2005. Check out the <a href="https://pdos.csail.mit.edu/archive/scigen/#talks">talks and video</a>. You can find more details in our <a href="https://pdos.csail.mit.edu/archive/scigen/blog/">blog</a>.</p> <p>Also, check out our 10th anniversary celebration project: <a href="https://pdos.csail.mit.edu/archive/scigen/scipher.html">SCIpher</a>!</p> https://tedneward.github.io/Research/tools/sharpcells/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sharpcells/index.html <p><a href="https://www.sharpcells.com/">Website</a></p> <ul> <li>Quickly and easily create custom functions and commands with with robustness and power of F# and .NET</li> <li>Sharp Cells can connect to data sources from anywhere in your organisation or across the web</li> <li>Sharp Cells commands let you automate almost any user action with a similar programming interface to VBA</li> <li>Sharp Cells scripts run on your machine and are saved with the workbook giving you the same single file portability as VBA</li> </ul> https://tedneward.github.io/Research/tools/slidev/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/slidev/index.html <p><a href="https://sli.dev/">Website</a> | <a href="https://github.com/slidevjs/slidev">Source</a></p> https://tedneward.github.io/Research/tools/sparta/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/sparta/index.html <p>A C++ library of software components for building high-performance static analyzers.</p> <p><a href="https://code.fb.com/open-source/sparta/">Website</a> | <a href="https://github.com/facebookincubator/SPARTA">Source</a></p> <p>Easy Abstract Interpretation with SPARTA<br> - Strange Loop 2019; Arnaud Venet and Jez Ng<br> - <a href="https://www.youtube.com/watch?v=_fA7vkVJhF8">https://www.youtube.com/watch?v=_fA7vkVJhF8</a><br> - <a href="https://thestrangeloop.com/2019/easy-abstract-interpretation-with-sparta.html">https://thestrangeloop.com/2019/easy-abstract-interpretation-with-sparta.html</a></p> https://tedneward.github.io/Research/tools/proguard/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/proguard/index.html <p><a href="https://github.com/Guardsquare/proguard-core">Source</a></p> <p>ProGuardCORE is a free library to read, analyze, modify, and write Java class files. It is the core of the well-known shrinker, optimizer, and obfuscator ProGuard, the ProGuard Assembler and Disassembler, and the Kotlin Metadata Printer.</p> <p>Typical applications:</p> <ul> <li>Read and write class files, including any Kotlin metadata.</li> <li>Search for instruction patterns.</li> <li>Create byte code instrumentation tools.</li> <li>Analyze code with abstract evaluation.</li> <li>Build static code analysis tools.</li> <li>Optimize and obfuscate, like ProGuard itself.</li> </ul> https://tedneward.github.io/Research/tools/pyew/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pyew/index.html <p>It does have support for hexadecimal viewing, disassembly (Intel 16, 32 and 64 bits), PE and ELF file formats (it performs code analysis and let you write scripts using an API to perform many types of analysis), follows direct call/jmp instructions in the interactive command line, displays function names and string data references; supports OLE2 format, PDF format and more. It also supports plugins to add more features to the tool.</p> <p><a href="https://github.com/joxeankoret/pyew">Github</a></p> https://tedneward.github.io/Research/tools/raddebugger/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/raddebugger/index.html <p><a href="https://github.com/EpicGamesExt/raddebugger">Source</a></p> https://tedneward.github.io/Research/tools/redex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/redex/index.html <p><a href="http://docs.racket-lang.org/redex/index.html">Website</a></p> https://tedneward.github.io/Research/tools/replit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/replit/index.html <p><a href="https://replit.com/">Website</a> Commercial, has a free tier</p> <p>Looks like cloud-hosted browser-based development IDEs/environments?</p> https://tedneward.github.io/Research/tools/resterm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/resterm/index.html <p><a href="https://github.com/unkn0wn-root/resterm">Source</a> | <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs">Docs</a></p> <hr> <p>Resterm is a terminal client for working with <strong>HTTP</strong>, <strong>GraphQL</strong>, and <strong>gRPC</strong> services. No cloud sync, no signups, no heavy desktop app, no bullshit.<br> It pairs a Vim-like-style editor with a workspace explorer, response diff, history, profiler and scripting so you can iterate on requests without leaving the keyboard (no mouse support is a feature here).</p> <h2>Highlights</h2> <ul> <li><strong>Workspace</strong> navigator that filters <code>.http</code> / <code>.rest</code> files, supports recursion and keeps request lists in sync as you edit.</li> <li><strong>Editor</strong> with inline syntax highlighting, search (<code>Ctrl+F</code>), clipboard motions, and inline metadata completions (type <code>@</code> for contextual hints).</li> <li><strong>Variable</strong> scopes with <code>@global</code> (environment-wide), <code>@var file</code> (document), <code>@var request</code> (per-call), plus compile-time constants (<code>@const</code>), captures, JavaScript hooks, and multi-step workflows with per-step expectations and overrides.</li> <li><strong>GraphQL</strong> helpers (<code>@graphql</code>, <code>@variables</code>, <code>@query</code>) and gRPC directives (<code>@grpc</code>, <code>@grpc-descriptor</code>, reflection, metadata).</li> <li><strong>WebSockets and SSE</strong> with scripted <code>@ws</code> steps, automatic transcripts and an interactive console for ad-hoc frames.</li> <li><strong>OpenAPI importer</strong> converts OpenAPI specs into Resterm-ready <code>.http</code> collections from the CLI.</li> <li><strong>Inline</strong> requests and <strong>curl</strong> import for one-off calls (<code>Ctrl+Enter</code> on a URL or curl block).</li> <li><strong>Pretty/Raw/Header/Diff/History/Stream</strong> views with optional split panes, pinned comparisons, and live event playback.</li> <li><strong>Built-in</strong> OAuth 2.0 client plus support for basic, bearer, API key, and custom header auth.</li> <li><strong>Latency</strong> with <code>@profile</code> to benchmark endpoints and render histograms right inside the TUI.</li> <li><strong>Tracing and Timeline</strong> with <code>@trace</code> to enable request tracing.</li> <li><strong>Multi-step workflows</strong> let you compose several named requests into one workflow (<code>@workflow</code> + <code>@step</code>), override per-step variables, and review aggregated results in History.</li> <li><strong>Multi-environment compare</strong> via <code>@compare</code> directives or the global <code>--compare</code> flag + <code>g+c</code> shortcut so you can run the same request across diffrent envs. (dev/stage/prod etc.)</li> </ul> <h2>Quick Start</h2> <p>If you just want to hit an API <strong>right now</strong>, you don’t need any files.</p> <ol> <li><strong>Start Resterm</strong></li> </ol> <pre><code class="language-bash"># In a project directory (no files required) resterm # Or explicitly point to a workspace resterm --workspace path/to/project </code></pre> <ol> <li><strong>Type a minimal request</strong></li> </ol> <ul> <li>Focus the editor pane (use <code>Tab</code> if needed).</li> <li>Press <code>i</code> or <code>a</code> and type a simple request, for example: <pre><code class="language-http">GET https://httpbin.org/status/204 </code></pre> </li> </ul> <ol> <li><strong>Send it</strong></li> </ol> <ul> <li>Press <code>Ctrl+Enter</code> to send the request.</li> <li>The response appears in the response pane on the right.</li> </ul> <ol> <li><strong>Move around</strong></li> </ol> <ul> <li><code>Tab</code> / <code>Shift+Tab</code> - cycle focus between sidebar, editor, and response.</li> <li><code>g+p</code> - jump directly to the response pane.</li> <li><code>g+i</code> - jump back to the editor.</li> </ul> <p>That’s enough to be productive: open Resterm, type a URL, hit <code>Ctrl+Enter</code>, see the response.</p> <hr> <h3>Quick Start (but with files)</h3> <p>If you work with <strong><code>.http</code> / <code>.rest</code> files</strong>, Resterm will discover and use them.</p> <ol> <li><strong>Open a directory with request files</strong></li> </ol> <ul> <li>Create or open a directory that contains <code>.http</code> / <code>.rest</code> files<br> (see <code>_examples/</code> in this repo for samples).</li> <li>Run: <pre><code class="language-bash">resterm # or resterm --workspace path/to/project </code></pre> </li> </ul> <ol> <li><strong>Pick and send a request</strong></li> </ol> <ul> <li>Use the <strong>sidebar</strong> to select a request from your <code>.http</code> / <code>.rest</code> files.</li> <li>Press <code>Ctrl+Enter</code> to send it.</li> <li>The response shows up in the right pane.</li> </ul> <ol> <li><strong>Minimal request file example</strong></li> </ol> <pre><code class="language-http">### Status check # @name status GET https://httpbin.org/status/204 User-Agent: resterm ### Authenticated echo # @name bearerEcho # @auth bearer {{auth.token}} GET https://httpbin.org/bearer Accept: application/json </code></pre> <hr> <h3>Navigation &amp; Layout Cheat Sheet</h3> <p>A few keys that make Resterm feel “native” quickly:</p> <ul> <li><strong>Pane focus &amp; layout</strong></li> <li><code>Tab</code> / <code>Shift+Tab</code> - move focus between sidebar, editor, and response.</li> <li><code>g+r</code> - jump to <strong>Requests</strong> (sidebar).</li> <li><code>g+i</code> - jump to <strong>Editor</strong>.</li> <li><code>g+p</code> - jump to <strong>Response</strong>.</li> <li><code>g+h</code> / <code>g+l</code> - adjust layout: <ul> <li>When the <strong>left pane</strong> (sidebar) is focused: change sidebar width.</li> <li>Otherwise: change editor/response split.</li> </ul> </li> <li><code>g+v</code> / <code>g+s</code> - toggle response pane between inline and stacked layout.</li> <li><code>g+1</code>, <code>g+2</code>, <code>g+3</code> + minimize/restore sidebar, editor, response.</li> <li><code>g+z</code> / <code>g+Z</code> - zoom the focused pane / clear zoom.</li> <li> <p><strong>Environments &amp; globals</strong></p> </li> <li><code>Ctrl+E</code> - switch environments.</li> <li><code>Ctrl+G</code> - inspect captured globals.</li> <li> <p><strong>Working with responses</strong></p> </li> <li><code>Ctrl+V</code> / <code>Ctrl+U</code> - split the response pane for side-by-side comparison.</li> <li>When response pane is focused: <ul> <li><code>Ctrl+Shift+C</code> or <code>g y</code> - copy the entire Pretty/Raw/Headers tab<br> to the clipboard (no mouse selection needed).</li> </ul> </li> </ul> <hr> <blockquote> <p>[!TIP]<br> <strong>If you only remember three shortcuts...</strong><br> - <code>Ctrl+Enter</code> - send request<br> - <code>Tab</code> / <code>Shift+Tab</code> - switch panes<br> - <code>g+p</code> - jump to response</p> </blockquote> <h2>Inline curl import</h2> <p>Drop a curl command into the editor and press <code>Ctrl+Enter</code> anywhere inside to turn it into a structured request. Resterm understands common flags (<code>-X</code>, <code>-H</code>, <code>--data*</code>, <code>--json</code>, <code>--url</code>, <code>--user</code>, <code>--compressed</code>, <code>-F/--form</code>, etc.), merges repeated data segments, and respects multipart uploads.</p> <pre><code class="language-bash">curl \ --compressed \ --url "https://httpbin.org/post?source=resterm&amp;case=multipart" \ --request POST \ -H "Accept: application/json" \ -H "X-Client: resterm-dev" \ --user resterm:test123 \ -F file=@README.md \ --form-string memo='Testing resterm inline curl with multiline value' \ --form-string meta='{"env":"test","attempt":1}' </code></pre> <p>If you copied the command from a shell, prefixes like <code>sudo</code> or <code>$</code> are ignored automatically. Resterm loads the file attachment, preserves multiline form fields, and applies compression/auth headers without extra tweaks.</p> <h2>Quick Configuration Overview</h2> <ul> <li>Environment files: <code>resterm.env.json</code> (or legacy <code>rest-client.env.json</code>) discovered in the file directory, workspace root, or current working directory. Explicit dotenv files (<code>.env</code>, <code>.env.*</code>, <code>*.env</code>) are supported via <code>--env-file</code> only; we derive the environment name from a <code>workspace</code> entry, then the file stem, falling back to <code>default</code> for bare <code>.env</code> files. Important: Dotenv loading is <strong>single-workspace</strong> for now. Use JSON when you need multiple environments in one file.</li> <li>CLI flags: <code>--workspace</code>, <code>--file</code>, <code>--env</code>, <code>--env-file</code>, <code>--timeout</code>, <code>--insecure</code>, <code>--follow</code>, <code>--proxy</code>, <code>--recursive</code>, <code>--from-openapi</code>, <code>--http-out</code>, <code>--openapi-base-var</code>, <code>--openapi-resolve-refs</code>, <code>--openapi-include-deprecated</code>, <code>--openapi-server-index</code>.</li> <li>Config directory: <code>$HOME/Library/Application Support/resterm</code>, <code>%APPDATA%\resterm</code>, or <code>$HOME/.config/resterm</code> (override with <code>RESTERM_CONFIG_DIR</code>).</li> <li>Themes: add <code>.toml</code> or <code>.json</code> files under <code>~/.config/resterm/themes</code> (override with <code>RESTERM_THEMES_DIR</code>) and switch them at runtime with <code>Ctrl+Alt+T</code> (or chord <code>g</code> then <code>t</code>).</li> </ul> <h2>Workflows</h2> <ul> <li>Combine existing requests with <code>@workflow</code> + <code>@step</code> blocks to build repeatable scenarios that run inside the TUI.</li> <li>Set per-step assertions (<code>expect.status</code>, <code>expect.statuscode</code>) and pass data between steps via <code>vars.request.*</code> and <code>vars.workflow.*</code> namespaces.</li> <li>View progress in the sidebar, and inspect the aggregated summary in History after the run.</li> <li>See <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#workflows-multi-step-workflows"><code>docs/resterm.md</code></a> for the full reference and <code>_examples/workflows.http</code> for a runnable sample workflow.</li> </ul> <h2>Compare Runs</h2> <blockquote> <p>Try <code>_examples/compare.http</code> to see <code>@compare</code> directives and the <code>g+c</code> shortcut in action (pair it with <code>resterm --compare dev,stage,prod</code> for instant multi-environment sweeps).</p> </blockquote> <p>Modern API work rarely stops at a single environment, so Resterm bakes in a compare workflow that takes seconds to use:</p> <ol> <li>Add <code># @compare dev stage prod base=stage</code> to any request (or launch the app with <code>--compare dev,stage,prod --compare-base stage</code>).</li> <li>Press <code>g+c</code> (or <code>Enter</code> if you mapped the command) to send the current request to every listed environment. Resterm flips into a split view automatically so you can watch progress live.</li> <li>When the run finishes, move to the Compare tab and use the arrow keys (PgUp/PgDn/Home/End work too) to highlight any environment. Press <code>Enter</code> and the primary pane shows that environment, the secondary pane pins the baseline and you land in the Diff tab so Pretty/Raw/Headers all reflect selected ↔ baseline.</li> <li>Loading a compare entry from History gives the same experience even if you are offline. Resterm rehydrates the snapshots so you can keep auditing deltas without rerunning requests.</li> </ol> <p>For more info. about compare runs, see <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#compare-runs">docs/resterm.md#compare-runs</a>.</p> <h2>Tracing &amp; Timeline</h2> <ul> <li>Enable per-phase network tracing by adding <code># @trace</code> metadata to a request. Budgets use <code>phase&lt;=duration</code> syntax (for example <code>dns&lt;=50ms total&lt;=300ms tolerance=25ms</code>). Supported phases mirror the HTTP client hooks: <code>dns</code>, <code>connect</code>, <code>tls</code>, <code>request_headers</code>, <code>request_body</code>, <code>ttfb</code>, <code>transfer</code>, and <code>total</code>.</li> <li>When a traced response arrives, a <strong>Timeline</strong> tab appears beside Pretty/Raw/Headers. It renders a proportional bar chart, annotates overruns, and lists budget breaches. Jump straight to it with <code>Ctrl+Alt+L</code> (or the <code>g+t</code> chord) and exit via the standard tab navigation.</li> <li>Trace data is available to scripts through the <code>trace</code> binding (<code>trace.enabled()</code>, <code>trace.phases()</code>, <code>trace.breaches()</code>, <code>trace.withinBudget()</code>, etc.), making CI assertions straightforward.</li> <li><code>_examples/trace.http</code> contains two runnable requests (one within budget, one intentionally breaching) for quick experimentation.</li> <li>Resterm can export spans to OpenTelemetry when <code>RESTERM_TRACE_OTEL_ENDPOINT</code> (or <code>--trace-otel-endpoint</code>) is set. Optional extras: <code>RESTERM_TRACE_OTEL_INSECURE</code> / <code>--trace-otel-insecure</code>, <code>RESTERM_TRACE_OTEL_SERVICE</code> / <code>--trace-otel-service</code>, <code>RESTERM_TRACE_OTEL_TIMEOUT</code>, and <code>RESTERM_TRACE_OTEL_HEADERS</code>.</li> <li>Spans are emitted only when tracing is enabled. Budget breaches and HTTP failures automatically mark spans with an error status so distributed traces surface anomalies clearly.</li> </ul> <p>Deep dive into budgets, keyboard shortcuts, and trace scripting in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#timeline--tracing">docs/resterm.md#timeline--tracing</a>.</p> <h2>OpenAPI imports</h2> <p>Resterm can translate an OpenAPI 3 specification into a <code>.http</code> collection directly from the CLI.</p> <pre><code class="language-bash">resterm \ --from-openapi openapi-test.yml \ --http-out openapi-test.http \ --openapi-base-var apiBase \ --openapi-resolve-refs \ --openapi-server-index 1 </code></pre> <ul> <li><code>--from-openapi</code> points at the source spec, <code>--http-out</code> controls the generated <code>.http</code> file (defaults to <code>&lt;spec&gt;.http</code> when omitted).</li> <li><code>--openapi-base-var</code> overrides the variable name injected for the base URL (falls back to <code>baseUrl</code>).</li> <li><code>--openapi-resolve-refs</code> enables kin-openapi's <code>$ref</code> resolution before generation.</li> <li><code>--openapi-include-deprecated</code> keeps deprecated operations that are skipped by default.</li> <li><code>--openapi-server-index</code> picks which server entry (0-based) should populate the base URL if multiple servers are defined.</li> </ul> <p>The repository ships with <code>openapi-specs.yml</code>, an intentionally full-featured spec that covers array/object query parameters, callbacks, and unsupported constructs (for example OpenID Connect). Those unsupported pieces surface as <code>Warning:</code> lines within the generated header comment so you can verify warning handling end-to-end.</p> <blockquote> <p>[!NOTE]<br> Resterm relies on <a href="https://github.com/getkin/kin-openapi"><code>kin-openapi</code></a>, which currently supports OpenAPI documents up to <strong>v3.0.1</strong>. Work on v3.1 support is tracked in <a href="https://github.com/getkin/kin-openapi/pull/1102">getkin/kin-openapi#1102</a>.</p> </blockquote> <p>More about OpenAPI importer and available flags in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#importing-openapi-specs">docs/resterm.md#importing-openapi-specs</a>.</p> <h2>Streaming (WebSocket &amp; SSE)</h2> <p>Streaming requests are first-class citizens in Resterm. Enable the <strong>Stream</strong> response tab to watch events in real time. See <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#streaming-sse--websocket">docs/resterm.md#streaming-sse--websocket</a> for the complete directive list.</p> <h3>Server-Sent Events</h3> <p>Annotate any HTTP request with <code># @sse</code> to keep the connection open and capture events:</p> <pre><code class="language-http">### Notification feed # @name streamNotifications # @sse duration=1m idle=5s max-events=50 GET https://api.example.com/notifications Accept: text/event-stream </code></pre> <p><code>@sse</code> accepts:</p> <ul> <li><code>duration</code> / <code>timeout</code> total session timeout before Resterm aborts the stream.</li> <li><code>idle</code> / <code>idle-timeout</code> maximum gap between events before the stream is closed.</li> <li><code>max-events</code> stop after N events (Resterm still records the transcript).</li> <li><code>max-bytes</code> / <code>limit-bytes</code> cap downloaded payload size.</li> </ul> <p>The Pretty/Raw/Headers tabs collapse into a JSON transcript when a stream finishes and the history entry exposes a summary (<code>events</code>, <code>bytes</code>, <code>reason</code>). More SSE options and transcript fields live in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#server-sent-events-sse">docs/resterm.md#server-sent-events-sse</a>.</p> <h3>WebSockets</h3> <p>Switch any request to WebSocket mode with <code># @websocket</code> and describe scripted steps with <code># @ws</code> lines:</p> <pre><code class="language-http">### Chat handshake # @name websocketChat # @websocket timeout=10s receive-timeout=5s subprotocols=chat.v2,json # @ws send {"type":"hello"} # @ws wait 1s # @ws send-json {"type":"message","text":"Hi"} # @ws close 1000 "client done" wss://chat.example.com/stream </code></pre> <p>or if you prefer just to open websocket connection:</p> <pre><code class="language-http">### Chat # @name websocketChat # @websocket ws://chat.example.com/stream </code></pre> <p>WebSocket options mirror runtime controls:</p> <ul> <li><code>timeout</code> - handshake deadline.</li> <li><code>receive-timeout</code> - idle receive window (0 keeps it open indefinitely).</li> <li><code>max-message-bytes</code> - hard cap for inbound payloads.</li> <li><code>subprotocols</code> - comma-separated list sent during the handshake.</li> <li><code>compression=&lt;true|false&gt;</code> - explicitly enable or disable per-message compression.</li> </ul> <p>Each <code>@ws</code> directive emits a step:</p> <ul> <li><code>send</code>/<code>send-json</code>/<code>send-base64</code>/<code>send-file</code> send text, JSON, base64, or file payloads.</li> <li><code>ping</code> / <code>pong</code> transmit control frames.</li> <li><code>wait &lt;duration&gt;</code> pauses before the next scripted action.</li> <li><code>close [code] [reason]</code> ends the session with an optional status.</li> </ul> <p>The transcript records sender/receiver, opcode, sizes, close metadata and elapsed time. History entries keep the conversation for later review or scripted assertions. See <a href="./docs/resterm.md#websockets-websocket-ws">docs/resterm.md#websockets-websocket-ws</a>.</p> <h3>Stream viewer &amp; console</h3> <ul> <li>Focus the response pane with <code>g+p</code>, then switch to the Stream tab using the left/right arrow keys (or <code>Ctrl+H</code> / <code>Ctrl+L</code>). Follow events live, bookmark frames and scrub after the stream completes.</li> <li>Toggle the interactive WebSocket console with <code>Ctrl+I</code> or <code>g+r</code> while the Stream tab is focused. Use <code>F2</code> to cycle payload modes (text, JSON, base64, file), <code>Ctrl+S</code> (or <code>Ctrl+Enter</code>) to send, arrows to navigate history, <code>Ctrl+P</code> for ping, <code>Ctrl+W</code> to close and <code>Ctrl+L</code> to clear the buffer.</li> <li>Scripted tests can consume transcripts via the <code>stream</code> API (<code>stream.kind</code>, <code>stream.summary</code>, <code>stream.events</code>, <code>stream.onEvent()</code>), enabling assertions on streaming workloads.<br> Find every stream history/export hook in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#stream-tab-history-and-console">docs/resterm.md#stream-tab-history-and-console</a>.</li> </ul> <h2>Custom themes</h2> <p>Resterm ships with a default palette, but you can provide your own by dropping theme definitions into the themes directory mentioned above. Each theme can be written in TOML or JSON and only needs to override the parts you care about.</p> <p>A ready-to-use sample lives in <a href="https://github.com/unkn0wn-root/resterm/blob/main/_examples/themes/aurora.toml">_examples/themes/aurora.toml</a>. Point <code>RESTERM_THEMES_DIR</code> env var at that folder to try it immediately.</p> <pre><code class="language-toml">[metadata] name = "Oceanic" author = "You" [styles.header_title] foreground = "#5fd1ff" bold = true [colors] pane_active_foreground = "#5fd1ff" pane_border_focus_file = "#1f6feb" </code></pre> <p>Save the file as <code>~/.config/resterm/themes/oceanic.toml</code> (or to your <code>RESTERM_THEMES_DIR</code>) and press <code>Ctrl+Alt+T</code> (or type <code>g</code> then <code>t</code>) inside Resterm to pick it as the default. The selected theme is persisted to <code>settings.toml</code> so it is restored on the next launch. The theming primer (directory layout, schema, and testing tips) lives in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#theming">docs/resterm.md#theming</a>.</p> <h2>Custom bindings</h2> <p>Resterm looks for <code>${RESTERM_CONFIG_DIR}/bindings.toml</code> first (and <code>bindings.json</code> second). Each entry maps an action ID to one or more bindings:</p> <pre><code class="language-toml">[bindings] save_file = ["ctrl+shift+s"] set_main_split_horizontal = ["g s", "ctrl+alt+s"] send_request = ["ctrl+enter", "cmd+enter"] </code></pre> <ul> <li>Modifiers use <code>+</code> (<code>ctrl+shift+o</code>), while chord steps are separated by spaces (<code>"g s"</code>).</li> <li>Only up to two steps are supported; <code>send_request</code> must stay single-step so it can fire while you type.</li> <li>Unknown actions or two actions sharing the same binding will be rejected (Resterm logs the error and keeps defaults).</li> </ul> <p>The complete action catalog and a binding reference live in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md#custom-bindings">docs/resterm.md#custom-bindings</a>.</p> <p>Check out the sample in <a href="https://github.com/unkn0wn-root/resterm/blob/main/_examples/bindings/bindings.toml">_examples/bindings/bindings.toml</a></p> <h2>Documentation</h2> <p>The full reference, including request syntax, metadata, directive tables, scripting APIs, transport settings and advanced workflows, lives in <a href="https://github.com/unkn0wn-root/resterm/blob/main/docs/resterm.md"><code>docs/resterm.md</code></a>.</p> https://tedneward.github.io/Research/tools/rockbot/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rockbot/index.html <p><a href="https://github.com/MarimerLLC/rockbot">Source</a> | <a href="https://hub.docker.com/r/rockylhotka/rockbot-agent/">Docker Hub</a></p> <h2>Resources</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://blog.lhotka.net/2026/03/03/The-RockBot-Band">Announcement</a></li> </ul> https://tedneward.github.io/Research/tools/pixi/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pixi/index.html <p><a href="https://pixi.prefix.dev/latest/">Website</a> | <a href="https://github.com/prefix-dev/pixi">Source</a></p> <h2>Overview</h2> <p><code>pixi</code> is a cross-platform, multi-language package manager and workflow tool built on the foundation of the conda ecosystem. It provides developers with an exceptional experience similar to popular package managers like <a href="https://doc.rust-lang.org/cargo/"><code>cargo</code></a> or <a href="https://docs.npmjs.com"><code>npm</code></a>, but for any language.</p> <ul> <li>Supports <strong>multiple languages</strong> including Python, C++, and R using Conda packages. You can find available packages on <a href="https://prefix.dev">prefix.dev</a>.</li> <li>Compatible with all major operating systems: Linux, Windows, macOS (including Apple Silicon).</li> <li>Always includes an up-to-date <a href="https://pixi.sh/latest/workspace/lockfile/"><strong>lock file</strong></a>.</li> <li>Provides a clean and simple Cargo-like <strong>command-line interface</strong>.</li> <li>Allows you to install tools <strong>per-project</strong> or <strong>system-wide</strong>.</li> <li>Entirely written in <strong>Rust</strong> and built on top of the <strong><a href="https://github.com/conda/rattler">rattler</a></strong> library.</li> </ul> <h2>Installation</h2> <p><code>pixi</code> can be installed on macOS, Linux, and Windows. The provided scripts will automatically download the latest version of <code>pixi</code>, extract it, and move the <code>pixi</code> binary to <code>~/.pixi/bin</code>. If this directory does not exist, the script will create it.</p> <h3>macOS and Linux</h3> <p>To install Pixi on macOS and Linux, open a terminal and run the following command:</p> <pre><code class="language-bash">curl -fsSL https://pixi.sh/install.sh | sh # or with brew brew install pixi </code></pre> <p>The script will also update your <code>~/.bashrc</code> to include <code>~/.pixi/bin</code> in your <code>PATH</code>, allowing you to invoke the <code>pixi</code> command from anywhere.<br> You might need to restart your terminal or source your shell for the changes to take effect.</p> <p>Starting with macOS Catalina <a href="https://support.apple.com/en-us/102360">zsh is the default login shell and interactive shell</a>. Therefore, you might want to use <code>zsh</code> instead of <code>bash</code> in the install command:</p> <pre><code class="language-zsh">curl -fsSL https://pixi.sh/install.sh | zsh </code></pre> <p>The script will also update your <code>~/.zshrc</code> to include <code>~/.pixi/bin</code> in your <code>PATH</code>, allowing you to invoke the <code>pixi</code> command from anywhere.</p> <h3>Windows</h3> <p>To install Pixi on Windows, open a PowerShell terminal (you may need to run it as an administrator) and run the following command:</p> <pre><code class="language-powershell">powershell -ExecutionPolicy ByPass -c "irm -useb https://pixi.sh/install.ps1 | iex" </code></pre> <p>Changing the <a href="https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_execution_policies?view=powershell-7.4#powershell-execution-policies">execution policy</a> allows running a script from the internet.<br> Check the script you would be running with:</p> <pre><code class="language-powershell">powershell -c "irm -useb https://pixi.sh/install.ps1 | more" </code></pre> <p>The script will inform you once the installation is successful and add the <code>~/.pixi/bin</code> directory to your <code>PATH</code>, which will allow you to run the <code>pixi</code> command from any location.<br> Or with <code>winget</code></p> <pre><code class="language-shell">winget install prefix-dev.pixi </code></pre> <h3>Autocompletion</h3> <p>To get autocompletion follow the instructions for your shell.<br> Afterwards, restart the shell or source the shell config file.</p> <h4>Bash (default on most Linux systems)</h4> <p>Add the following to the end of <code>~/.bashrc</code>:</p> <pre><code class="language-bash"># ~/.bashrc eval "$(pixi completion --shell bash)" </code></pre> <h4>Zsh (default on macOS)</h4> <p>Add the following to the end of <code>~/.zshrc</code>:</p> <pre><code class="language-zsh"># ~/.zshrc eval "$(pixi completion --shell zsh)" </code></pre> <h4>PowerShell (pre-installed on all Windows systems)</h4> <p>Add the following to the end of <code>Microsoft.PowerShell_profile.ps1</code>.<br> You can check the location of this file by querying the <code>$PROFILE</code> variable in PowerShell.<br> Typically the path is <code>~\Documents\PowerShell\Microsoft.PowerShell_profile.ps1</code> or<br> <code>~/.config/powershell/Microsoft.PowerShell_profile.ps1</code> on -Nix.</p> <pre><code class="language-pwsh">(&amp; pixi completion --shell powershell) | Out-String | Invoke-Expression </code></pre> <h4>Fish</h4> <p>Add the following to the end of <code>~/.config/fish/config.fish</code>:</p> <pre><code class="language-fish"># ~/.config/fish/config.fish pixi completion --shell fish | source </code></pre> <h4>Nushell</h4> <p>Add the following to your Nushell config file (find it by running <code>$nu.config-path</code> in Nushell):</p> <pre><code class="language-nushell">mkdir $"($nu.data-dir)/vendor/autoload" pixi completion --shell nushell | save --force $"($nu.data-dir)/vendor/autoload/pixi-completions.nu" </code></pre> <h4>Elvish</h4> <p>Add the following to the end of <code>~/.elvish/rc.elv</code>:</p> <pre><code class="language-elv"># ~/.elvish/rc.elv eval (pixi completion --shell elvish | slurp) </code></pre> <h4>Arch Linux</h4> <p>You can install <code>pixi</code> from the <a href="https://archlinux.org/packages/extra/x86_64/pixi/">extra repository</a> using <a href="https://wiki.archlinux.org/title/Pacman">pacman</a>:</p> <pre><code class="language-shell">pacman -S pixi </code></pre> <h4>Alpine Linux</h4> <p><code>pixi</code> is available for <a href="https://pkgs.alpinelinux.org/packages?name=pixi&amp;branch=edge">Alpine Edge</a>. It can be installed via <a href="https://wiki.alpinelinux.org/wiki/Alpine_Package_Keeper">apk</a> after enabling the <a href="https://wiki.alpinelinux.org/wiki/Repositories">testing repository</a>.</p> <pre><code class="language-shell">apk add pixi </code></pre> <h2>Build/install from source</h2> <p><code>pixi</code> is 100% written in Rust and therefore it can be installed, built and tested with cargo.<br> To start using <code>pixi</code> from a source build run:</p> <pre><code class="language-shell">cargo install --locked --git https://github.com/prefix-dev/pixi.git pixi </code></pre> <p>We don't publish to <code>crates.io</code> anymore, so you need to install it from the repository.<br> The reason for this is that we depend on some unpublished crates which disallows us to publish to <code>crates.io</code>.</p> <p>or when you want to make changes use:</p> <pre><code class="language-shell">cargo build cargo test </code></pre> <p>If you have any issues building because of the dependency on <code>rattler</code> checkout<br> it's <a href="https://github.com/conda/rattler/tree/main#give-it-a-try">compile steps</a></p> <h2>Uninstall</h2> <p>To uninstall, the Pixi binary should be removed.<br> Delete <code>pixi</code> from the <code>$PIXI_DIR</code> which is default to <code>~/.pixi/bin/pixi</code></p> <p>So on Linux its:</p> <pre><code class="language-shell">rm ~/.pixi/bin/pixi </code></pre> <p>and on Windows:</p> <pre><code class="language-shell">$PIXI_BIN = "$Env:LocalAppData\pixi\bin\pixi"; Remove-Item -Path $PIXI_BIN </code></pre> <p>After this command you can still use the tools you installed with <code>pixi</code>.<br> To remove these as well just remove the whole <code>~/.pixi</code> directory and remove the directory from your path.</p> <h1>Usage</h1> <p>The cli looks as follows:</p> <pre><code class="language-bash">➜ pixi Pixi [version 0.59.0] - Developer Workflow and Environment Management for Multi-Platform, Language-Agnostic Workspaces. Pixi is a versatile developer workflow tool designed to streamline the management of your workspace's dependencies, tasks, and environments. Built on top of the Conda ecosystem, Pixi offers seamless integration with the PyPI ecosystem. Basic Usage: Initialize pixi for a workspace: $ pixi init $ pixi add python numpy pytest Run a task: $ pixi task add test 'pytest -s' $ pixi run test Usage: pixi [OPTIONS] [COMMAND] Commands: add Adds dependencies to the workspace [aliases: a] auth Login to prefix.dev or anaconda.org servers to access private channels build Workspace configuration clean Cleanup the environments completion Generates a completion script for a shell config Configuration management exec Run a command and install it in a temporary environment [aliases: x] global Subcommand for global package management actions [aliases: g] info Information about the system, workspace and environments for the current machine init Creates a new workspace import Imports a file into an environment in an existing workspace. install Install an environment, both updating the lockfile and installing the environment [aliases: i] list List the packages of the current workspace [aliases: ls] lock Solve environment and update the lock file without installing the environments reinstall Re-install an environment, both updating the lockfile and re-installing the environment remove Removes dependencies from the workspace [aliases: rm] run Runs task in the pixi environment [aliases: r] search Search a conda package shell Start a shell in a pixi environment, run `exit` to leave the shell [aliases: s] shell-hook Print the pixi environment activation script task Interact with tasks in the workspace tree Show a tree of workspace dependencies [aliases: t] update The `update` command checks if there are newer versions of the dependencies and updates the `pixi.lock` file and environments accordingly upgrade Checks if there are newer versions of the dependencies and upgrades them in the lockfile and manifest file upload Upload a conda package workspace Modify the workspace configuration file through the command line help Print this message or the help of the given subcommand(s) Options: -V, --version Print version Global Options: -h, --help Display help information -v, --verbose... Increase logging verbosity (-v for warnings, -vv for info, -vvv for debug, -vvvv for trace) -q, --quiet... Decrease logging verbosity (quiet mode) --color &lt;COLOR&gt; Whether the log needs to be colored [env: PIXI_COLOR=] [default: auto] [possible values: always, never, auto] --no-progress Hide all progress bars, always turned on if stderr is not a terminal [env: PIXI_NO_PROGRESS=] --list List all installed commands (built-in and extensions) </code></pre> <h2>Creating a Pixi workspace</h2> <p>Initialize a new workspace and navigate to the workspace directory</p> <pre><code>pixi init myworkspace cd myworkspace </code></pre> <p>Add the dependencies you want to use</p> <pre><code>pixi add cowpy </code></pre> <p>Run the installed package in its environment</p> <pre><code class="language-bash">pixi run cowpy "Thanks for using pixi" </code></pre> <p>Activate a shell in the environment</p> <pre><code class="language-shell">pixi shell cowpy "Thanks for using pixi" exit </code></pre> <p>Check out <a href="https://pixi.sh/dev/first_workspace/">https://pixi.sh/dev/first_workspace/</a> for a more detailed introduction to workspaces.</p> <h2>Installing a conda package globally</h2> <p>You can also globally install conda packages into their own environment.<br> This behavior is similar to <a href="https://github.com/pypa/pipx"><code>pipx</code></a> or <a href="https://github.com/mariusvniekerk/condax"><code>condax</code></a>.</p> <pre><code class="language-bash">pixi global install cowpy </code></pre> https://tedneward.github.io/Research/tools/prose/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/prose/index.html <p><a href="https://www.microsoft.com/en-us/research/project/prose-framework/">Website</a> | <a href="https://github.com/microsoft/prose">Source</a></p> <blockquote> <p>Microsoft PROSE is a framework of technologies for programming by examples: automatic generation of programs from input-output examples. Given a domain-specific language (DSL) and input-output examples for the desired program’s behavior, PROSE synthesizes a ranked set of DSL programs that are consistent with the examples.</p> </blockquote> https://tedneward.github.io/Research/tools/qbdi/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/qbdi/index.html <p><a href="https://qbdi.quarkslab.com">Website</a> | <a href="https://github.com/quarkslab/QBDI">Source</a></p> <p><a href="https://events.ccc.de/congress/2017/Fahrplan/events/9006.html">Implementing an LLVM based Dynamic Binary Instrumentation framework</a>, 34C3 (2017)<br> - Example: plugging Triton on top of QBDI - <a href="http://shell-storm.org/repo/Notepad/qbdi_with_triton.txt">http://shell-storm.org/repo/Notepad/qbdi_with_triton.txt</a><br> - A Preliminary Test of QBDI - <a href="https://www.johnfxgalea.com/2018/01/13/a-preliminary-test-of-qbdi/">https://www.johnfxgalea.com/2018/01/13/a-preliminary-test-of-qbdi/</a><br> - Example: SRAC - a Simple Return Address Checker - <a href="https://github.com/johnfxgalea/SRAC">https://github.com/johnfxgalea/SRAC</a></p> https://tedneward.github.io/Research/tools/raycast/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/raycast/index.html <p><a href="https://www.raycast.com/">Website</a> | Freemium/commercial</p> https://tedneward.github.io/Research/tools/redpen/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/redpen/index.html <p><a href="https://redpen.cc/">Website</a> | <a href="https://github.com/redpen-cc/redpen">Source</a> <em>(Last update ~2019?)</em></p> <h1>Features</h1> <ul> <li>Language independent</li> <li>Highly customizable</li> <li>Multi platform (MacOS, Windows, and Linux)</li> <li>Support various markup text formats (Wiki, Markdown, AsciiDoc, LaTeX, Re:VIEW, reStructuredText)</li> <li><a href="http://blog.redpen.cc/2015/09/08/writing-extension-with-javascript/">Plugin system: users can write their extensions with JavaScript</a></li> <li><a href="https://blog.redpen.cc/2016/06/14/introduction-of-annotation-for-suppressing-errors-from-document-checking-tool-redpen/">Annotation based error suppression</a></li> <li><a href="http://redpen.herokuapp.com/">Practical REST API and UI</a></li> </ul> https://tedneward.github.io/Research/tools/repoassist/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/repoassist/index.html <p><a href="https://github.com/githubnext/agentics/blob/main/docs/repo-assist.md">Source</a></p> <p>Using Repo Assist is simple:</p> <ul> <li>You add it to your repository</li> <li>It does useful things automatically and repeatedly, at a pace you choose</li> <li>It reports in an issue</li> </ul> <h2>Reading</h2> <ul> <li><a href="https://dsyme.net/2026/02/25/repo-assist-a-repository-assistant/">Repo Assist: Crunching the Technical Debt with GitHub Agentic Workflows</a></li> </ul> https://tedneward.github.io/Research/tools/retdec/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/retdec/index.html <p>Supported file formats: ELF, PE, Mach-O, COFF, AR (archive), Intel HEX, and raw machine code.</p> <p>Supported architectures (32b only): Intel x86, ARM, MIPS, PIC32, and PowerPC.</p> <p><a href="https://github.com/avast-tl/retdec">Github</a></p> https://tedneward.github.io/Research/tools/pmd/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pmd/index.html <p><a href="https://pmd.github.io/">Website</a> | <a href="https://github.com/pmd/pmd">Source</a></p> https://tedneward.github.io/Research/tools/pulp/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pulp/index.html <p><a href="https://pulpproject.org/">Website</a> | <a href="https://github.com/pulp/pulpcore">Source</a></p> https://tedneward.github.io/Research/tools/qiew/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/qiew/index.html <p><a href="https://github.com/mtivadar/qiew">Github</a></p> <p>ELF and PE plugins available</p> https://tedneward.github.io/Research/tools/reaper/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/reaper/index.html <p>Source: <a href="https://github.com/EmergeTools/emerge-android">Android</a> <a href="https://github.com/getsentry/Reaper-iOS">iOS</a> <a href="https://github.com/getsentry/reaper-server">Server</a> | <a href="https://docs.emergetools.com/docs/quickstart">Docs</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://blog.sentry.io/an-open-source-sdk-for-finding-dead-code/">https://blog.sentry.io/an-open-source-sdk-for-finding-dead-code/</a></li> </ul> https://tedneward.github.io/Research/tools/refine/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/refine/index.html <p><a href="https://refine.dev/">Website</a> | <a href="https://github.com/refinedev/refine">Source</a></p> https://tedneward.github.io/Research/tools/rescuezilla/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rescuezilla/index.html <p><a href="https://rescuezilla.com/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/i-use-rescuezilla-to-swap-between-operating-systems-and-i-love-it/">I use Rescuezilla to swap between operating systems and I love it</a></li> </ul> https://tedneward.github.io/Research/tools/robomongo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/robomongo/index.html <p><a href="https://robomongo.org/">Website</a> | <a href="https://github.com/Studio3T/robomongo">Source</a></p> https://tedneward.github.io/Research/tools/poetry/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/poetry/index.html <p><a href="https://python-poetry.org/">Website</a> | <a href="https://github.com/python-poetry/poetry">Source</a></p> https://tedneward.github.io/Research/tools/pulumi/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pulumi/index.html <p><a href="https://www.pulumi.com/">Website</a> | <a href="https://github.com/pulumi/pulumi">Source</a></p> https://tedneward.github.io/Research/tools/rad/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/rad/index.html <p><a href="https://amterp.github.io/rad/">Website</a> | <a href="https://github.com/amterp/rad">Source</a> | <a href="https://amterp.github.io/rad/">Docs</a></p> <h2><a href="https://amterp.github.io/rad/guide/getting-started/">Getting Started</a></h2> <p>macOS: <code>brew install amterp/rad/rad</code></p> <h2>Examples</h2> <p>Fetch and display GitHub commits in one, minimal script:</p> <pre><code>#!/usr/bin/env rad args: repo str # The repo to query. Format: user/project limit int = 20 # Max commits to return url = "https://api.github.com/repos/{repo}/commits?per_page={limit}" Time = json[].commit.author.date Author = json[].commit.author.name SHA = json[].sha rad url: fields Time, Author, SHA </code></pre> <p>Put this <code>commits</code> script on your PATH, and invoke:</p> <pre><code>\&gt; commits spf13/cobra 3 Querying url: https://api.github.com/repos/spf13/cobra/commits?per\_page=3 Time Author SHA 2025-03-07T14:53:22Z styee 4f9ef8cdbbc88c5302be95e0e67fd78ebbfa9dd2 2025-02-21T12:46:14Z Fraser Waters 1995054b003053cc1e404bccfbf6d168e8731509 2025-02-17T19:16:17Z Yedaya Katsman f98cf4216d3cb5235e6e0cd00ee00959deb1dc65 </code></pre> <p>No <code>curl | jq | awk</code> gymnastics. No argparse boilerplate. Just readable code that does what you want.</p> <p>Explaining the above script:</p> <ol> <li>The script takes two args: a repo string and an optional limit (defaults to 20). <ul> <li>The <code>#</code> comments are read by Rad and used to generate helpful docs / usage strings for the script.</li> </ul> </li> <li>It uses string interpolation to build the url we will query, based on the supplied args.</li> <li>It defines the fields to extract from the JSON response.</li> <li>It executes the query, extracting the specified fields from the response, and displays the resulting data as a table. <ul> <li>Note the <code>rad url</code> syntax: "rad" actually stands for "request and display", which is what this built-in syntax does.</li> </ul> </li> </ol> <p>This example is kept somewhat minimal - there are Rad features we could use to further improve this.</p> <p>Some alternative valid invocations for this example:</p> <ul> <li><code>&gt; commits amterp/rad</code></li> <li><code>&gt; commits --repo amterp/rad --limit 5</code></li> <li><code>&gt; commits --limit 5 --repo amterp/rad</code></li> <li><code>&gt; commits amterp/rad --limit 5</code></li> </ul> https://tedneward.github.io/Research/tools/recaf/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/recaf/index.html <p><a href="https://recaf.coley.software/home.html">Website</a> | <a href="https://github.com/Col-E/Recaf">Source</a></p> https://tedneward.github.io/Research/tools/remedybg/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/remedybg/index.html <p><a href="https://remedybg.itch.io/remedybg">Website</a> |</p> https://tedneward.github.io/Research/tools/reshape/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/reshape/index.html <p><a href="https://github.com/fabianlindfors/reshape">Source</a> Not production-ready</p> https://tedneward.github.io/Research/tools/robotframework/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/robotframework/index.html <p><a href="https://robotframework.org/">Website</a> | <a href="https://github.com/robotframework/robotframework">Source</a></p> <p>Example:</p> <pre><code>*** Settings *** Documentation A test suite with a single test for valid login. ... ... This test has a workflow that is created using keywords in ... the imported resource file. Resource resource.txt *** Test Cases *** Valid Login Open Browser To Login Page Input Username demo Input Password mode Submit Credentials Welcome Page Should Be Open [Teardown] Close Browser </code></pre> https://tedneward.github.io/Research/tools/nuxt/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nuxt/index.html <p><a href="https://nuxtjs.org/">Website</a> | <a href="https://github.com/netlify/staticgen">Souce</a></p> https://tedneward.github.io/Research/tools/omnisharp/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/omnisharp/index.html <p><a href="http://www.omnisharp.net/">Website</a></p> https://tedneward.github.io/Research/tools/opencloud/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/opencloud/index.html <p><a href="https://opencloud.eu/en">Website</a> | <a href="https://github.com/opencloud-eu">Source</a> | Documentation: <a href="https://docs.opencloud.eu/docs/user/">User</a>, <a href="https://docs.opencloud.eu/docs/admin/">Admin</a>, <a href="https://docs.opencloud.eu/docs/dev/">Developer</a></p> <h2>Getting Started</h2> <h3><a href="https://docs.opencloud.eu/docs/admin/getting-started/container/docker">Docker</a></h3> <p>Create Required Directories for Bind Mounts</p> <pre><code> mkdir -p $HOME/opencloud/opencloud-config mkdir -p $HOME/opencloud/opencloud-data </code></pre> <p>Pull OpenCloud Image</p> <pre><code> docker pull opencloudeu/opencloud-rolling:latest </code></pre> <p>Initialize OpenCloud (First-time Setup) - You can set your own password using <code>IDM_ADMIN_PASSWORD=your_password</code>. If not set, a password will be auto-generated</p> <pre><code> docker run --rm -it \ -v $HOME/opencloud/opencloud-config:/etc/opencloud \ -v $HOME/opencloud/opencloud-data:/var/lib/opencloud \ -e IDM_ADMIN_PASSWORD=admin \ opencloudeu/opencloud-rolling:latest init </code></pre> <p>Start OpenCloud</p> <pre><code> docker run \ --name opencloud \ --rm \ -d \ -p 9200:9200 \ -v $HOME/opencloud/opencloud-config:/etc/opencloud \ -v $HOME/opencloud/opencloud-data:/var/lib/opencloud \ -e OC_INSECURE=true \ -e PROXY_HTTP_ADDR=0.0.0.0:9200 \ -e OC_URL=https://localhost:9200 \ opencloudeu/opencloud-rolling:latest </code></pre> <p>Login with your browser:</p> <pre><code> https://localhost:9200 user: admin password: admin </code></pre> <h3><a href="https://docs.opencloud.eu/docs/admin/getting-started/container/docker-compose/docker-compose-base/">Docker Compose</a></h3> <p>Goal: Install a internet facing OpenCloud with SSL certification with Docker Compose. This installation documentation is for Ubuntu and Debian systems. The software can also be installed on other Linux distributions, but the commands and package managers may differ.</p> <h4>Prerequisites</h4> <p>Four domains pointing to your server:</p> <ul> <li>cloud.YOUR.DOMAIN → OpenCloud frontend</li> <li>collabora.YOUR.DOMAIN → Collabora Online Server</li> <li>wopiserver.YOUR.DOMAIN → WOPI server for document editing</li> <li>traefik.YOUR.DOMAIN → Traefik dashboard</li> </ul> <p>Alternatively, you can use a wildcard domain (*.YOUR.DOMAIN)</p> <p>A hosted server (e.g., Hetzner, AWS, or your own VPS) with Linux and SSH access</p> <p>Log into your server via SSH: <code>ssh root@YOUR.SERVER.IP</code></p> <p>First, perform an update and upgrade: <code>apt update &amp;&amp; apt upgrade -y</code></p> <p>Install Docker following the official Docker guide. Once Docker is installed, enable and start the service: <code>systemctl enable docker &amp;&amp; systemctl start docker</code></p> <p>Download the necessary configuration files: <code>git clone https://github.com/opencloud-eu/opencloud-compose.git</code></p> <h4>Configure the .env File for Staging Certificates</h4> <p>Before requesting real SSL certificates, test the setup with Let's Encrypt’s staging environment.</p> <p>Navigate to the OpenCloud configuration folder: <code>cd opencloud-compose</code></p> <p>Create environment file: <code>cp .env.example .env</code></p> <p>NOTE: The repository includes .env.example as a template with default settings and documentation. Your actual .env file is excluded from version control (via .gitignore) to prevent accidentally committing sensitive information like passwords and domain-specific settings.</p> <p>Edit the .env file with the editor of your choice.</p> <ul> <li> <p>Disable insecure mode</p> <pre><code># INSECURE=true </code></pre> </li> <li> <p>Set your domain names</p> <pre><code>TRAEFIK_DOMAIN=traefik.YOUR.DOMAIN OC_DOMAIN=cloud.YOUR.DOMAIN COLLABORA_DOMAIN=collabora.YOUR.DOMAIN WOPISERVER_DOMAIN=wopiserver.YOUR.DOMAIN </code></pre> </li> <li> <p>Set your admin password</p> <pre><code>INITIAL_ADMIN_PASSWORD=YourSecurePassword </code></pre> </li> <li> <p>Set your email for SSL certification</p> <pre><code>TRAEFIK_ACME_MAIL=your@email.com </code></pre> </li> <li> <p>Use Let's Encrypt staging certificates (for testing)</p> <pre><code>TRAEFIK_ACME_CASERVER=https://acme-staging-v02.api.letsencrypt.org/directory </code></pre> </li> <li> <p>Set your deployment options. For Example without Collabora:</p> <pre><code>COMPOSE_FILE=docker-compose.yml:traefik/opencloud.yml </code></pre> </li> </ul> <p>Save and exit.</p> <h4>Production Setup Consideration</h4> <p>Production Setup Recommended: By default, OpenCloud stores configuration and data inside internal Docker volumes. This works fine for local development or quick evaluations — but is not suitable for production environments.</p> <p>In production, you should mount persistent local directories for configuration and data to ensure:</p> <ul> <li>Data durability</li> <li>Easier backups and recovery</li> <li>Full control over storage location and permissions</li> </ul> <p>Update your .env file with custom paths:</p> <pre><code> OC_CONFIG_DIR=/your/local/path/opencloud/config OC_DATA_DIR=/your/local/path/opencloud/data </code></pre> <p>Folder Permissions: Ensure these folders exist and are owned by user and group 1000:1000, which the Docker containers use by default:</p> <pre><code> sudo mkdir -p /your/local/path/opencloud/{config,data} sudo chown -R 1000:1000 /your/local/path/opencloud </code></pre> <p>If these variables are left unset, Docker will use internal volumes, which do not persist if the containers are removed — not recommended for real-world use.</p> <blockquote> <p>Security Warning: The user with UID 1000 on your host system will have full access to these mounted directories. This means that any local user account with this ID can read, modify, or delete OpenCloud config and data files. This can pose a security risk in shared or multi-user environments. Make sure to implement proper user and permission management and consider isolating access to these directories.</p> </blockquote> <p>Use production release container: To avoid accidentally updating to a version with breaking changes, you should specify the production container version to be used in your .env file:</p> <pre><code> OC_DOCKER_IMAGE=opencloudeu/opencloud OC_DOCKER_TAG=2 </code></pre> <h4>Start OpenCloud</h4> <p>Launch OpenCloud using Docker Compose: <code>docker compose up -d</code> This will start all required services in the background.</p> <h4>Verify SSL Certification</h4> <p>In your web browser, visit:</p> <p><a href="https://cloud.YOUR.DOMAIN">https://cloud.YOUR.DOMAIN</a></p> <p>You should see a security warning because the staging certificate is not fully trusted. Same should appear with the other domains you are using.</p> <p>Certificate Details: Check the certificate details to confirm it’s from Let's Encrypt Staging.</p> <h4>Apply a Real SSL Certificate</h4> <p>Once the staging certificate works, switch to a production certificate.</p> <p>Stop Docker Compose: <code>docker compose down</code></p> <p>Remove old staging certificates: <code>rm -r certs</code></p> <p>Disable staging mode in .env: Edit .env</p> <p>Comment the staging server:</p> <pre><code> # TRAEFIK_ACME_CASERVER=https://acme-staging-v02.api.letsencrypt.org/directory </code></pre> <p>Restart OpenCloud with a real SSL certificate</p> <pre><code> docker compose up -d </code></pre> <p>Now, visiting <a href="https://cloud.YOUR.DOMAIN">https://cloud.YOUR.DOMAIN</a> should show a secure connection with a valid SSL certificate.</p> <h4>Log into OpenCloud</h4> <p>Open a browser and visit:</p> <pre><code> https://cloud.YOUR.DOMAIN </code></pre> <p>Login with:</p> <pre><code> Username: admin Password: (your password) </code></pre> https://tedneward.github.io/Research/tools/opentofu/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/opentofu/index.html <p><a href="https://opentofu.org/">Website</a> | <a href="https://github.com/opentofu/opentofu">Source</a></p> <p>What is OpenTofu? OpenTofu is an infrastructure as code tool that lets you define both cloud and on-prem resources in human-readable configuration files that you can version, reuse, and share. You can then use a consistent workflow to provision and manage all of your infrastructure throughout its lifecycle. OpenTofu can manage low-level components like compute, storage, and networking resources, as well as high-level components like DNS entries and SaaS features.</p> https://tedneward.github.io/Research/tools/pandoc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pandoc/index.html <p><a href="https://pandoc.org/">Website</a> | <a href="https://github.com/jgm/pandoc/">Source</a> | <a href="https://pandoc.org/demos.html">Demos</a></p> <p>Written in Haskell.</p> <p>Converts to/from more formats than most people realize exist:</p> <p>(← = conversion from; → = conversion to; ↔︎ = conversion from and to)</p> <p>Lightweight markup formats:</p> <ul> <li>↔︎ Markdown (including CommonMark and GitHub-flavored Markdown)</li> <li>↔︎ reStructuredText</li> <li>↔︎ AsciiDoc</li> <li>↔︎ Emacs Org-Mode</li> <li>↔︎ Emacs Muse</li> <li>↔︎ Textile</li> <li>→ Markua</li> <li>← txt2tags</li> <li>↔︎ djot</li> <li>→ BBCode</li> </ul> <p>HTML formats</p> <ul> <li>↔︎ (X)HTML 4</li> <li>↔︎ HTML5</li> <li>→ Chunked HTML</li> </ul> <p>Ebooks</p> <ul> <li>↔︎ EPUB version 2 or 3</li> <li>↔︎ FictionBook2</li> </ul> <p>Documentation formats</p> <ul> <li>→ GNU TexInfo</li> <li>← pod</li> <li>↔︎ Haddock markup</li> <li>→ Vimdoc</li> </ul> <p>Roff formats</p> <ul> <li>↔︎ roff man</li> <li>→ roff ms</li> <li>← mdoc</li> </ul> <p>TeX formats</p> <ul> <li>↔︎ LaTeX</li> <li>→ ConTeXt</li> </ul> <p>XML formats</p> <ul> <li>↔︎ DocBook version 4 or 5</li> <li>↔︎ JATS</li> <li>← BITS</li> <li>→ TEI Simple</li> <li>→ OpenDocument XML</li> </ul> <p>Outline formats</p> <ul> <li>↔︎ OPML</li> </ul> <p>Bibliography formats</p> <ul> <li>↔︎ BibTeX</li> <li>↔︎ BibLaTeX</li> <li>↔︎ CSL JSON</li> <li>↔︎ CSL YAML</li> <li>← RIS</li> <li>← EndNote XML</li> </ul> <p>Word processor formats</p> <ul> <li>↔︎ Microsoft Word docx</li> <li>↔︎ Rich Text Format RTF</li> <li>↔︎ OpenOffice/LibreOffice ODT</li> </ul> <p>Interactive notebook formats</p> <ul> <li>↔︎ Jupyter notebook (ipynb)</li> </ul> <p>Page layout formats</p> <ul> <li>→ InDesign ICML</li> <li>↔︎ Typst</li> </ul> <p>Wiki markup formats</p> <ul> <li>↔︎ MediaWiki markup</li> <li>↔︎ DokuWiki markup</li> <li>← TikiWiki markup</li> <li>← TWiki markup</li> <li>← Vimwiki markup</li> <li>→ XWiki markup</li> <li>→ ZimWiki markup</li> <li>↔︎ Jira wiki markup</li> <li>← Creole</li> </ul> <p>Slide show formats</p> <ul> <li>→ LaTeX Beamer</li> <li>↔︎ Microsoft PowerPoint</li> <li>→ Slidy</li> <li>→ reveal.js</li> <li>→ Slideous</li> <li>→ S5</li> <li>→ DZSlides</li> </ul> <p>Data formats</p> <ul> <li>← CSV tables</li> <li>← TSV tables</li> <li>← Microsoft Excel spreadsheets</li> </ul> <p>Terminal output</p> <ul> <li>→ ANSI-formatted text</li> </ul> <p>Serialization formats</p> <ul> <li>↔︎ Haskell AST</li> <li>↔︎ JSON representation of AST</li> <li>↔︎ XML representation of AST</li> </ul> <p>Custom formats</p> <ul> <li>↔︎ custom readers and writers can be written in Lua</li> </ul> <p>PDF</p> <ul> <li>→ via pdflatex, lualatex, xelatex, latexmk, tectonic, wkhtmltopdf, weasyprint, prince, pagedjs-cli, context, or pdfroff.</li> </ul> <h2>Installation</h2> <ul> <li>macOS: <code>brew install pandoc</code>; integrations may be useful too: <code>brew install librsvg python homebrew/cask/basictex</code></li> <li>Docker: <code>docker run --rm --volume "</code>pwd<code>:/data" --user</code>id -u<code>:</code>id -g<code>pandoc/latex README.md -o README.pdf</code> converts README.md to README.pdf <ul> <li>The official Docker images for pandoc can be found at <a href="https://github.com/pandoc/dockerfiles">https://github.com/pandoc/dockerfiles</a> and at dockerhub.</li> <li>The <a href="https://hub.docker.com/r/pandoc/core">pandoc/core</a> image contains pandoc. The <a href="https://hub.docker.com/r/pandoc/latex">pandoc/latex</a> image also contains the minimal LaTeX installation needed to produce PDFs using pandoc.</li> </ul> </li> </ul> <h2>Build Actions</h2> <p>Pandoc can be run through <a href="https://github.com/features/actions">GitHub Actions</a>. For some examples, see <a href="https://github.com/pandoc/pandoc-action-example">https://github.com/pandoc/pandoc-action-example</a>.</p> <p>Pandoc can be run through <a href="https://about.gitlab.com/stages-devops-lifecycle/continuous-integration/">GitLab CI/CD</a>. For some examples, see <a href="https://gitlab.com/pandoc/pandoc-ci-example">https://gitlab.com/pandoc/pandoc-ci-example</a>.</p> https://tedneward.github.io/Research/tools/pdfkeeper/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pdfkeeper/index.html <p><a href="https://www.pdfkeeper.org/">Website</a> | <a href="https://github.com/rffrasca/PDFKeeper/">Source</a></p> <p>PDFKeeper began as an idea to solve a personal document storage issue. With both digital and paper documents stored in many places, it was becoming a challenge to locate documents quickly, especially at tax time. What proved to be an even bigger challenge was locating a free solution that could store PDF documents in a free, full-text search ready, relational database that could be accessed by anyone on the local area network. In addition, I had a requirement to store notes in the database with the PDF document that would also be searchable.</p> <p>​In early 2009, after being unsuccessful in locating a solution, I started building PDFKeeper, an open-source application that would use a free, full-text search ready, relational database for PDF document and notes storage, targeted at the Small Office Home Office and Small Business community. To develop the application quickly, I decided to build it with Open Object REXX, a scripting language with GUI support that I was very familiar with from my OS/2 days. As for the database, I chose to use Oracle® Database XE since it was free and was capable of indexing PDF documents; but it was my plan to expand database compatibility to other platforms over time.</p> <p>​## Reading</p> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/free-open-source-app-only-utility-need-for-managing-hundreds-of-pdfs/">https://www.xda-developers.com/free-open-source-app-only-utility-need-for-managing-hundreds-of-pdfs/</a></li> </ul> https://tedneward.github.io/Research/tools/performance/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/performance/index.html <h2>Software</h2> <ul> <li>benchmark (Google) <ul> <li><a href="https://github.com/google/benchmark">https://github.com/google/benchmark</a></li> <li><a href="https://opensource.googleblog.com/2014/01/introducing-benchmark.html">https://opensource.googleblog.com/2014/01/introducing-benchmark.html</a></li> </ul> </li> <li>Celero <ul> <li><a href="https://github.com/DigitalInBlue/Celero">https://github.com/DigitalInBlue/Celero</a></li> </ul> </li> <li>hayai: the C++ benchmarking framework <ul> <li><a href="https://github.com/nickbruun/hayai">https://github.com/nickbruun/hayai</a></li> </ul> </li> <li>moodycamel::microbench <ul> <li><a href="https://github.com/cameron314/microbench">https://github.com/cameron314/microbench</a></li> </ul> </li> <li>geiger: A micro benchmark library in C++ that supports hardware performance counters <ul> <li><a href="https://github.com/david-grs/geiger">https://github.com/david-grs/geiger</a></li> </ul> </li> <li>Nonius: A C++ micro-benchmarking framework <ul> <li><a href="https://github.com/libnonius/nonius">https://github.com/libnonius/nonius</a></li> </ul> </li> </ul> <h2>Readings</h2> <ul> <li>Examples: <ul> <li><a href="http://www.bfilipek.com/2016/01/micro-benchmarking-libraries-for-c.html">http://www.bfilipek.com/2016/01/micro-benchmarking-libraries-for-c.html</a></li> <li><a href="http://www.bfilipek.com/2016/02/revisiting-old-benchmark-vector-of.html">http://www.bfilipek.com/2016/02/revisiting-old-benchmark-vector-of.html</a></li> <li><a href="http://www.bfilipek.com/2016/05/google-benchmark-library.html">http://www.bfilipek.com/2016/05/google-benchmark-library.html</a></li> </ul> </li> <li>Systems Benchmarking Crimes - Gernot Heiser - <a href="https://www.cse.unsw.edu.au/~gernot/benchmarking-crimes.html">https://www.cse.unsw.edu.au/~gernot/benchmarking-crimes.html</a></li> </ul> <h1>Memory</h1> <h2>Memory - Benchmarking</h2> <ul> <li>Intel Memory Latency Checker (MLC) <ul> <li>a tool used to measure memory latencies and bandwidth, and how they change with increasing load on the system</li> <li><a href="https://www.intel.com/software/mlc">https://www.intel.com/software/mlc</a></li> </ul> </li> <li>Memory Bandwidth Benchmark <ul> <li>MBW determines the "copy" memory bandwidth available to userspace programs. Its simplistic approach models that of real applications. It is not tuned to extremes and it is not aware of hardware architecture, just like your average software package.</li> <li><a href="https://github.com/raas/mbw">https://github.com/raas/mbw</a></li> </ul> </li> <li>pmbw: Parallel Memory Bandwidth Benchmark / Measurement <ul> <li>a set of assembler routines to measure the parallel memory (cache and RAM) bandwidth of modern multi-core machines</li> <li><a href="http://panthema.net/2013/pmbw/">http://panthema.net/2013/pmbw/</a></li> <li><a href="https://github.com/bingmann/pmbw">https://github.com/bingmann/pmbw</a></li> </ul> </li> <li>Spatter: Benchmark for measuring the performance of sparse and irregular memory access <ul> <li><a href="https://github.com/hpcgarage/spatter">https://github.com/hpcgarage/spatter</a></li> <li>Spatter: A Tool for Evaluating Gather / Scatter Performance <ul> <li><a href="https://arxiv.org/abs/1811.03743">https://arxiv.org/abs/1811.03743</a></li> </ul> </li> </ul> </li> <li>STREAM: Sustainable Memory Bandwidth in High Performance Computers <ul> <li><a href="http://www.cs.virginia.edu/stream/">http://www.cs.virginia.edu/stream/</a></li> <li>STREAM benchmark - <a href="https://github.com/jeffhammond/STREAM">https://github.com/jeffhammond/STREAM</a></li> <li>NUMA-STREAM - <a href="https://github.com/larsbergstrom/NUMA-STREAM">https://github.com/larsbergstrom/NUMA-STREAM</a></li> <li>BabelStream: STREAM, for lots of devices written in many programming models <ul> <li><a href="https://github.com/UoB-HPC/BabelStream">https://github.com/UoB-HPC/BabelStream</a></li> <li><a href="http://uob-hpc.github.io/BabelStream/">http://uob-hpc.github.io/BabelStream/</a></li> </ul> </li> </ul> </li> <li>tinymembench: simple benchmark for memory throughput and latency <ul> <li><a href="https://github.com/ssvb/tinymembench">https://github.com/ssvb/tinymembench</a></li> </ul> </li> </ul> <h2>Memory - Profiling</h2> <ul> <li>Heaptrack - A Heap Memory Profiler for Linux <ul> <li><a href="https://github.com/KDE/heaptrack">https://github.com/KDE/heaptrack</a></li> <li><a href="http://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux">http://milianw.de/blog/heaptrack-a-heap-memory-profiler-for-linux</a></li> <li><a href="https://www.kdab.com/heaptrack-v1-0-0-release/">https://www.kdab.com/heaptrack-v1-0-0-release/</a></li> </ul> </li> <li>How to Write a Heap Memory Profiler <ul> <li>CppCon 2019; Milian Wolff</li> <li><a href="https://www.youtube.com/watch?v=YB0QoWI-g8E">https://www.youtube.com/watch?v=YB0QoWI-g8E</a></li> <li>Slides &amp; code: <a href="https://github.com/milianw/how-to-write-a-memory-profiler">https://github.com/milianw/how-to-write-a-memory-profiler</a></li> </ul> </li> <li>MALT &amp; NUMAPROF: Memory Profiling for HPC Applications <ul> <li>NUMAPROF: a NUMA memory profiler based on Pintool to track remote memory accesses <ul> <li><a href="https://memtt.github.io/numaprof">https://memtt.github.io/numaprof</a></li> <li><a href="https://github.com/memtt/numaprof">https://github.com/memtt/numaprof</a></li> </ul> </li> <li>MALT: a MALloc Tracker to find where and how your made your memory allocations in C/C++/Fortran applications <ul> <li><a href="https://memtt.github.io/malt/">https://memtt.github.io/malt/</a></li> <li><a href="https://github.com/memtt/malt">https://github.com/memtt/malt</a></li> </ul> </li> <li>MALT: A Malloc Tracker <ul> <li>International Workshop on Software Engineering for Parallel Systems (SEPS) 2017</li> <li>Sébastien Valat, Andres S. Charif-Rubial, William Jalby</li> <li>paper: <a href="https://memtt.github.io/malt/downloads/2017-seps-malt.pdf">https://memtt.github.io/malt/downloads/2017-seps-malt.pdf</a></li> <li>slides: <a href="https://svalat.github.io/docs/2017-10-MALT-SEPS17.pdf">https://svalat.github.io/docs/2017-10-MALT-SEPS17.pdf</a></li> </ul> </li> <li>FOSDEM 2019; Sébastien Valat <ul> <li><a href="https://www.youtube.com/watch?v=TnMOjdIy_Ow">https://www.youtube.com/watch?v=TnMOjdIy_Ow</a></li> <li><a href="https://fosdem.org/2019/schedule/event/numaprof/">https://fosdem.org/2019/schedule/event/numaprof/</a></li> </ul> </li> </ul> </li> <li>Memoro: A Detailed Heap Profiler <ul> <li><a href="https://epfl-vlsc.github.io/memoro/">https://epfl-vlsc.github.io/memoro/</a></li> <li><a href="https://github.com/epfl-vlsc/memoro">https://github.com/epfl-vlsc/memoro</a></li> <li>Detailed Heap Profiling <ul> <li>International Symposium on Memory Management (ISMM) 2018</li> <li>Stuart Byma, Jim Larus</li> <li><a href="https://dl.acm.org/citation.cfm?id=3210564">https://dl.acm.org/citation.cfm?id=3210564</a></li> </ul> </li> <li>Memoro: Scaling an LLVM-based Heap profiler <ul> <li>2019 LLVM Developers’ Meeting; Thierry Treyer</li> <li><a href="https://www.youtube.com/watch?v=fm47XsATelI">https://www.youtube.com/watch?v=fm47XsATelI</a></li> <li><a href="https://llvm.org/devmtg/2019-10/slides/Treyer-Memoro.pdf">https://llvm.org/devmtg/2019-10/slides/Treyer-Memoro.pdf</a></li> </ul> </li> </ul> </li> <li>memory-profiler: A memory profiler for Linux <ul> <li><a href="https://github.com/nokia/memory-profiler">https://github.com/nokia/memory-profiler</a></li> </ul> </li> <li>memtrail: A LD_PRELOAD based memory profiler and leak detector for Linux <ul> <li>​<a href="https://github.com/jrfonseca/memtrail">https://github.com/jrfonseca/memtrail</a></li> </ul> </li> <li>memusage - profile memory usage of a program <ul> <li><a href="http://man7.org/linux/man-pages/man1/memusage.1.html">http://man7.org/linux/man-pages/man1/memusage.1.html</a></li> </ul> </li> <li>MTuner - a C/C++ memory profiler and memory leak finder for Windows, PlayStation 4, PlayStation 3, etc. <ul> <li><a href="https://milostosic.github.io/MTuner/">https://milostosic.github.io/MTuner/</a></li> <li><a href="https://github.com/milostosic/MTuner">https://github.com/milostosic/MTuner</a></li> </ul> </li> <li>PerfMemPlus: A Tool for Automatic Discovery of Memory Performance Problems <ul> <li>Tool for memory performance analysis based on Linux perf.</li> <li><a href="https://github.com/helchr/perfMemPlus">https://github.com/helchr/perfMemPlus</a></li> <li>PerfMemPlus: A Tool for Automatic Discovery of Memory Performance Problems <ul> <li>ISC 2019</li> <li>Christian Helm, Kenjiro Taura</li> <li><a href="https://doi.org/10.1007/978-3-030-20656-7_11">https://doi.org/10.1007/978-3-030-20656-7_11</a></li> </ul> </li> <li>On the Correct Measurement of Application Memory Bandwidth and Memory Access Latency <ul> <li>HPC Asia 2020</li> <li>Christian Helm, Kenjiro Taura</li> <li><a href="https://doi.org/10.1145/3368474.3368476">https://doi.org/10.1145/3368474.3368476</a></li> </ul> </li> </ul> </li> <li>Typegrind <ul> <li>a type preserving heap profiler for C++ - collects memory allocation information with type information</li> <li><a href="https://typegrind.github.io/">https://typegrind.github.io/</a></li> <li><a href="https://github.com/typegrind/typegrind">https://github.com/typegrind/typegrind</a></li> </ul> </li> <li>Valgrind <ul> <li><a href="http://valgrind.org/">http://valgrind.org/</a></li> <li>DHAT: a dynamic heap analysis tool - <a href="http://valgrind.org/docs/manual/dh-manual.html">http://valgrind.org/docs/manual/dh-manual.html</a></li> <li>Massif: a heap profiler - ​<a href="http://valgrind.org/docs/manual/ms-manual.html">http://valgrind.org/docs/manual/ms-manual.html</a></li> </ul> </li> </ul> <h1>Microarchitecture</h1> <ul> <li>Tools for microarchitectural benchmarking <ul> <li><a href="https://dendibakh.github.io/blog/2018/04/03/Tools-for-microarchitectural-benchmarking">https://dendibakh.github.io/blog/2018/04/03/Tools-for-microarchitectural-benchmarking</a></li> </ul> </li> <li>asmbench: A Benchmark Toolkit for Assembly Instructions Using the LLVM JIT <ul> <li><a href="https://github.com/RRZE-HPC/asmbench">https://github.com/RRZE-HPC/asmbench</a></li> <li>OoO Instruction Benchmarking Framework on the Back of Dragons <ul> <li>2018 SC18 ACM SRC Poster</li> <li>J. Hammer, G. Hager, G. Wellein</li> <li><a href="https://sc18.supercomputing.org/proceedings/src_poster/src_poster_pages/spost115.html">https://sc18.supercomputing.org/proceedings/src_poster/src_poster_pages/spost115.html</a></li> </ul> </li> </ul> </li> <li>BHive: A Benchmark Suite and Measurement Framework for Validating x86-64 Basic Block Performance Models <ul> <li>IISWC 2019</li> <li>Yishen Chen, Ajay Brahmakshatriya, Charith Mendis, Alex Renda, Eric Atkinson, Ondrej Sykora, Saman Amarasinghe, Michael Carbin</li> <li><a href="http://groups.csail.mit.edu/commit/papers/19/ithemal-measurement.pdf">http://groups.csail.mit.edu/commit/papers/19/ithemal-measurement.pdf</a></li> <li><a href="https://github.com/ithemal/bhive">https://github.com/ithemal/bhive</a></li> </ul> </li> <li>Intel Architecture Code Analyzer (IACA) <ul> <li><a href="https://software.intel.com/en-us/articles/intel-architecture-code-analyzer">https://software.intel.com/en-us/articles/intel-architecture-code-analyzer</a></li> </ul> </li> <li>ibench: Measure instruction latency and throughput <ul> <li><a href="https://github.com/hofm/ibench">https://github.com/hofm/ibench</a></li> </ul> </li> <li>Ithemal: Instruction THroughput Estimator using MAchine Learning <ul> <li><a href="https://github.com/psg-mit/Ithemal">https://github.com/psg-mit/Ithemal</a></li> <li>Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks <ul> <li>ICML 2019</li> <li>Charith Mendis, Alex Renda, Saman Amarasinghe, Michael Carbin</li> <li><a href="https://arxiv.org/abs/1808.07412">https://arxiv.org/abs/1808.07412</a></li> <li><a href="http://proceedings.mlr.press/v97/mendis19a.html">http://proceedings.mlr.press/v97/mendis19a.html</a></li> </ul> </li> </ul> </li> <li>llvm-exegesis – LLVM Machine Instruction Benchmark <ul> <li><a href="https://llvm.org/docs/CommandGuide/llvm-exegesis.html">https://llvm.org/docs/CommandGuide/llvm-exegesis.html</a></li> <li><a href="https://github.com/llvm-mirror/llvm/tree/master/tools/llvm-exegesis">https://github.com/llvm-mirror/llvm/tree/master/tools/llvm-exegesis</a></li> <li>Static Performance Analysis with LLVM <ul> <li>2018 European LLVM Developers Meeting</li> <li>C. Courbet, O. Sykora, G. Chatelet, B. De Backer</li> <li><a href="https://youtu.be/XinMk-t8N-w">https://youtu.be/XinMk-t8N-w</a></li> <li><a href="http://llvm.org/devmtg/2018-04/slides/Courbet-Static%20Performance%20Analysis%20with%20LLVM.pdf">http://llvm.org/devmtg/2018-04/slides/Courbet-Static%20Performance%20Analysis%20with%20LLVM.pdf</a></li> </ul> </li> <li>Measuring x86 instruction latencies with LLVM <ul> <li>2018 European LLVM Developers Meeting</li> <li>G. Chatelet, C. Courbet, B. De Backer, O. Sykora</li> <li><a href="https://youtu.be/ex_C27OoApI">https://youtu.be/ex_C27OoApI</a></li> <li><a href="http://llvm.org/devmtg/2018-04/slides/Chatelet-Measuring%20x86%20instruction%20latencies%20with%20LLVM.pdf">http://llvm.org/devmtg/2018-04/slides/Chatelet-Measuring%20x86%20instruction%20latencies%20with%20LLVM.pdf</a></li> </ul> </li> </ul> </li> <li>llvm-mca - LLVM Machine Code Analyzer <ul> <li><a href="https://llvm.org/docs/CommandGuide/llvm-mca.html">https://llvm.org/docs/CommandGuide/llvm-mca.html</a></li> <li><a href="https://github.com/llvm-mirror/llvm/tree/master/tools/llvm-mca">https://github.com/llvm-mirror/llvm/tree/master/tools/llvm-mca</a></li> <li>Understanding the performance of code using LLVM's Machine Code Analyzer (llvm-mca) <ul> <li>2018 LLVM Developers’ Meeting; Andrea Di Biagio &amp; Matt Davis</li> <li><a href="https://www.youtube.com/watch?v=Ku2D8bjEGXk">https://www.youtube.com/watch?v=Ku2D8bjEGXk</a></li> </ul> </li> <li>MC Ruler: Seamless llvm-mca CMake integration <ul> <li><a href="https://github.com/jeremyong/mc_ruler">https://github.com/jeremyong/mc_ruler</a></li> </ul> </li> </ul> </li> <li>nanoBench: A tool for running small microbenchmarks on recent Intel and AMD x86 CPUs <ul> <li>used for running the microbenchmarks for obtaining the latency, throughput, and port usage data available on <a href="http://uops.info">http://uops.info</a></li> <li><a href="https://github.com/andreas-abel/nanoBench">https://github.com/andreas-abel/nanoBench</a></li> <li><a href="https://uops.info/">https://uops.info/</a></li> <li>nanoBench Cache Analyzer <ul> <li><a href="https://github.com/andreas-abel/nanoBench/tree/master/tools/CacheAnalyzer">https://github.com/andreas-abel/nanoBench/tree/master/tools/CacheAnalyzer</a></li> <li><a href="https://uops.info/cache.html">https://uops.info/cache.html</a></li> </ul> </li> <li>uops.info: Characterizing Latency, Throughput, and Port Usage of Instructions on Intel Microarchitectures <ul> <li>ASPLOS 2019</li> <li>Andreas Abel, Jan Reineke</li> <li><a href="https://arxiv.org/abs/1810.04610">https://arxiv.org/abs/1810.04610</a></li> </ul> </li> <li>nanoBench: A Low-Overhead Tool for Running Microbenchmarks on x86 Systems <ul> <li>2019; Andreas Abel, Jan Reineke</li> <li><a href="https://arxiv.org/abs/1911.03282">https://arxiv.org/abs/1911.03282</a></li> </ul> </li> </ul> </li> <li>Open Power/Performance Analysis Tool (OPPAT) <ul> <li>a cross-OS, cross-architecture Power and Performance Analysis Tool</li> <li>cross-OS: supports Windows ETW trace files and Linux/Android perf/trace-cmd trace files</li> <li>cross-architecture: supports Intel and ARM chips hardware events (using perf and/or PCM)</li> <li><a href="https://patinnc.github.io/">https://patinnc.github.io/</a></li> <li><a href="https://github.com/patinnc/oppat">https://github.com/patinnc/oppat</a></li> </ul> </li> <li>OSACA: Open Source Architecture Code Analyzer <ul> <li><a href="https://github.com/RRZE-HPC/osaca">https://github.com/RRZE-HPC/osaca</a></li> <li><a href="https://hpc.fau.de/research/tools/">https://hpc.fau.de/research/tools/</a></li> <li>Automated Instruction Stream Throughput Prediction for Intel and AMD Microarchitectures <ul> <li>Performance Modeling, Benchmarking and Simulation of High Performance Computer Systems (PMBS) 2018</li> <li>Jan Laukemann, Julian Hammer, Johannes Hofmann, Georg Hager, Gerhard Wellein</li> <li><a href="https://arxiv.org/abs/1809.00912">https://arxiv.org/abs/1809.00912</a></li> </ul> </li> <li>Automatic Throughput and Critical Path Analysis of x86 and ARM Assembly Kernels <ul> <li>arXiv 2019</li> <li>Jan Laukemann, Julian Hammer, Georg Hager, Gerhard Wellein</li> <li><a href="https://arxiv.org/abs/1910.00214">https://arxiv.org/abs/1910.00214</a></li> <li><a href="https://github.com/RRZE-HPC/OSACA-CP-2019">https://github.com/RRZE-HPC/OSACA-CP-2019</a></li> </ul> </li> <li>Cross-Architecture Automatic Critical Path Detection For In-Core Performance Analysis <ul> <li>2020 Master Thesis; Jan Laukemann</li> <li><a href="https://hpc.fau.de/files/2020/02/Masterarbeit_JL-_final.pdf">https://hpc.fau.de/files/2020/02/Masterarbeit_JL-_final.pdf</a></li> <li>Measurements and reproducibility instructions <ul> <li><a href="https://github.com/RRZE-HPC/OSACA-Artifact-Appendix">https://github.com/RRZE-HPC/OSACA-Artifact-Appendix</a></li> </ul> </li> </ul> </li> </ul> </li> <li>PMEvo: Portable Inference of Port Mappings for Out-of-Order Processors by Evolutionary Optimization <ul> <li>PLDI 2020</li> <li>Fabian Ritter, Sebastian Hack</li> <li><a href="https://compilers.cs.uni-saarland.de/papers/ritter_pmevo_pldi20.pdf">https://compilers.cs.uni-saarland.de/papers/ritter_pmevo_pldi20.pdf</a></li> <li><a href="https://compilers.cs.uni-saarland.de/projects/portmap/">https://compilers.cs.uni-saarland.de/projects/portmap/</a></li> <li><a href="https://github.com/cdl-saarland/pmevo-artifact">https://github.com/cdl-saarland/pmevo-artifact</a></li> </ul> </li> <li>timing-harness: Harness for profiling arbitrary basic blocks. <ul> <li><a href="https://github.com/ithemal/timing-harness">https://github.com/ithemal/timing-harness</a></li> </ul> </li> <li>uarch-bench: A benchmark for low-level CPU micro-architectural features <ul> <li><a href="https://github.com/travisdowns/uarch-bench">https://github.com/travisdowns/uarch-bench</a></li> </ul> </li> </ul> <h1>Optimization</h1> <ul> <li>BOLT: Binary Optimization and Layout Tool <ul> <li>A linux command-line utility used for optimizing performance of binaries</li> <li><a href="https://github.com/facebookincubator/BOLT">https://github.com/facebookincubator/BOLT</a></li> <li>Accelerate large-scale applications with BOLT <ul> <li><a href="https://code.fb.com/data-infrastructure/accelerate-large-scale-applications-with-bolt/">https://code.fb.com/data-infrastructure/accelerate-large-scale-applications-with-bolt/</a></li> </ul> </li> <li>Building Binary Optimizer with LLVM <ul> <li>2016 EuroLLVM Developers' Meeting; Maksim Panchenko</li> <li><a href="https://llvm.org/devmtg/2016-03/Presentations/BOLT_EuroLLVM_2016.pdf">https://llvm.org/devmtg/2016-03/Presentations/BOLT_EuroLLVM_2016.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=gw3iDO3By5Y">https://www.youtube.com/watch?v=gw3iDO3By5Y</a></li> </ul> </li> <li>BOLT: A Practical Binary Optimizer for Data Centers and Beyond <ul> <li>Maksim Panchenko, Rafael Auler, Bill Nell, Guilherme Ottoni</li> <li><a href="https://arxiv.org/abs/1807.06735">https://arxiv.org/abs/1807.06735</a></li> </ul> </li> </ul> </li> <li>MAQAO (Modular Assembly Quality Analyzer and Optimizer) <ul> <li><a href="http://www.maqao.org/">http://www.maqao.org/</a></li> <li><a href="http://maqao.bordeaux.inria.fr/">http://maqao.bordeaux.inria.fr/</a></li> </ul> </li> </ul> <h1>Profiling</h1> <ul> <li>Agner Fog's test programs for measuring clock cycles and performance monitoring <ul> <li><a href="http://www.agner.org/optimize/#testp">http://www.agner.org/optimize/#testp</a></li> </ul> </li> <li>BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more <ul> <li><a href="https://iovisor.github.io/bcc/">https://iovisor.github.io/bcc/</a></li> <li><a href="https://github.com/iovisor/bcc">https://github.com/iovisor/bcc</a></li> <li><a href="https://github.com/iovisor/bpf-docs">https://github.com/iovisor/bpf-docs</a></li> <li><a href="http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.html">http://www.brendangregg.com/blog/2016-03-05/linux-bpf-superpowers.html</a></li> <li><a href="http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-ahead-2016.html">http://www.brendangregg.com/blog/2016-03-28/linux-bpf-bcc-road-ahead-2016.html</a></li> <li><a href="http://www.brendangregg.com/blog/2016-06-14/ubuntu-xenial-bcc-bpf.html">http://www.brendangregg.com/blog/2016-06-14/ubuntu-xenial-bcc-bpf.html</a></li> <li><a href="https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/">https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/</a></li> </ul> </li> <li>Caliper: A Performance Analysis Toolbox in a Library <ul> <li>an instrumentation and performance profiling library</li> <li><a href="https://github.com/LLNL/Caliper">https://github.com/LLNL/Caliper</a></li> </ul> </li> <li>Coz: Finding Code that Counts with Causal Profiling <ul> <li><a href="https://github.com/plasma-umass/coz/">https://github.com/plasma-umass/coz/</a></li> <li>Charlie Curtsinger, Emery Berger</li> <li>SOSP 2015 <ul> <li><a href="https://www.sigops.org/sosp/sosp15/current/2015-Monterey/printable/090-curtsinger.pdf">https://www.sigops.org/sosp/sosp15/current/2015-Monterey/printable/090-curtsinger.pdf</a></li> </ul> </li> <li>;login: 41(2) (2016) <ul> <li><a href="https://www.usenix.org/publications/login/summer2016/curtsinger">https://www.usenix.org/publications/login/summer2016/curtsinger</a></li> </ul> </li> <li>Performance Matters - Strange Loop 2019; Emery Berger <ul> <li><a href="https://www.youtube.com/watch?v=r-TLSBdHe1A">https://www.youtube.com/watch?v=r-TLSBdHe1A</a></li> </ul> </li> <li>Coz vs. Sampling Profilers <ul> <li><a href="https://easyperf.net/blog/2020/02/26/coz-vs-sampling-profilers">https://easyperf.net/blog/2020/02/26/coz-vs-sampling-profilers</a></li> </ul> </li> </ul> </li> <li>easy_profiler: Lightweight cross-platform profiler library for C++ <ul> <li><a href="https://github.com/yse/easy_profiler">https://github.com/yse/easy_profiler</a></li> </ul> </li> <li>Event Tracing for Windows (ETW) / Windows Performance Toolkit – Xperf <ul> <li><a href="https://blogs.msdn.microsoft.com/ntdebugging/2008/04/03/windows-performance-toolkit-xperf/">https://blogs.msdn.microsoft.com/ntdebugging/2008/04/03/windows-performance-toolkit-xperf/</a></li> <li>ETW Central: <a href="https://randomascii.wordpress.com/2015/09/24/etw-central/">https://randomascii.wordpress.com/2015/09/24/etw-central/</a></li> </ul> </li> <li>gperftools (originally Google Performance Tools) <ul> <li>"The fastest malloc we’ve seen; works particularly well with threads and STL. Also: thread-friendly heap-checker, heap-profiler, and cpu-profiler."</li> <li><a href="https://github.com/gperftools/gperftools">https://github.com/gperftools/gperftools</a></li> </ul> </li> <li>gprof2dot <ul> <li>"Python script to convert the output from many profilers into a dot graph."</li> <li><a href="https://github.com/jrfonseca/gprof2dot">https://github.com/jrfonseca/gprof2dot</a></li> </ul> </li> <li>HawkTracer <ul> <li>a highly portable, low-overhead, configurable profiling tool for getting performance metrics from low-end devices</li> <li>Linux, Windows, macOS; C &amp; C++ library; Python &amp; Rust wrappers</li> <li><a href="https://www.hawktracer.org/">https://www.hawktracer.org/</a></li> <li><a href="https://github.com/amzn/hawktracer">https://github.com/amzn/hawktracer</a></li> <li>Low-end platform profiling with HawkTracer profiler <ul> <li>FOSDEM 2020; Marcin Kolny</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_hawktrace/">https://fosdem.org/2020/schedule/event/debugging_hawktrace/</a></li> </ul> </li> </ul> </li> <li>Hotspot - the Linux perf GUI for performance analysis <ul> <li><a href="https://www.kdab.com/hotspot-gui-linux-perf-profiler/">https://www.kdab.com/hotspot-gui-linux-perf-profiler/</a></li> <li><a href="https://github.com/KDAB/hotspot">https://github.com/KDAB/hotspot</a></li> <li><a href="https://www.kdab.com/hotspot-video/">https://www.kdab.com/hotspot-video/</a></li> </ul> </li> <li>Likwid: Performance monitoring and benchmarking suite <ul> <li><a href="https://github.com/RRZE-HPC/likwid">https://github.com/RRZE-HPC/likwid</a></li> <li><a href="https://github.com/RRZE-HPC/likwid/wiki">https://github.com/RRZE-HPC/likwid/wiki</a></li> <li><a href="https://github.com/RRZE-HPC/likwid/wiki/TutorialStart">https://github.com/RRZE-HPC/likwid/wiki/TutorialStart</a></li> <li><a href="https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr">https://github.com/RRZE-HPC/likwid/wiki/likwid-perfctr</a></li> <li><a href="https://github.com/RRZE-HPC/likwid/wiki/TutorialMarkerC">https://github.com/RRZE-HPC/likwid/wiki/TutorialMarkerC</a></li> <li><a href="https://github.com/RRZE-HPC/likwid/wiki/PatternsHaswellEP">https://github.com/RRZE-HPC/likwid/wiki/PatternsHaswellEP</a></li> </ul> </li> <li>microprofile: an embeddable profiler <ul> <li><a href="https://github.com/jonasmr/microprofile">https://github.com/jonasmr/microprofile</a></li> </ul> </li> <li>Optick: C++ Profiler For Games <ul> <li><a href="https://github.com/bombomby/optick">https://github.com/bombomby/optick</a></li> <li><a href="https://optick.dev/">https://optick.dev/</a></li> </ul> </li> <li>perf <ul> <li><a href="https://perf.wiki.kernel.org/">https://perf.wiki.kernel.org/</a></li> <li><a href="http://www.brendangregg.com/perf.html">http://www.brendangregg.com/perf.html</a></li> <li><a href="https://github.com/torvalds/linux/tree/master/tools/perf/Documentation">https://github.com/torvalds/linux/tree/master/tools/perf/Documentation</a></li> <li><a href="https://suchakra.wordpress.com/2016/06/17/deconstructing-perfs-data-file/">https://suchakra.wordpress.com/2016/06/17/deconstructing-perfs-data-file/</a></li> <li><a href="http://www.brendangregg.com/linuxperf.html">http://www.brendangregg.com/linuxperf.html</a></li> </ul> </li> <li>perf-tools - <a href="https://github.com/brendangregg/perf-tools">https://github.com/brendangregg/perf-tools</a></li> <li>perf_events: The Unofficial Linux Perf Events Web-Page <ul> <li><a href="http://web.eece.maine.edu/~vweaver/projects/perf_events/">http://web.eece.maine.edu/~vweaver/projects/perf_events/</a></li> </ul> </li> <li>perfmon2 - <a href="http://perfmon2.sourceforge.net/">http://perfmon2.sourceforge.net/</a> <ul> <li>"Perfmon2 aims to be a portable interface across all modern processors. It is designed to give full access to a given PMU and all the corresponding hardware performance counters. Typically the PMU hardware implementations use a different number of registers, counters with different length and possibly other unique features, a complexity that the software has to cope with. Although processors have different PMU implementations, they usually use configurations registers and data registers. Perfmon2 provides a uniform abstract model of these registers and exports read/write operations accordingly."</li> </ul> </li> <li>Performance Application Programming Interface (PAPI) <ul> <li><a href="http://icl.cs.utk.edu/papi/">http://icl.cs.utk.edu/papi/</a></li> <li><a href="http://icl.cs.utk.edu/projects/papi/wiki/Main_Page">http://icl.cs.utk.edu/projects/papi/wiki/Main_Page</a></li> <li><a href="http://www.drdobbs.com/tools/performance-monitoring-with-papi/184406109">http://www.drdobbs.com/tools/performance-monitoring-with-papi/184406109</a></li> <li>papi-wrapper (C++ library) - <a href="https://github.com/sean-chester/papi-wrapper">https://github.com/sean-chester/papi-wrapper</a></li> <li>libpapipp: A C++ wrapper around libpapi - <a href="https://github.com/david-grs/papipp">https://github.com/david-grs/papipp</a></li> </ul> </li> <li>pmu tools: Intel PMU profiling tools <ul> <li><a href="https://github.com/andikleen/pmu-tools">https://github.com/andikleen/pmu-tools</a></li> <li><a href="https://github.com/andikleen/pmu-tools/wiki/toplev-manual">https://github.com/andikleen/pmu-tools/wiki/toplev-manual</a></li> <li>pmu-tools part I - introduction, ocperf - <a href="http://halobates.de/blog/p/245">http://halobates.de/blog/p/245</a></li> <li>pmu-tools part II - toplev - <a href="http://halobates.de/blog/p/262">http://halobates.de/blog/p/262</a></li> </ul> </li> <li>Processor Counter Monitor (PCM) <ul> <li><a href="https://github.com/opcm/pcm">https://github.com/opcm/pcm</a></li> <li>Intel Performance Counter Monitor (PCM) - <a href="http://www.intel.com/software/pcm">http://www.intel.com/software/pcm</a></li> </ul> </li> <li>Remotery: Single C file, Realtime CPU/GPU Profiler with Remote Web Viewer <ul> <li><a href="https://github.com/Celtoys/Remotery">https://github.com/Celtoys/Remotery</a></li> </ul> </li> <li>sysdig <ul> <li><a href="https://github.com/draios/sysdig">https://github.com/draios/sysdig</a></li> <li><a href="https://sysdig.com/blog/50-shades-of-system-calls/">https://sysdig.com/blog/50-shades-of-system-calls/</a></li> </ul> </li> <li>Tracy Profiler <ul> <li>Tracy is a real time, nanosecond resolution frame profiler that can be used for remote or embedded telemetry of your application. It can profile CPU (C++, Lua), GPU (OpenGL, Vulkan) and memory. It also can display locks held by threads and their interactions with each other.</li> <li><a href="https://bitbucket.org/wolfpld/tracy">https://bitbucket.org/wolfpld/tracy</a></li> <li>Introduction to the Tracy profiler - <a href="https://www.youtube.com/watch?v=fB5B46lbapc">https://www.youtube.com/watch?v=fB5B46lbapc</a></li> </ul> </li> </ul> <h1>Timing</h1> <ul> <li>low-overhead-timers: Very low-overhead timer/counter interfaces for C on Intel 64 processors <ul> <li><a href="https://github.com/jdmccalpin/low-overhead-timers">https://github.com/jdmccalpin/low-overhead-timers</a></li> <li>Comments on timing short code sections on Intel processors <ul> <li><a href="http://sites.utexas.edu/jdm4372/2018/07/23/comments-on-timing-short-code-sections-on-intel-processors/">http://sites.utexas.edu/jdm4372/2018/07/23/comments-on-timing-short-code-sections-on-intel-processors/</a></li> </ul> </li> </ul> </li> </ul> <h1>Visualization</h1> <ul> <li>Flame Graphs <ul> <li><a href="http://www.brendangregg.com/flamegraphs.html">http://www.brendangregg.com/flamegraphs.html</a></li> <li><a href="http://queue.acm.org/detail.cfm?id=2927301">http://queue.acm.org/detail.cfm?id=2927301</a></li> <li>Memory Leak (and Growth) Flame Graphs <ul> <li><a href="http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html">http://www.brendangregg.com/FlameGraphs/memoryflamegraphs.html</a></li> </ul> </li> </ul> </li> <li>FlameScope: a visualization tool for exploring different time ranges as Flame Graphs <ul> <li><a href="https://github.com/Netflix/flamescope">https://github.com/Netflix/flamescope</a></li> <li>Netflix FlameScope - <a href="https://medium.com/@NetflixTechBlog/netflix-flamescope-a57ca19d47bb">https://medium.com/@NetflixTechBlog/netflix-flamescope-a57ca19d47bb</a></li> </ul> </li> <li>pprof - a tool for visualization and analysis of profiling data <ul> <li><a href="https://github.com/google/pprof">https://github.com/google/pprof</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/pipedream/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pipedream/index.html <p><a href="https://pipedream.com/">Website</a> | <a href="https://github.com/PipedreamHQ/pipedream">Source</a></p> https://tedneward.github.io/Research/tools/obsidian/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/obsidian/index.html <p><a href="https://obsidian.md/">Website</a> | Commercial | <a href="https://github.com/obsidianmd">Releases</a></p> <h2>Resources</h2> <h3>Articles, Blogs, Essays</h3> <h3>Source/Repositories</h3> <ul> <li><a href="https://github.com/guyroyse/openclaw-obsidian-desktop">Docker setup for running OpenClaw and Obsidian together in a browser-accessible virtual desktop, with an OpenClaw skill for interacting with your vault via the Obsidian CLI.</a></li> </ul> https://tedneward.github.io/Research/tools/onecommander/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/onecommander/index.html <p><a href="https://www.onecommander.com/">Website</a></p> https://tedneward.github.io/Research/tools/opennotebook/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/opennotebook/index.html <p><a href="https://www.open-notebook.ai/">Website</a> | <a href="https://github.com/lfnovo/open-notebook">Source</a></p> <p>To run:</p> <pre><code>docker run -d \ --name open-notebook \ -p 8502:8502 -p 5055:5055 \ -v ./notebook_data:/app/data \ -v ./surreal_data:/mydata \ -e OPENAI_API_KEY=your_key \ lfnovo/open_notebook:v1-latest-single </code></pre> <ul> <li><code>-d</code> runs the container in detached mode</li> <li><code>--name open-notebook</code> names the container for easy reference</li> <li><code>-p 8502:8502 -p 5055:5055</code> maps ports for the web interface and API access</li> <li><code>-v ./notebook_data:/app/data</code> and <code>-v ./surreal_data:/mydata</code> mount local folders to persist notes and database files. This ensures that your data is stored on your machine and remains intact even if the container is restarted</li> <li><code>-e OPENAI_API_KEY=your_key</code> allows integration with OpenAI models if desired</li> <li><code>lfnovo/open_notebook:v1-latest-single</code> specifies the container image</li> </ul> <p>After running the container, navigate to:</p> <ul> <li>Main Interface: <a href="http://localhost:8502">http://localhost:8502</a></li> <li>API Access: <a href="http://localhost:5055">http://localhost:5055</a></li> <li>API Documentation: <a href="http://localhost:5055/docs">http://localhost:5055/docs</a></li> </ul> <p>After deployment, two folders appear in the local directory:</p> <ul> <li>notebook_data: Stores all notes, summaries, and AI-processed content</li> <li>surreal_data: Contains underlying database files for Open Notebook's internal storage</li> </ul> <hr> <pre><code>docker run -d \ --name open-notebook \ -p 8502:8502 -p 5055:5055 \ -v ./notebook_data:/app/data \ -v ./surreal_data:/mydata \ lfnovo/open_notebook:v1-latest-single </code></pre> <p>Docker compose</p> <pre><code>services: surrealdb: image: surrealdb/surrealdb:v2 command: start --user root --pass password --bind 0.0.0.0:8000 rocksdb:/mydata/mydatabase.db ports: - "8000:8000" volumes: - ./surreal_data:/mydata open_notebook: image: lfnovo/open_notebook:v1-latest-single pull_policy: always ports: - "8502:8502" # Web UI (React frontend) - "5055:5055" # API (required!) environment: # NO API KEYS NEEDED - Using Ollama (free, local) - OLLAMA_API_BASE=http://ollama:11434 # Database (required) - SURREAL_URL=ws://surrealdb:8000/rpc - SURREAL_USER=root - SURREAL_PASSWORD=password - SURREAL_NAMESPACE=open_notebook - SURREAL_DATABASE=open_notebook volumes: - ./notebook_data:/app/data - ./surreal_data:/mydata depends_on: - surrealdb restart: always ollama: image: ollama/ollama:latest ports: - "11434:11434" volumes: - ./ollama_models:/root/.ollama environment: # Optional: set GPU support if available - OLLAMA_NUM_GPU=0 restart: always </code></pre> <p>NOTE: If you want to run Ollama locally (to more easily use GPUs, I think), make sure <code>OLLAMA_API_BASE=http://host.docker.internal:11434</code></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://dev.to/theairabbit/testing-open-notebook-a-complete-walkthrough-with-gemini-ai-4ggj">https://dev.to/theairabbit/testing-open-notebook-a-complete-walkthrough-with-gemini-ai-4ggj</a></li> </ul> https://tedneward.github.io/Research/tools/openwebui/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/openwebui/index.html <p><a href="https://openwebui.com/">Website</a> | <a href="https://github.com/open-webui/open-webui">Source</a> | <a href="https://docs.openwebui.com/features">Docs</a></p> <h2>Key Features of Open WebUI ⭐</h2> <ul> <li> <p>Install using Docker or Kubernetes (kubectl, kustomize or helm) with support for both <code>:ollama</code> and <code>:cuda</code> tagged images.</p> </li> <li> <p>Ollama/OpenAI API Integration: Integrate OpenAI-compatible APIs for conversations alongside Ollama models. Customize the OpenAI API URL to link with LMStudio, GroqCloud, Mistral, OpenRouter, and more.</p> </li> <li> <p>Granular Permissions and User Groups: By allowing administrators to create detailed user roles and permissions, we ensure a secure user environment. This granularity not only enhances security but also allows for customized user experiences, fostering a sense of ownership and responsibility amongst users.</p> </li> <li> <p>Responsive Design: Enjoy a seamless experience across Desktop PC, Laptop, and Mobile devices.</p> </li> <li> <p>Progressive Web App (PWA) for Mobile: Enjoy a native app-like experience on your mobile device with our PWA, providing offline access on localhost and a seamless user interface.</p> </li> <li> <p>Full Markdown and LaTeX Support: Elevate your LLM experience with comprehensive Markdown and LaTeX capabilities for enriched interaction.</p> </li> <li> <p>Hands-Free Voice/Video Call: Experience seamless communication with integrated hands-free voice and video call features using multiple Speech-to-Text providers (Local Whisper, OpenAI, Deepgram, Azure) and Text-to-Speech engines (Azure, ElevenLabs, OpenAI, Transformers, WebAPI), allowing for dynamic and interactive chat environments.</p> </li> <li> <p>Model Builder: Easily create Ollama models via the Web UI. Create and add custom characters/agents, customize chat elements, and import models effortlessly through <a href="https://openwebui.com/">Open WebUI Community</a> integration.</p> </li> <li> <p>Native Python Function Calling Tool: Enhance your LLMs with built-in code editor support in the tools workspace. Bring Your Own Function (BYOF) by simply adding your pure Python functions, enabling seamless integration with LLMs.</p> </li> <li> <p>Persistent Artifact Storage: Built-in key-value storage API for artifacts, enabling features like journals, trackers, leaderboards, and collaborative tools with both personal and shared data scopes across sessions.</p> </li> <li> <p>Local RAG Integration: Dive into the future of chat interactions with groundbreaking Retrieval Augmented Generation (RAG) support using your choice of 9 vector databases and multiple content extraction engines (Tika, Docling, Document Intelligence, Mistral OCR, External loaders). Load documents directly into chat or add files to your document library, effortlessly accessing them using the <code>#</code> command before a query.</p> </li> <li> <p>Web Search for RAG: Perform web searches using 15+ providers including <code>SearXNG</code>, <code>Google PSE</code>, <code>Brave Search</code>, <code>Kagi</code>, <code>Mojeek</code>, <code>Tavily</code>, <code>Perplexity</code>, <code>serpstack</code>, <code>serper</code>, <code>Serply</code>, <code>DuckDuckGo</code>, <code>SearchApi</code>, <code>SerpApi</code>, <code>Bing</code>, <code>Jina</code>, <code>Exa</code>, <code>Sougou</code>, <code>Azure AI Search</code>, and <code>Ollama Cloud</code>, injecting results directly into your chat experience.</p> </li> <li> <p>Web Browsing Capability: Seamlessly integrate websites into your chat experience using the <code>#</code> command followed by a URL. This feature allows you to incorporate web content directly into your conversations, enhancing the richness and depth of your interactions.</p> </li> <li> <p>Image Generation &amp; Editing Integration: Create and edit images using multiple engines including OpenAI's DALL-E, Gemini, ComfyUI (local), and AUTOMATIC1111 (local), with support for both generation and prompt-based editing workflows.</p> </li> <li> <p>Many Models Conversations: Effortlessly engage with various models simultaneously, harnessing their unique strengths for optimal responses. Enhance your experience by leveraging a diverse set of models in parallel.</p> </li> <li> <p>Role-Based Access Control (RBAC): Ensure secure access with restricted permissions; only authorized individuals can access your Ollama, and exclusive model creation/pulling rights are reserved for administrators.</p> </li> <li> <p>Flexible Database &amp; Storage Options: Choose from SQLite (with optional encryption), PostgreSQL, or configure cloud storage backends (S3, Google Cloud Storage, Azure Blob Storage) for scalable deployments.</p> </li> <li> <p>Advanced Vector Database Support: Select from 9 vector database options including ChromaDB, PGVector, Qdrant, Milvus, Elasticsearch, OpenSearch, Pinecone, S3Vector, and Oracle 23ai for optimal RAG performance.</p> </li> <li> <p>Enterprise Authentication: Full support for LDAP/Active Directory integration, SCIM 2.0 automated provisioning, and SSO via trusted headers alongside OAuth providers. Enterprise-grade user and group provisioning through SCIM 2.0 protocol, enabling seamless integration with identity providers like Okta, Azure AD, and Google Workspace for automated user lifecycle management.</p> </li> <li> <p>Cloud-Native Integration: Native support for Google Drive and OneDrive/SharePoint file picking, enabling seamless document import from enterprise cloud storage.</p> </li> <li> <p>Production Observability: Built-in OpenTelemetry support for traces, metrics, and logs, enabling comprehensive monitoring with your existing observability stack.</p> </li> <li> <p>Horizontal Scalability: Redis-backed session management and WebSocket support for multi-worker and multi-node deployments behind load balancers.</p> </li> <li> <p>Multilingual Support: Experience Open WebUI in your preferred language with our internationalization (i18n) support. Join us in expanding our supported languages! We're actively seeking contributors!</p> </li> <li> <p>Pipelines, Open WebUI Plugin Support: Seamlessly integrate custom logic and Python libraries into Open WebUI using <a href="https://github.com/open-webui/pipelines">Pipelines Plugin Framework</a>. Launch your Pipelines instance, set the OpenAI URL to the Pipelines URL, and explore endless possibilities. <a href="https://github.com/open-webui/pipelines/tree/main/examples">Examples</a> include <strong>Function Calling</strong>, User <strong>Rate Limiting</strong> to control access, <strong>Usage Monitoring</strong> with tools like Langfuse, <strong>Live Translation with LibreTranslate</strong> for multilingual support, <strong>Toxic Message Filtering</strong> and much more.</p> </li> </ul> <h2>Getting Started</h2> <h3>Installation via Python pip</h3> <p>Open WebUI can be installed using using <strong>Python 3.11</strong>.</p> <ol> <li><strong>Install Open WebUI</strong>:<br> Open your terminal and run the following command to install Open WebUI:</li> </ol> <pre><code class="language-bash">pip install open-webui </code></pre> <ol> <li><strong>Running Open WebUI</strong>:<br> After installation, you can start Open WebUI by executing:</li> </ol> <pre><code class="language-bash">open-webui serve </code></pre> <p>This will start the Open WebUI server, which you can access at <a href="http://localhost:8080">http://localhost:8080</a></p> <h3>Quick Start with Docker</h3> <p>Please note that for certain Docker environments, additional configurations might be needed.</p> <p>When using Docker to install Open WebUI, make sure to include the <code>-v open-webui:/app/backend/data</code> in your Docker command. This step is crucial as it ensures your database is properly mounted and prevents any loss of data.</p> <p>If you wish to utilize Open WebUI with Ollama included or CUDA acceleration, we recommend utilizing our official images tagged with either <code>:cuda</code> or <code>:ollama</code>. To enable CUDA, you must install the <a href="https://docs.nvidia.com/dgx/nvidia-container-runtime-upgrade/">Nvidia CUDA container toolkit</a> on your Linux/WSL system.</p> <h3>Installation with Default Configuration</h3> <ul> <li><strong>If Ollama is on your computer</strong>, use this command:</li> </ul> <pre><code class="language-bash">docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main </code></pre> <ul> <li><strong>If Ollama is on a Different Server</strong>, use this command:</li> </ul> <p>To connect to Ollama on another server, change the <code>OLLAMA_BASE_URL</code> to the server's URL:</p> <pre><code class="language-bash">docker run -d -p 3000:8080 -e OLLAMA_BASE_URL=https://example.com -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main </code></pre> <ul> <li><strong>To run Open WebUI with Nvidia GPU support</strong>, use this command:</li> </ul> <pre><code class="language-bash">docker run -d -p 3000:8080 --gpus all --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:cuda </code></pre> <ul> <li><strong>After experimentation</strong>, I ended up with:</li> </ul> <pre><code>docker run -d -p 3000:8080 --name=openweb-ui --restart=always --env=DOCKER=true --env=OLLAMA_BASE_URL=http://host.docker.internal:11434 --env=WEBUI_AUTH=False --env=WEBUI_SECRET_KEY= --volume=./open-webui:/app/backend/data --workdir=/app/backend ghcr.io/open-webui/open-webui:main </code></pre> <p>Research:<br> * What does "--add-host" do in Docker?</p> <h3>Installation for OpenAI API Usage Only</h3> <ul> <li><strong>If you're only using OpenAI API</strong>, use this command:</li> </ul> <pre><code class="language-bash">docker run -d -p 3000:8080 -e OPENAI_API_KEY=your_secret_key -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main </code></pre> <h3>Installing Open WebUI with Bundled Ollama Support</h3> <p>This installation method uses a single container image that bundles Open WebUI with Ollama, allowing for a streamlined setup via a single command. Choose the appropriate command based on your hardware setup:</p> <ul> <li><strong>With GPU Support</strong>:<br> Utilize GPU resources by running the following command:</li> </ul> <pre><code class="language-bash">docker run -d -p 3000:8080 --gpus=all -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama </code></pre> <ul> <li><strong>For CPU Only</strong>:<br> If you're not using a GPU, use this command instead:</li> </ul> <pre><code class="language-bash">docker run -d -p 3000:8080 -v ollama:/root/.ollama -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:ollama </code></pre> <p>Both commands facilitate a built-in, hassle-free installation of both Open WebUI and Ollama, ensuring that you can get everything up and running swiftly.</p> <p>After installation, you can access Open WebUI at <a href="http://localhost:3000">http://localhost:3000</a>.</p> <p>Look at the <a href="https://docs.openwebui.com/getting-started/advanced-topics/development">Local Development Guide</a> for instructions on setting up a local development environment.</p> <h3>Troubleshooting</h3> <h4>Open WebUI: Server Connection Error</h4> <p>If you're experiencing connection issues, it’s often due to the WebUI docker container not being able to reach the Ollama server at 127.0.0.1:11434 (host.docker.internal:11434) inside the container . Use the <code>--network=host</code> flag in your docker command to resolve this. Note that the port changes from 3000 to 8080, resulting in the link: <code>http://localhost:8080</code>.</p> <p><strong>Example Docker Command</strong>:</p> <pre><code class="language-bash">docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:main </code></pre> <h3>Keeping Your Docker Installation Up-to-Date</h3> <p>In case you want to update your local Docker installation to the latest version, you can do it with <a href="https://containrrr.dev/watchtower/">Watchtower</a>:</p> <pre><code class="language-bash">docker run --rm --volume /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower --run-once open-webui </code></pre> <p>In the last part of the command, replace <code>open-webui</code> with your container name if it is different.</p> <p>Check our Updating Guide available in our <a href="https://docs.openwebui.com/getting-started/updating">Open WebUI Documentation</a>.</p> <h3>Offline Mode</h3> <p>If you are running Open WebUI in an offline environment, you can set the <code>HF_HUB_OFFLINE</code> environment variable to <code>1</code> to prevent attempts to download models from the internet.</p> <pre><code class="language-bash">export HF_HUB_OFFLINE=1 </code></pre> <hr> <p>Docker-compose setup</p> <pre><code>version: '3.8' services: ollama: image: ollama/ollama container_name: ollama volumes: - ollama:/root/.ollama ports: - "11434:11434" restart: unless-stopped openwebui: image: ghcr.io/open-webui/open-webui:main container_name: openwebui depends_on: - ollama environment: - OLLAMA_BASE_URL=http://ollama:11434 ports: - "3000:3000" volumes: - openwebui:/app/backend/data restart: unless-stopped volumes: ollama: openwebui: </code></pre> https://tedneward.github.io/Research/tools/patat/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/patat/index.html <p><a href="https://github.com/jaspervdj/patat">Source</a></p> <p>Available via Homebrew (<code>brew install patat</code>).</p> <p>patat (Presentations Atop The ANSI Terminal) is a feature-rich presentation tool that runs in the terminal.</p> <ul> <li>Understands most markdown extensions and many other input formats (rST, Org-mode...) by building on top of Pandoc.</li> <li>Evaluate code snippets and show the result.</li> <li>Syntax highlighting for nearly one hundred languages generated from Kate syntax files.</li> <li>Automatically reload your slides as you edit them.</li> <li>Display speaker notes in a second window or monitor.</li> <li>Incremental slide display.</li> <li>Experimental images support.</li> <li>Transition effects.</li> <li>Supports smart slide splitting.</li> <li>Auto advancing with configurable delay.</li> <li>Centering and re-wrapping text to terminal width with proper indentation.</li> <li>Theming support including 24-bit RGB.</li> <li>Highly portable as it only requires an ANSI terminal as opposed to something like ncurses.</li> </ul> https://tedneward.github.io/Research/tools/peggyjs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/peggyjs/index.html <p><a href="https://peggyjs.org/">Website</a> | <a href="https://github.com/pegjs/pegjs">Source</a></p> https://tedneward.github.io/Research/tools/pgit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pgit/index.html <p><a href="https://git.erock.io/">Website</a> | <a href="https://github.com/picosh/pgit">Source</a></p> <p>This golang binary will generate a commit log, files, and references based on a git repository and the provided revisions.</p> <p>It will only generate a commit log and files for the provided revisions.</p> <h2>Getting Started</h2> <pre><code>git clone git@github.com:picosh/pgit.git cd pgit make build </code></pre> <pre><code>make build ./pgit --revs main --label pico --out ./public </code></pre> <p>To learn more about the options run:</p> <pre><code>./pgit --help </code></pre> <p>Multiple Repos</p> <pre><code>pgit \ --out ./public/pico \ --home-url "https://git.erock.io" \ --revs main \ --repo ~/pico \ --root-relative "/pico/" pgit \ --out ./public/starfx \ --home-url "https://git.erock.io" \ --revs main \ --repo ~/starfx \ --root-relative "/starfx/" echo '&lt;html&gt;&lt;body&gt;&lt;a href="/pico"&gt;pico&lt;/a&gt;&lt;a href="/starfx"&gt;starfx&lt;/a&gt;&lt;/body&gt;&lt;/html&gt;' &gt; ./public/index.html rsync -rv ./public/ pgs.sh:/git </code></pre> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.howtogeek.com/this-easy-tool-gives-me-the-best-of-github-on-my-local-machine/">https://www.howtogeek.com/this-easy-tool-gives-me-the-best-of-github-on-my-local-machine/</a></li> </ul> https://tedneward.github.io/Research/tools/officecli/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/officecli/index.html <p><a href="https://www.officecli.ai/">Website</a> | <a href="https://github.com/iOfficeAI/OfficeCLI">Source</a></p> <p>Agent-chat installation: <code>curl -fsSL https://officecli.ai/SKILL.md</code> "Paste this into your AI agent's chat — it will read the skill file and install everything automatically. The skill file teaches the agent how to install the binary and use all commands."</p> <blockquote> <p>OfficeCLI ships with a <a href="https://github.com/iOfficeAI/OfficeCLI/blob/main/SKILL.md">SKILL.md</a> (239 lines, ~8K tokens) that covers command syntax, architecture, and common pitfalls. After installation, your agent can immediately create, read, and modify any Office document.</p> </blockquote> <h2>Quick Start</h2> <p>From zero to a finished presentation in seconds:</p> <pre><code># Create a new PowerPoint officecli create deck.pptx # Add a slide with a title and background color officecli add deck.pptx / --type slide --prop title="Q4 Report" --prop background=1A1A2E # Add a text shape to the slide officecli add deck.pptx /slide\[1\] --type shape \\ --prop text="Revenue grew 25%" --prop x=2cm --prop y=5cm \\ --prop font=Arial --prop size=24 --prop color=FFFFFF # View the outline of the presentation officecli view deck.pptx outline </code></pre> <p>Output:</p> <pre><code>Slide 1: Q4 Report Shape 1 [TextBox]: Revenue grew 25% </code></pre> <pre><code># Get structured JSON for any element officecli get deck.pptx /slide\[1\]/shape\[1\] --json </code></pre> <p>Output:</p> <pre><code>{ "tag": "shape", "path": "/slide\[1\]/shape\[1\]", "attributes": { "name": "TextBox 1", "text": "Revenue grew 25%", "x": "720000", "y": "1800000" } } </code></pre> <h2>Why OfficeCLI?</h2> <p>What used to take 50 lines of Python and 3 separate libraries:</p> <pre><code>from pptx import Presentation from pptx.util import Inches, Pt prs \= Presentation() slide \= prs.slides.add\_slide(prs.slide\_layouts\[0\]) title \= slide.shapes.title title.text \= "Q4 Report" \# ... 45 more lines ... prs.save('deck.pptx') </code></pre> <p>Now takes one command:</p> <pre><code>officecli add deck.pptx / --type slide --prop title="Q4 Report" </code></pre> <p><strong>What OfficeCLI can do:</strong></p> <ul> <li><strong>Create</strong> documents from scratch -- blank or with content</li> <li><strong>Read</strong> text, structure, styles, formulas -- in plain text or structured JSON</li> <li><strong>Analyze</strong> formatting issues, style inconsistencies, and structural problems</li> <li><strong>Modify</strong> any element -- text, fonts, colors, layout, formulas, charts, images</li> <li><strong>Reorganize</strong> content -- add, remove, move, copy elements across documents</li> </ul> <p><strong>Word</strong> — <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-paragraph">paragraphs</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-run">runs</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-table">tables</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-style">styles</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-header-footer">headers/footers</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-picture">images</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-equation">equations</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-comment">comments</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-footnote">footnotes</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-watermark">watermarks</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-bookmark">bookmarks</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-toc">TOC</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-chart">charts</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-hyperlink">hyperlinks</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-section">sections</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-formfield">form fields</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-sdt">content controls (SDT)</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-field">fields</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/word-document">document properties</a></p> <p><strong>Excel</strong> — <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-cell">cells</a>, formulas (150+ built-in functions with auto-evaluation), <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-sheet">sheets</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-table">tables</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-conditionalformatting">conditional formatting</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-chart">charts</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-pivottable">pivot tables</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-namedrange">named ranges</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-validation">data validation</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-picture">images</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-sparkline">sparklines</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-comment">comments</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-autofilter">autofilter</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/excel-shape">shapes</a>, CSV/TSV import, <code>$Sheet:A1</code> cell addressing</p> <p><strong>PowerPoint</strong> — <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-slide">slides</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-shape">shapes</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-picture">images</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-table">tables</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-chart">charts</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-slide">animations</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-morph-check">morph transitions</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-3dmodel">3D models (.glb)</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-zoom">slide zoom</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-equation">equations</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-theme">themes</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-connector">connectors</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-video">video/audio</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-group">groups</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-notes">notes</a>, <a href="https://github.com/iOfficeAI/OfficeCLI/wiki/ppt-placeholder">placeholders</a></p> <h2>Installation</h2> <p>Ships as a single self-contained binary. The .NET runtime is embedded -- nothing to install, no runtime to manage.</p> <p><strong>One-line install:</strong></p> <p>macOS / Linux: <code>curl -fsSL https://raw.githubusercontent.com/iOfficeAI/OfficeCLI/main/install.sh | bash</code></p> <p>Windows (PowerShell): <code>irm https://raw.githubusercontent.com/iOfficeAI/OfficeCLI/main/install.ps1 | iex</code></p> <p><strong>Or download manually</strong> from <a href="https://github.com/iOfficeAI/OfficeCLI/releases">GitHub Releases</a>:</p> <table> <thead> <tr> <th> Platform </th> <th> Binary </th> </tr> </thead> <tbody> <tr> <td> macOS Apple Silicon </td> <td> <code>officecli-mac-arm64</code> </td> </tr> <tr> <td> macOS Intel </td> <td> <code>officecli-mac-x64</code> </td> </tr> <tr> <td> Linux x64 </td> <td> <code>officecli-linux-x64</code> </td> </tr> <tr> <td> Linux ARM64 </td> <td> <code>officecli-linux-arm64</code> </td> </tr> <tr> <td> Windows x64 </td> <td> <code>officecli-win-x64.exe</code> </td> </tr> <tr> <td> Windows ARM64 </td> <td> <code>officecli-win-arm64.exe</code> </td> </tr> </tbody> </table> <p>Verify installation: <code>officecli --version</code></p> <p><strong>Or self-install from a downloaded binary:</strong> <code>officecli install</code></p> <p>Updates are checked automatically in the background. Disable with <code>officecli config autoUpdate false</code> or skip per-invocation with <code>OFFICECLI_SKIP_UPDATE=1</code>. Configuration lives under <code>~/.officecli/config.json</code>.</p> <h2>Three-Layer Architecture</h2> <p>Start simple, go deep only when needed.</p> <table> <thead> <tr> <th> Layer </th> <th> Purpose </th> <th> Commands </th> </tr> </thead> <tbody> <tr> <td> <strong>L1: Read</strong> </td> <td> Semantic views of content </td> <td> <code>view</code> (text, annotated, outline, stats, issues, html) </td> </tr> <tr> <td> <strong>L2: DOM</strong> </td> <td> Structured element operations </td> <td> <code>get</code>, <code>query</code>, <code>set</code>, <code>add</code>, <code>remove</code>, <code>move</code> </td> </tr> <tr> <td> <strong>L3: Raw XML</strong> </td> <td> Direct XPath access — universal fallback </td> <td> <code>raw</code>, <code>raw-set</code>, <code>add-part</code>, <code>validate</code> </td> </tr> </tbody> </table> <h1>L1 — high-level views</h1> <pre><code>officecli view report.docx annotated officecli view budget.xlsx text --cols A,B,C --max-lines 50 </code></pre> <h1>L2 — element-level operations</h1> <pre><code>officecli query report.docx "run:contains(TODO)" officecli add budget.xlsx / --type sheet --prop name="Q2 Report" officecli move report.docx /body/p\[5\] --to /body --index 1 </code></pre> <h1>L3 — raw XML when L2 isn't enough</h1> <pre><code>officecli raw deck.pptx /slide\[1\] officecli raw-set report.docx document \\ --xpath "//w:p\[1\]" --action append \\ --xml '&lt;w:r&gt;&lt;w:t&gt;Injected text&lt;/w:t&gt;&lt;/w:r&gt;' </code></pre> <h2>AI Integration</h2> <h3>MCP Server</h3> <p>Built-in <a href="https://modelcontextprotocol.io/">MCP</a> server — register with one command:</p> <pre><code>officecli mcp claude # Claude Code officecli mcp cursor # Cursor officecli mcp vscode # VS Code / Copilot officecli mcp lmstudio # LM Studio officecli mcp list # Check registration status </code></pre> <p>Exposes all document operations as tools over JSON-RPC — no shell access needed.</p> <h3>Direct CLI Integration</h3> <p>Get OfficeCLI working with your AI agent in two steps:</p> <ol> <li><strong>Install the binary</strong> -- one command (see <a href="https://github.com/iOfficeAI/OfficeCLI#installation">Installation</a>)</li> <li><strong>Done.</strong> OfficeCLI automatically detects your AI tools (Claude Code, GitHub Copilot, Codex) by checking known config directories and installs its skill file. Your agent can immediately create, read, and modify any Office document.</li> </ol> <p><strong>Manual setup (optional)</strong></p> <p>If auto-install doesn't cover your setup, you can install the skill file manually:</p> <p><strong>Feed SKILL.md to your agent directly:</strong></p> <pre><code>curl -fsSL https://officecli.ai/SKILL.md </code></pre> <p><strong>Install as a local skill for Claude Code:</strong></p> <pre><code>curl -fsSL https://officecli.ai/SKILL.md -o ~/.claude/skills/officecli.md </code></pre> <p><strong>Other agents:</strong> Include the contents of <code>SKILL.md</code> (239 lines, ~8K tokens) in your agent's system prompt or tool description.</p> <p><strong>Call from any language:</strong></p> <p>Python:</p> <pre><code>import subprocess, json def cli(\*args): return subprocess.check\_output(\["officecli", \*args\], text\=True) cli("create", "deck.pptx") cli("set", "deck.pptx", "/slide\[1\]/shape\[1\]", "--prop", "text=Hello") </code></pre> <p>JavaScript:</p> <pre><code>const { execFileSync } \= require('child\_process') const cli \= (...args) \=&gt; execFileSync('officecli', args, { encoding: 'utf8' }) cli('set', 'deck.pptx', '/slide\[1\]/shape\[1\]', '--prop', 'text=Hello') </code></pre> <p>Every command supports <code>--json</code> for structured output. Path-based addressing means agents don't need to understand XML namespaces.</p> https://tedneward.github.io/Research/tools/oneurl/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/oneurl/index.html <p><a href="https://www.oneurl.live/">Website</a> | <a href="https://github.com/KartikLabhshetwar/oneurl">Source</a></p> https://tedneward.github.io/Research/tools/openpeon/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/openpeon/index.html <p><a href="https://openpeon.com/">Website</a> | <a href="https://openpeon.com/spec">Specification</a> | <a href="https://openpeon.com/packs">Packs</a></p> https://tedneward.github.io/Research/tools/orca-jam/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/orca-jam/index.html <p><a href="https://orca.handmade.network/">Website</a> |</p> https://tedneward.github.io/Research/tools/pck/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pck/index.html <p>"It currently supports Yacc/Bison and Flex/lex. Support is coming for GOLD, ANTLR and possibly Coco/R. ... In addition, Pck is unique in that it exposes much of the generation process (including grammar transformation) to the console, so that you can run intermediate tools on the output before feeding into the input of the next phase of the transformation. For example, you may convert a high level representation of a grammar into the low level representation used by the engine, and then left factor it for LL(1) parsing before passing it to the generator, a bit like your compiler and linker work together." -- from <a href="https://www.codeproject.com/Articles/5163943/Pck-The-Parser-Construction-Kit">CodeProject Article</a></p> <p><a href="https://github.com/codewitch-honey-crisis/pck">Github</a></p> https://tedneward.github.io/Research/tools/pencil-project/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pencil-project/index.html <p><a href="https://pencil.evolus.vn/">Website</a> | <a href="https://github.com/evolus/pencil">Source</a></p> https://tedneward.github.io/Research/tools/phasar/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/phasar/index.html <p><a href="http://phasar.org/">Website</a> | <a href="https://github.com/secure-software-engineering/phasar">Source</a></p> <ul> <li>Static Analysis for C++ with Phasar <ul> <li><a href="https://pldi18.sigplan.org/event/pldi-2018-pldi-tutorials-static-analysis-for-c-with-phasar">https://pldi18.sigplan.org/event/pldi-2018-pldi-tutorials-static-analysis-for-c-with-phasar</a></li> </ul> </li> <li>PhASAR: An Inter-Procedural Static Analysis Framework for C/C++ <ul> <li>TACAS 2019</li> <li>Philipp D. Schubert, Ben Hermann, Eric Bodden</li> <li><a href="http://www.thewhitespace.de/publications/shb19-phasar.pdf">http://www.thewhitespace.de/publications/shb19-phasar.pdf</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/ollydbg/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ollydbg/index.html <p><a href="https://www.ollydbg.de/">Website</a> | No source but free dowload</p> <p>Work is suspended on 64-bit version.</p> https://tedneward.github.io/Research/tools/onyx/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/onyx/index.html <p><a href="https://onyx.app/">Website</a> | <a href="https://github.com/onyx-dot-app/onyx">Source</a></p> https://tedneward.github.io/Research/tools/openrewrite/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/openrewrite/index.html <p><a href="https://docs.openrewrite.org/">Website</a> | <a href="https://github.com/openrewrite/rewrite">Source</a> | <a href="https://docs.openrewrite.org/recipes">Recipe Catalog</a> | <a href="https://docs.openrewrite.org/authoring-recipes">Authoring Recipes</a></p> <p>Moderne is a cloud-hosted commercial implementation of OpenRewrite.</p> <h2>Articles</h2> <ul> <li>Baeldung: <a href="https://www.baeldung.com/java-openrewrite">"A Guide to OpenRewrite"</a></li> </ul> <h2>Notes</h2> <p><strong><em>automatically refactor source code by applying recipes to the project</em></strong> Each recipe can perform specific refactoring tasks. These recipes are written in Java code and included in the build process using the OpenRewrite Maven or Gradle plugin. Comes with a variety of built-in recipes for common transformations. Community contributes recipes for others.</p> <p>Uses <a href="https://docs.openrewrite.org/concepts-and-explanations/lossless-semantic-trees">"Lossless Semantic Trees"</a> (basically boosted ASTs) to enable more fine-grained input as source to a collection of rules (recipes) that are run over source:</p> <p>Unlike the traditional Abstract Syntax Tree (AST), OpenRewrite's LST offers a unique set of characteristics that make it possible to perform accurate transformations and searches across a repository:</p> <p>Type-attributed. Each LST is imbued with type information. For example, when referencing a field, the source code may just refer to it as myField. The OpenRewrite LST for myField, on the other hand, would contain additional information about what the type of myField is, even if it isn't defined in the same source file or even the same project.<br> Format-preserving. Whitespace before and after LSTs are preserved in the tree so the tree can be printed out to reconstitute the original source code without clobbering formatting. Additionally, refactoring operations that insert code are sensitive to the local style of the code around them and match the local style.<br> Type attribution is necessary for the accurate matching of patterns. For example, if we are looking for SLF4J log statements and we see a statement like the following, without type attribution how do we know if logger is an SLF4J or a Logback logger?</p> <pre><code>logger.info("Hi"); </code></pre> <p>Format preservation is necessary if formatting isn't 100% consistent throughout the whole repository.</p> <p>When you run an OpenRewrite recipe locally:</p> <ol> <li>The OpenRewrite process creates an LST which is stored in memory. This reflects the current state of the repository on disk.</li> <li>The process continues by making transformations on the LST. This could be adding a search marker (~~&gt;) if the recipe is a search recipe -- or it could be more significant code changes.</li> <li>Once the recipe finishes running, the LST is converted back to text which then is used to overwrite any existing files that have changed.</li> <li>After all of the files have been overwritten, the process ends. Nothing is stored between recipe runs.</li> <li>If you go to run another recipe after the first one is done, a new LST will be generated at that time.</li> <li>If the previous recipe made changes and those changes exist locally, then the newly generated LST will have all of those changes. If the previous recipe made no changes, then the LST will effectively be the same one as before (but regenerated as the previous one no longer exists).</li> </ol> <p><a href="https://docs.openrewrite.org/concepts-and-explanations/lst-examples">Java Examples of LSTs</a></p> https://tedneward.github.io/Research/tools/panda/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/panda/index.html <p><a href="https://github.com/panda-re/panda">Source</a></p> https://tedneward.github.io/Research/tools/pdf24/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pdf24/index.html <p><a href="https://tools.pdf24.org/en/">Website</a> | Downloadable desktop version</p> <p>Merge PDF | Split PDF | Compress PDF | Edit PDF | Sign PDF | PDF Converter | Images to PDF | PDF to images | Extract PDF images | Protect PDF | Unlock PDF | Rotate PDF pages | Remove PDF pages | Extract PDF pages | Rearrange PDF pages | Webpage to PDF | Create PDF job application | Create PDF with a camera | PDF OCR | Add watermark | Add page numbers | View as PDF | PDF Overlay | Compare PDFs | Web optimize PDF | Annotate PDF | Redact PDF | Create PDF | PDF to Word | JPG to PDF</p> https://tedneward.github.io/Research/tools/peonping/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/peonping/index.html <p><a href="https://www.peonping.com/">Website</a></p> https://tedneward.github.io/Research/tools/pin/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/pin/index.html <p><a href="http://www.pintool.org/">Website</a> | <a href="https://en.wikipedia.org/wiki/Pin_(computer_program)">Wikipedia</a></p> <h3>Readings</h3> <ul> <li>Analyzing Parallel Programs With Pin <ul> <li>IEEE Computer 43(3) 2010</li> <li>Moshe Bach, Mark Charney, Robert Cohn, Elena Demikhovsky, Tevi Devor, Kim Hazelwood, Aamer Jaleel, Chi-Keung Luk, Gail Lyons, Harish Patil, Ady Tal</li> <li><a href="https://doi.org/10.1109/mc.2010.60">https://doi.org/10.1109/mc.2010.60</a></li> </ul> </li> <li>Pin: Building Customized Program Analysis Tools with Dynamic Instrumentation <ul> <li>PLDI 2005</li> <li>Chi-Keung Luk, Robert Cohn, Robert Muth, Harish Patil, Artur Klauser, Geoff Lowney, Steven Wallace, Vijay Janapa Reddi, Kim Hazelwood</li> <li><a href="https://doi.org/10.1145/1065010.1065034">https://doi.org/10.1145/1065010.1065034</a></li> </ul> </li> <li>Deterministic PinPoints: Representative and Repeatable Simulation Region Selection with PinPlay and Sniper <ul> <li>HPCA 2013 Tutorial</li> <li>Harish Patil, Trevor E. Carlson</li> <li><a href="https://snipersim.org/w/Tutorial:HPCA_2013_PinPoints">https://snipersim.org/w/Tutorial:HPCA_2013_PinPoints</a></li> </ul> </li> <li>Dynamic Binary Analysis with Intel Pin <ul> <li><a href="https://blog.netspi.com/dynamic-binary-analysis-intel-pin/">https://blog.netspi.com/dynamic-binary-analysis-intel-pin/</a></li> <li><a href="https://github.com/NetSPI/Pin">https://github.com/NetSPI/Pin</a></li> </ul> </li> <li>Pinballs: Portable and Shareable User-level Checkpoints for Reproducible Analysis and Simulation <ul> <li>REPRODUCE 2014: Workshop on Reproducible Research Methodologies</li> <li>Harish Patil, Trevor E. Carlson</li> <li></li> </ul> </li> </ul> <p><a href="https://www.researchgate.net/publication/280306016_Pinballs_Portable_and_Shareable_User-level_Checkpoints_for_Reproducible_Analysis_and_Simulation_Appeared_in_REPRODUCE_2014_Workshop_on_Reproducible_Research_Methodologies">https://www.researchgate.net/publication/280306016_Pinballs_Portable_and_Shareable_User-level_Checkpoints_for_Reproducible_Analysis_and_Simulation_Appeared_in_REPRODUCE_2014_Workshop_on_Reproducible_Research_Methodologies</a><br> - PinPlay: a framework for deterministic replay and reproducible analysis of parallel programs<br> - Code Generation and Optimization (CGO) 2010<br> - Harish Patil, Cristiano Pereira, Mack Stallcup, Gregory Lueck, James Cownie<br> - <a href="https://doi.org/10.1145/1772954.1772958">https://doi.org/10.1145/1772954.1772958</a><br> - PinPlay: Using PinPlay for Reproducible Analysis and Replay Debugging<br> - PLDI 2015 Tutorial<br> - Harish Patil, Milind Chabbi<br> - <a href="https://pldi15.sigplan.org/details/pldi2015-workshops/6/PINPLAY-Using-PinPlay-for-Reproducible-Analysis-and-Replay-Debugging">https://pldi15.sigplan.org/details/pldi2015-workshops/6/PINPLAY-Using-PinPlay-for-Reproducible-Analysis-and-Replay-Debugging</a><br> - <a href="https://sites.google.com/site/pinplaypldi2015tutotial/">https://sites.google.com/site/pinplaypldi2015tutotial/</a><br> - PinPoints: Pinpointing Representative Portions of Large Intel® Itanium® Programs with Dynamic Instrumentation<br> - MICRO 2004<br> - H. Patil, R. Cohn, M. Charney, R. Kapoor, A. Sun, A. Karunanidhi<br> - <a href="https://doi.org/10.1109/MICRO.2004.28">https://doi.org/10.1109/MICRO.2004.28</a><br> - <a href="https://software.intel.com/en-us/articles/pin-a-binary-instrumentation-tool-pinpoints">https://software.intel.com/en-us/articles/pin-a-binary-instrumentation-tool-pinpoints</a><br> - PinPoints: Simulation Region Selection with PinPlay and Sniper<br> - ISCA 2014 Tutorial<br> - Harish Patil, Mack Stallcup<br> - <a href="https://sites.google.com/site/pinpointstutorialisca14/">https://sites.google.com/site/pinpointstutorialisca14/</a><br> - Some thoughts about code-coverage measurement with Pin<br> - <a href="https://doar-e.github.io/blog/2013/08/31/some-thoughts-about-code-coverage-measurement-with-pin/">https://doar-e.github.io/blog/2013/08/31/some-thoughts-about-code-coverage-measurement-with-pin/</a></p> <h3>Projects</h3> <ul> <li>Ablation: Augmenting Static Analysis Using Pintool <ul> <li><a href="https://github.com/paulmehta/Ablation">https://github.com/paulmehta/Ablation</a></li> <li>Black Hat 2016; Paul Mehta</li> <li><a href="https://www.youtube.com/watch?v=wHIlNRK_HiQ">https://www.youtube.com/watch?v=wHIlNRK_HiQ</a></li> </ul> </li> <li>BasicBlocks: Pin tool for printing the address of each basic block executed in a program. <ul> <li><a href="https://github.com/chokepoint/BasicBlocks">https://github.com/chokepoint/BasicBlocks</a></li> </ul> </li> <li>Pin++: A C++ template meta-programmable framework for authoring Pintools <ul> <li><a href="https://github.com/SEDS/PinPP">https://github.com/SEDS/PinPP</a></li> <li>Pin++: A Object-oriented Framework for Writing Pintools <ul> <li>Generative Programming: Concepts and Experiences (GPCE) 2014</li> <li>James H. Hill and Dennis C. Feiock</li> <li><a href="https://core.ac.uk/download/pdf/46962422.pdf">https://core.ac.uk/download/pdf/46962422.pdf</a></li> </ul> </li> </ul> </li> <li>PinCTF: use Intel's Pin Tool to instrument binaries and count instructions <ul> <li><a href="https://github.com/ChrisTheCoolHut/PinCTF">https://github.com/ChrisTheCoolHut/PinCTF</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/magictrace/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/magictrace/index.html <p><a href="https://magic-trace.org/">Website</a> | <a href="https://github.com/janestreet/magic-trace">Source</a></p> <p>magic-trace only supports Linux. It relies on perf, which is Linux specific. We do not think this is a fundamental limitation of magic-trace, and would welcome patches to support other operating systems.</p> <p>magic-trace relies on Intel Processor Trace, which only really starts to be functional on relatively recent Intel CPUs. That means it doesn't support ARM, and it doesn't support AMD.</p> https://tedneward.github.io/Research/tools/markitdown/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/markitdown/index.html <p><a href="https://github.com/microsoft/markitdown">Source</a></p> <h3>Articles</h3> <ul> <li><a href="https://www.infoworld.com/article/3963991/markitdown-microsofts-open-source-tool-for-markdown-conversion.html">https://www.infoworld.com/article/3963991/markitdown-microsofts-open-source-tool-for-markdown-conversion.html</a></li> </ul> https://tedneward.github.io/Research/tools/mdq/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mdq/index.html <p><a href="https://yshavit.github.io/mdq-playground/">Website</a> | <a href="https://github.com/yshavit/mdq">Source</a></p> https://tedneward.github.io/Research/tools/middleman/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/middleman/index.html <p><a href="https://middlemanapp.com/">Website</a> | <a href="https://github.com/middleman/middleman">Source</a></p> <p>Ruby gem.</p> https://tedneward.github.io/Research/tools/mistral-vibe/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mistral-vibe/index.html <p><a href="https://github.com/mistralai/mistral-vibe">Source</a></p> <p>Mistral Vibe is a command-line coding assistant powered by Mistral's models. It provides a conversational interface to your codebase, allowing you to use natural language to explore, modify, and interact with your projects through a powerful set of tools.</p> https://tedneward.github.io/Research/tools/moon/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/moon/index.html <p><a href="https://moonrepo.dev/">Website</a> | <a href="https://github.com/moonrepo">Source</a></p> https://tedneward.github.io/Research/tools/msft-power-toys/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/msft-power-toys/index.html <p><a href="https://aka.ms/getPowertoys">Microsoft Store</a> | <a href="https://docs.microsoft.com/en-us/windows/powertoys/install">Docs</a></p> https://tedneward.github.io/Research/tools/ndepend/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ndepend/index.html <p><a href="https://www.ndepend.com/">Website</a> | Commercial</p> https://tedneward.github.io/Research/tools/nixos/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nixos/index.html <p><a href="https://nixos.org">Website</a> | <a href="https://github.com/NixOS">Source</a></p> <h2>Articles</h2> <ul> <li> <p><a href="https://nixos.org/nix/manual/#ch-expression-language">Nix Manual - Nix expression language</a></p> </li> <li> <p><a href="https://jonathanlorimer.dev/posts/nix-thesis.html">The Nix Thesis</a>:</p> </li> <li> <p><a href="https://medium.com/@MrJamesFisher/nix-by-example-a0063a1a4c55">James Fisher - Nix by example - Part 1: The Nix expression language</a></p> </li> <li> <p><a href="https://ebzzry.io/en/nix/#nix">Rommel Martinez - A Gentle Introduction to the Nix Family</a></p> </li> <li> <p><a href="https://jvns.ca/blog/2023/02/28/some-notes-on-using-nix/">Some notes on using nix</a></p> </li> </ul> <h2><a href="https://learnxinyminutes.com/docs/nix/">Learn X in Y minutes Quick Intro:</a></h2> <hr> <p>Nix is a simple functional language developed for the <a href="https://nixos.org/nix/">Nix package manager</a> and <a href="https://nixos.org/">NixOS</a>.</p> <p>You can evaluate Nix expressions using <a href="https://nixos.org/nix/manual/#sec-nix-instantiate">nix-instantiate</a> or <a href="https://nixos.org/nix/manual/#ssec-relnotes-2.0"><code>nix repl</code></a>.</p> <pre><code>with builtins; [ # Comments #========================================= # Inline comments look like this. /* Multi-line comments look like this. */ # Booleans #========================================= (true &amp;&amp; false) # And #=&gt; false (true || false) # Or #=&gt; true (if 3 &lt; 4 then "a" else "b") # Conditional #=&gt; "a" # Integers and Floats #========================================= # There are two numeric types: integers and floats 1 0 42 (-3) # Some integers 123.43 .27e13 # A couple of floats # Operations will preserve numeric type (4 + 6 + 12 - 2) # Addition #=&gt; 20 (4 - 2.5) #=&gt; 1.5 (7 / 2) # Division #=&gt; 3 (7 / 2.0) #=&gt; 3.5 # Strings #========================================= "Strings literals are in double quotes." " String literals can span multiple lines. " '' This is called an "indented string" literal. It intelligently strips leading whitespace. '' '' a b '' #=&gt; "a\n b" ("ab" + "cd") # String concatenation #=&gt; "abcd" # Antiquotation lets you embed values into strings. ("Your home directory is ${getEnv "HOME"}") #=&gt; "Your home directory is /home/alice" # Paths #========================================= # Nix has a primitive data type for paths. /tmp/tutorials/learn.nix # A relative path is resolved to an absolute path at parse # time, relative to the file in which it occurs. tutorials/learn.nix #=&gt; /the-base-path/tutorials/learn.nix # A path must contain at least one slash, so a relative # path for a file in the same directory needs a ./ prefix, ./learn.nix #=&gt; /the-base-path/learn.nix # The / operator must be surrounded by whitespace if # you want it to signify division. 7/2 # This is a path literal (7 / 2) # This is integer division # Imports #========================================= # A nix file contains a single top-level expression with no free # variables. An import expression evaluates to the value of the # file that it imports. (import /tmp/foo.nix) # Imports can also be specified by strings. (import "/tmp/foo.nix") # Import paths must be absolute. Path literals # are automatically resolved, so this is fine. (import ./foo.nix) # But this does not happen with strings. (import "./foo.nix") #=&gt; error: string ‘foo.nix’ doesn't represent an absolute path # Let #========================================= # `let` blocks allow us to bind values to variables. (let x = "a"; in x + x + x) #=&gt; "aaa" # Bindings can refer to each other, and their order does not matter. (let y = x + "b"; x = "a"; in y + "c") #=&gt; "abc" # Inner bindings shadow outer bindings. (let a = 1; in let a = 2; in a) #=&gt; 2 # Functions #========================================= (n: n + 1) # Function that adds 1 ((n: n + 1) 5) # That same function, applied to 5 #=&gt; 6 # There is no syntax for named functions, but they # can be bound by `let` blocks like any other value. (let succ = (n: n + 1); in succ 5) #=&gt; 6 # A function has exactly one argument. # Multiple arguments can be achieved with currying. ((x: y: x + "-" + y) "a" "b") #=&gt; "a-b" # We can also have named function arguments, # which we'll get to later after we introduce sets. # Lists #========================================= # Lists are denoted by square brackets. (length [1 2 3 "x"]) #=&gt; 4 ([1 2 3] ++ [4 5]) #=&gt; [1 2 3 4 5] (concatLists [[1 2] [3 4] [5]]) #=&gt; [1 2 3 4 5] (head [1 2 3]) #=&gt; 1 (tail [1 2 3]) #=&gt; [2 3] (elemAt ["a" "b" "c" "d"] 2) #=&gt; "c" (elem 2 [1 2 3]) #=&gt; true (elem 5 [1 2 3]) #=&gt; false (filter (n: n &lt; 3) [1 2 3 4]) #=&gt; [ 1 2 ] # Sets #========================================= # A "set" is an unordered mapping with string keys. { foo = [1 2]; bar = "x"; } # The . operator pulls a value out of a set. { a = 1; b = 2; }.a #=&gt; 1 # The ? operator tests whether a key is present in a set. ({ a = 1; b = 2; } ? a) #=&gt; true ({ a = 1; b = 2; } ? c) #=&gt; false # The // operator merges two sets. ({ a = 1; } // { b = 2; }) #=&gt; { a = 1; b = 2; } # Values on the right override values on the left. ({ a = 1; b = 2; } // { a = 3; c = 4; }) #=&gt; { a = 3; b = 2; c = 4; } # The rec keyword denotes a "recursive set", # in which attributes can refer to each other. (let a = 1; in { a = 2; b = a; }.b) #=&gt; 1 (let a = 1; in rec { a = 2; b = a; }.b) #=&gt; 2 # Nested sets can be defined in a piecewise fashion. { a.b = 1; a.c.d = 2; a.c.e = 3; }.a.c #=&gt; { d = 2; e = 3; } # Sets are immutable, so you can't redefine an attribute: { a = { b = 1; }; a.b = 2; } #=&gt; attribute 'a.b' at (string):3:5 already defined at (string):2:11 # However, an attribute's set members can also be defined piecewise # way even if the attribute itself has been directly assigned. { a = { b = 1; }; a.c = 2; } #=&gt; { a = { b = 1; c = 2; }; } # With #========================================= # The body of a `with` block is evaluated with # a set's mappings bound to variables. (with { a = 1; b = 2; }; a + b) # =&gt; 3 # Inner bindings shadow outer bindings. (with { a = 1; b = 2; }; (with { a = 5; }; a + b)) #=&gt; 7 # This first line of tutorial starts with "with builtins;" # because builtins is a set that contains all of the built-in # functions (length, head, tail, filter, etc.). This saves # us from having to write, for example, "builtins.length" # instead of just "length". # Set patterns #========================================= # Sets are useful when we need to pass multiple values # to a function. (args: args.x + "-" + args.y) { x = "a"; y = "b"; } #=&gt; "a-b" # This can be written more clearly using set patterns. ({x, y}: x + "-" + y) { x = "a"; y = "b"; } #=&gt; "a-b" # By default, the pattern fails on sets containing extra keys. ({x, y}: x + "-" + y) { x = "a"; y = "b"; z = "c"; } #=&gt; error: anonymous function called with unexpected argument ‘z’ # Adding ", ..." allows ignoring extra keys. ({x, y, ...}: x + "-" + y) { x = "a"; y = "b"; z = "c"; } #=&gt; "a-b" # Errors #========================================= # `throw` causes evaluation to abort with an error message. (2 + (throw "foo")) #=&gt; error: foo # `tryEval` catches thrown errors. (tryEval 42) #=&gt; { success = true; value = 42; } (tryEval (2 + (throw "foo"))) #=&gt; { success = false; value = false; } # `abort` is like throw, but it's fatal; it cannot be caught. (tryEval (abort "foo")) #=&gt; error: evaluation aborted with the following error message: ‘foo’ # `assert` evaluates to the given value if true; # otherwise it throws a catchable exception. (assert 1 &lt; 2; 42) #=&gt; 42 (assert 1 &gt; 2; 42) #=&gt; error: assertion failed at (string):1:1 (tryEval (assert 1 &gt; 2; 42)) #=&gt; { success = false; value = false; } # Impurity #========================================= # Because repeatability of builds is critical to the Nix package # manager, functional purity is emphasized in the Nix language # used to describe Nix packages. But there are a few impurities. # You can refer to environment variables. (getEnv "HOME") #=&gt; "/home/alice" # The trace function is used for debugging. It prints the first # argument to stderr and evaluates to the second argument. (trace 1 2) #=&gt; trace: 1 #=&gt; 2 # You can write files into the Nix store. Although impure, this is # fairly safe because the file name is derived from the hash of # its contents. You can read files from anywhere. In this example, # we write a file into the store, and then read it back out. (let filename = toFile "foo.txt" "hello!"; in [filename (builtins.readFile filename)]) #=&gt; [ "/nix/store/ayh05aay2anx135prqp0cy34h891247x-foo.txt" "hello!" ] # We can also download files into the Nix store. (fetchurl "https://example.com/package-1.2.3.tgz") #=&gt; "/nix/store/2drvlh8r57f19s9il42zg89rdr33m2rm-package-1.2.3.tgz" ] </code></pre> https://tedneward.github.io/Research/tools/maigret/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/maigret/index.html <p><a href="https://github.com/soxoj/maigret">Source</a></p> <p>Maigret collects a dossier on a person by username only, checking for accounts on a huge number of sites and gathering all the available information from web pages. No API keys are required. Maigret is an easy-to-use and powerful fork of Sherlock.</p> <p>Currently supports more than 3000 sites (full list), search is launched against 500 popular sites in descending order of popularity by default. Also supported checking Tor sites, I2P sites, and domains (via DNS resolving).</p> https://tedneward.github.io/Research/tools/marktext/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/marktext/index.html <p><a href="https://www.marktext.me/">Website</a> | <a href="https://github.com/marktext/marktext">Source</a></p> https://tedneward.github.io/Research/tools/medicat/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/medicat/index.html <p><a href="https://medicatusb.com/">Website</a> | <a href="https://medicatusb.com/docs/">Docs</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://gbatemp.net/threads/medicat-usb-a-multiboot-linux-usb-for-pc-repair.361577/">https://gbatemp.net/threads/medicat-usb-a-multiboot-linux-usb-for-pc-repair.361577/</a></li> </ul> https://tedneward.github.io/Research/tools/mill/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mill/index.html <p><a href="https://mill-build.org/mill/0.11.12/Java_Intro_to_Mill.html">Website</a> | <a href="https://github.com/com-lihaoyi/mill">Source</a></p> https://tedneward.github.io/Research/tools/mizu/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mizu/index.html <p><a href="https://getmizu.io/">Website</a> | <a href="https://github.com/up9inc/mizu">Source</a></p> https://tedneward.github.io/Research/tools/mps/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mps/index.html <h2>Tutorials/Courses</h2> <p><a href="https://github.com/markusvoelter/mpsintrocourse">MPS Introduction Course</a>: A slightly advanced introduction to MPS: two languages, a generator, an interpreter, language composition and non-textual notations.</p> https://tedneward.github.io/Research/tools/msty/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/msty/index.html <p><a href="https://msty.ai/">Website</a> | Commercial w/free option</p> https://tedneward.github.io/Research/tools/nexe/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nexe/index.html <p><a href="https://github.com/nexe/nexe">Source</a></p> <p>Motivation and Features</p> <ul> <li>Self contained applications</li> <li>Ability to run multiple applications with different node.js runtimes.</li> <li>Distribute binaries without needing node / npm.</li> <li>Idempotent builds</li> <li>Start and deploy faster.</li> <li>Lockdown specific application versions, and easily rollback.</li> <li>Flexible build pipeline</li> <li>Cross platform builds</li> </ul> https://tedneward.github.io/Research/tools/nmap/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nmap/index.html <p><a href="https://nmap.org/">Website</a></p> <p>Many systems and network administrators also find it useful for tasks such as network inventory, managing service upgrade schedules, and monitoring host or service uptime. Nmap uses raw IP packets in novel ways to determine what hosts are available on the network, what services (application name and version) those hosts are offering, what operating systems (and OS versions) they are running, what type of packet filters/firewalls are in use, and dozens of other characteristics. It was designed to rapidly scan large networks, but works fine against single hosts. Nmap runs on all major computer operating systems, and official binary packages are available for Linux, Windows, and Mac OS X. In addition to the classic command-line Nmap executable, the Nmap suite includes an advanced GUI and results viewer (Zenmap), a flexible data transfer, redirection, and debugging tool (Ncat), a utility for comparing scan results (Ndiff), and a packet generation and response analysis tool (Nping).</p> https://tedneward.github.io/Research/tools/manuskript/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/manuskript/index.html <p><a href="https://www.theologeek.ch/manuskript/">Website</a> | <a href="https://github.com/olivierkes/manuskript">Source</a></p> https://tedneward.github.io/Research/tools/mattermost/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mattermost/index.html <p><a href="https://mattermost.com/">Website</a> | <a href="https://github.com/mattermost/">Source</a></p> https://tedneward.github.io/Research/tools/mermaid/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mermaid/index.html <p><a href="https://mermaid-js.github.io/mermaid/#/">Website</a> | <a href="https://github.com/mermaid-js">Source</a> | <a href="https://github.com/mermaid-js/mermaid-cli">CLI</a> | <a href="https://mermaid.live/edit">Live Editor</a></p> <ul> <li><a href="https://www.freecodecamp.org/news/use-mermaid-javascript-library-to-create-flowcharts/">"Use Mermaid JS library to create flowcharts"</a></li> <li><a href="https://www.freecodecamp.org/news/how-to-become-an-analytical-programmer-compare-five-projects/">"How to Become an Analytical Programmer – Solve the "Rock, Paper, Scissors" Game 5 Ways Using JavaScript &amp; Mermaid.js"</a></li> </ul> <p>To create a Mermaid diagram in these pages, use <code>&lt;pre class="mermaid"&gt;</code> ... <code>&lt;/pre&gt;</code> to enclose the diagram text.</p> <p>Frontmatter configuration: The entire mermaid configuration (except the secure configs) can be overridden by the diagram author in the frontmatter of the diagram. The frontmatter is a YAML block at the top of the diagram.</p> <pre><code>--- title: Hello Title config: theme: base themeVariables: primaryColor: "#00ff00" --- flowchart Hello --&gt; World </code></pre> <pre class="mermaid">--- title: Hello Title config: theme: base themeVariables: primaryColor: "#00ff00" --- flowchart Hello --&gt; World </pre> <hr> <p><a href="https://mermaid.js.org/syntax/flowchart.html">Flowchart</a></p> <pre><code>flowchart TD A[Christmas] --&gt;|Get money| B(Go shopping) B --&gt; C{Let me think} C --&gt;|One| D[Laptop] C --&gt;|Two| E[iPhone] C --&gt;|Three| F[fa:fa-car Car] </code></pre> <pre class="mermaid">flowchart TD A[Christmas] --&gt;|Get money| B(Go shopping) B --&gt; C{Let me think} C --&gt;|One| D[Laptop] C --&gt;|Two| E[iPhone] C --&gt;|Three| F[fa:fa-car Car] </pre> <hr> <p><a href="https://mermaid.js.org/syntax/sequenceDiagram.html">Sequence diagram</a></p> <pre><code>sequenceDiagram Alice-&gt;&gt;+John: Hello John, how are you? Alice-&gt;&gt;+John: John, can you hear me? John--&gt;&gt;-Alice: Hi Alice, I can hear you! John--&gt;&gt;-Alice: I feel great! </code></pre> <pre class="mermaid">sequenceDiagram Alice-&gt;&gt;+John: Hello John, how are you? Alice-&gt;&gt;+John: John, can you hear me? John--&gt;&gt;-Alice: Hi Alice, I can hear you! John--&gt;&gt;-Alice: I feel great! </pre> <hr> <p><a href="https://mermaid.js.org/syntax/classDiagram.html">Class diagram</a></p> <pre><code>--- title: Animal example --- classDiagram Animal &lt;|-- Duck note for Duck "can fly\ncan swim\ncan dive\ncan help in debugging" Animal &lt;|-- Fish Animal &lt;|-- Zebra Animal : +int age Animal : +String gender Animal: +isMammal() Animal: +mate() class Duck{ +String beakColor +swim() +quack() } class Fish{ -int sizeInFeet -canEat() } class Zebra{ +bool is_wild +run() } class Square~Shape~{ int id List~int~ position setPoints(List~int~ points) getPoints() List~int~ } Square : -List~string~ messages Square : +setMessages(List~string~ messages) Square : +getMessages() List~string~ Square : +getDistanceMatrix() List~List~int~~ classA --|&gt; classB : Inheritance classC --* classD : Composition classE --o classF : Aggregation classG --&gt; classH : Association classI -- classJ : Link(Solid) classK ..&gt; classL : Dependency classM ..|&gt; classN : Realization classO .. classP : Link(Dashed) </code></pre> <pre class="mermaid">--- title: Animal example --- classDiagram Animal &lt;|-- Duck Animal &lt;|-- Fish Animal &lt;|-- Zebra Animal : +int age Animal : +String gender Animal: +isMammal() Animal: +mate() class Duck{ +String beakColor +swim() +quack() } class Fish{ -int sizeInFeet -canEat() } class Zebra{ +bool is_wild +run() } class Square~Shape~{ int id List~int~ position setPoints(List~int~ points) getPoints() List~int~ } <pre><code>Square : -List~string~ messages Square : +setMessages(List~string~ messages) Square : +getMessages() List~string~ Square : +getDistanceMatrix() List~List~int~~ classA --|&gt; classB : Inheritance classC --* classD : Composition classE --o classF : Aggregation classG --&gt; classH : Association classI -- classJ : Link(Solid) classK ..&gt; classL : Dependency classM ..|&gt; classN : Realization classO .. classP : Link(Dashed) </code></pre> </pre> <hr> <p><a href="https://mermaid.js.org/syntax/stateDiagram.html">State diagram</a></p> <pre><code>stateDiagram-v2 [*] --&gt; Still Still --&gt; [*] Still --&gt; Moving Moving --&gt; Still Moving --&gt; Crash Crash --&gt; [*] </code></pre> <pre class="mermaid">stateDiagram-v2 [*] --&gt; Still Still --&gt; [*] Still --&gt; Moving Moving --&gt; Still Moving --&gt; Crash Crash --&gt; [*] </pre> <hr> <p><a href="">Quadrant chart</a></p> <pre><code>quadrantChart title Reach and engagement of campaigns x-axis Low Reach --&gt; High Reach y-axis Low Engagement --&gt; High Engagement quadrant-1 We should expand quadrant-2 Need to promote quadrant-3 Re-evaluate quadrant-4 May be improved Campaign A: [0.9, 0.0] radius: 12 Campaign B:::class1: [0.8, 0.1] color: #ff3300, radius: 10 Campaign C: [0.7, 0.2] radius: 25, color: #00ff33, stroke-color: #10f0f0 Campaign D: [0.6, 0.3] radius: 15, stroke-color: #00ff0f, stroke-width: 5px ,color: #ff33f0 Campaign E:::class2: [0.5, 0.4] Campaign F:::class3: [0.4, 0.5] color: #0000ff classDef class1 color: #109060 classDef class2 color: #908342, radius : 10, stroke-color: #310085, stroke-width: 10px classDef class3 color: #f00fff, radius : 10 </code></pre> <pre class="mermaid">quadrantChart title Reach and engagement of campaigns x-axis Low Reach --&gt; High Reach y-axis Low Engagement --&gt; High Engagement quadrant-1 We should expand quadrant-2 Need to promote quadrant-3 Re-evaluate quadrant-4 May be improved Campaign A: [0.9, 0.0] radius: 12 Campaign B:::class1: [0.8, 0.1] color: #ff3300, radius: 10 Campaign C: [0.7, 0.2] radius: 25, color: #00ff33, stroke-color: #10f0f0 Campaign D: [0.6, 0.3] radius: 15, stroke-color: #00ff0f, stroke-width: 5px ,color: #ff33f0 Campaign E:::class2: [0.5, 0.4] Campaign F:::class3: [0.4, 0.5] color: #0000ff classDef class1 color: #109060 classDef class2 color: #908342, radius : 10, stroke-color: #310085, stroke-width: 10px classDef class3 color: #f00fff, radius : 10 </pre> https://tedneward.github.io/Research/tools/mineru/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mineru/index.html <p><a href="https://mineru.net/">Website</a> | <a href="https://github.com/opendatalab/mineru">Source</a> | <a href="https://deepwiki.com/opendatalab/MinerU">DeepWiki</a></p> https://tedneward.github.io/Research/tools/mkdocs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mkdocs/index.html <p><a href="https://www.mkdocs.org/">Website</a></p> https://tedneward.github.io/Research/tools/mq/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mq/index.html <p><a href="https://mqlang.org/">Website</a> | <a href="https://github.com/harehare/mq">Source</a></p> <h2><a href="https://github.com/harehare/mq#why-mq">Why mq?</a></h2> <p>mq makes working with Markdown files as easy as jq makes working with JSON. It's especially useful for:</p> <ul> <li><strong>LLM Workflows</strong>: Efficiently manipulate and process Markdown used in LLM prompts and outputs</li> <li><strong>LLM Input Generation</strong>: Generate structured Markdown content optimized for LLM consumption, since Markdown serves as the primary input format for most language models</li> <li><strong>Documentation Management</strong>: Extract, transform, and organize content across multiple documentation files</li> <li><strong>Content Analysis</strong>: Quickly extract specific sections or patterns from Markdown documents</li> <li><strong>Batch Processing</strong>: Apply consistent transformations across multiple Markdown files</li> </ul> <p>Since LLM inputs are primarily in Markdown format, mq provides efficient tools for generating and processing the structured Markdown content that LLMs require.</p> <h2><a href="https://github.com/harehare/mq#features">Features</a></h2> <ul> <li><strong>Slice and Filter</strong>: Extract specific parts of your Markdown documents with ease.</li> <li><strong>Map and Transform</strong>: Apply transformations to your Markdown content.</li> <li><strong>Command-line Interface</strong>: Simple and intuitive CLI for quick operations.</li> <li><strong>Extensibility</strong>: Easily extendable with custom functions.</li> <li><strong>Built-in support</strong>: Filter and transform content with many built-in functions and selectors.</li> <li><strong>REPL Support</strong>: Interactive command-line REPL for testing and experimenting.</li> <li><strong>IDE Support</strong>: VSCode Extension and Language Server <strong>Protocol</strong> (LSP) support for custom function development.</li> <li><strong>Debugger</strong>: Includes an experimental debugger (<code>mq-dbg</code>) for inspecting and stepping through mq queries interactively.</li> <li><strong>External Subcommands</strong>: Extend mq with custom subcommands by placing executable files starting with <code>mq-</code> in <code>~/.mq/bin/</code>.</li> </ul> <h2><a href="https://github.com/harehare/mq#installation">Getting Started</a></h2> <h3><a href="https://github.com/harehare/mq#quick-install">Quick Install</a></h3> <p><code>curl -sSL https://mqlang.org/install.sh | bash</code></p> <p>The installer will:</p> <ul> <li>Download the latest mq binary for your platform</li> <li>Install it to <code>~/.mq/bin/</code></li> <li>Update your shell profile to add mq to your PATH</li> </ul> <h3><a href="https://github.com/harehare/mq#binaries">Binaries</a></h3> <p>You can download pre-built binaries from the <a href="https://github.com/harehare/mq/releases">GitHub releases page</a>:</p> <h1>macOS (Apple Silicon): <code>curl -L https://github.com/harehare/mq/releases/download/v0.5.6/mq-aarch64-apple-darwin -o /usr/local/bin/mq &amp;&amp; chmod +x /usr/local/bin/mq</code></h1> <h1>Linux x86_64: <code>curl -L https://github.com/harehare/mq/releases/download/v0.5.6/mq-x86\_64-unknown-linux-gnu -o /usr/local/bin/mq &amp;&amp; chmod +x /usr/local/bin/mq</code></h1> <h1>Linux arm64: <code>curl -L https://github.com/harehare/mq/releases/download/v0.5.6/mq-aarch64-unknown-linux-gnu -o /usr/local/bin/mq &amp;&amp; chmod +x /usr/local/bin/mq</code></h1> <h1>Windows (PowerShell): <code>Invoke-WebRequest -Uri https://github.com/harehare/mq/releases/download/v0.5.6/mq-x86\_64-pc-windows-msvc.exe -OutFile "$env:USERPROFILE\\bin\\mq.exe"</code></h1> <h3><a href="https://github.com/harehare/mq#homebrew">Homebrew</a></h3> <pre><code># Using Homebrew (macOS and Linux) brew install mq </code></pre> <h3><a href="https://github.com/harehare/mq#docker">Docker</a></h3> <pre><code>$ docker run --rm ghcr.io/harehare/mq:0.5.6 </code></pre> <h3><a href="https://github.com/harehare/mq#python">Python</a></h3> <p>You can use mq in Python through the <code>markdown-query</code> package:</p> <pre><code># Install from PyPI pip install markdown-query </code></pre> https://tedneward.github.io/Research/tools/mu/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mu/index.html <p><a href="https://codewith.mu/en/">Website</a> | <a href="https://github.com/mu-editor/mu">Source</a></p> https://tedneward.github.io/Research/tools/ninja/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ninja/index.html <p><a href="https://ninja-build.org/">Website</a> | <a href="https://github.com/ninja-build/ninja">Github</a></p> <ul> <li>Benchmarking the Ninja build system <ul> <li><a href="http://david.rothlis.net/ninja-benchmark/">http://david.rothlis.net/ninja-benchmark/</a></li> </ul> </li> <li>Building Like (a) Ninja [Pt1] <ul> <li><a href="https://vector-of-bool.github.io/2018/12/20/build-like-ninja-1.html">https://vector-of-bool.github.io/2018/12/20/build-like-ninja-1.html</a></li> </ul> </li> <li>GN: a meta-build system that generates build files for Ninja <ul> <li><a href="https://gn.googlesource.com/gn">https://gn.googlesource.com/gn</a></li> <li><a href="https://is.gd/gn_intro">https://is.gd/gn_intro</a></li> <li>Basic //build directory for GN-based projects <ul> <li><a href="https://github.com/timniederhausen/gn-build">https://github.com/timniederhausen/gn-build</a></li> </ul> </li> </ul> </li> <li>ninjatracing: Convert .ninja_log files to chrome's about:tracing format. <ul> <li><a href="https://github.com/nico/ninjatracing">https://github.com/nico/ninjatracing</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/nomic/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nomic/index.html <p><a href="https://www.nomic.ai/gpt4all">Website</a> | Commercial/closed</p> https://tedneward.github.io/Research/tools/markdoc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/markdoc/index.html <p><a href="https://markdoc.io/">Website</a> | <a href="https://github.com/markdoc/markdoc">Source</a></p> <p>Appears to have tools to render to an AST, which I find intriguing....</p> <p><code>npm install @markdoc/markdoc</code></p> <p>then</p> <pre><code>const Markdoc = require('@markdoc/markdoc'); // or import Markdoc from '@markdoc/markdoc'; const doc = ` # Markdoc README {% image src="/logo.svg" /%} `; const ast = Markdoc.parse(doc); const content = Markdoc.transform(ast); return Markdoc.renderers.react(content, React); </code></pre> https://tedneward.github.io/Research/tools/mdbook/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mdbook/index.html <p><a href="https://rust-lang.github.io/mdBook/">Website</a> | <a href="https://github.com/rust-lang/mdBook">Source</a></p> https://tedneward.github.io/Research/tools/meson/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/meson/index.html <p><a href="http://mesonbuild.com/">Website</a> | <a href="https://mesonbuild.com/Videos.html">Videos</a> | <a href="https://github.com/mesonbuild/meson">Github</a></p> <p>Getting started with Meson build system and C++: <a href="https://medium.com/@germandiagogomez/getting-started-with-meson-build-system-and-c-83270f444bee">Part 1</a> | <a href="https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-2-58150354ff17">Part 2</a> | <a href="https://medium.com/@germandiagogomez/getting-started-with-meson-in-c-part-3-70b9bc419957">Part 3</a> | <a href="https://medium.com/@germandiagogomez/getting-started-with-meson-part-4-8bceec6149e1">Part 4</a></p> https://tedneward.github.io/Research/tools/mint/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mint/index.html <p><code>brew install mint</code> | <a href="https://github.com/yonaskolb/Mint">Source</a></p> https://tedneward.github.io/Research/tools/mockoon/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/mockoon/index.html <p><a href="https://mockoon.com">Website</a> | <a href="https://github.com/mockoon/mockoon">Source</a></p> https://tedneward.github.io/Research/tools/msbuild/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/msbuild/index.html <p><a href="https://github.com/Microsoft/msbuild">Website</a> | <a href="https://docs.microsoft.com/en-us/visualstudio/msbuild/">Docs</a></p> <ul> <li>MSBuild (Visual C++) - <a href="https://docs.microsoft.com/en-us/cpp/build/msbuild-visual-cpp">https://docs.microsoft.com/en-us/cpp/build/msbuild-visual-cpp</a></li> <li>Make VC++ Compiles Fast Through Parallel Compilation <ul> <li><a href="https://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/">https://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/</a></li> </ul> </li> <li>Microsoft AnyBuild <ul> <li>Remote build execution technology for use in MSBuild, Microsoft Build Accelerator, and other build engines</li> <li><a href="https://github.com/microsoft/AnyBuild">https://github.com/microsoft/AnyBuild</a></li> </ul> </li> <li>MSBuild (Visual C++) - <a href="https://docs.microsoft.com/en-us/cpp/build/msbuild-visual-cpp">https://docs.microsoft.com/en-us/cpp/build/msbuild-visual-cpp</a></li> <li>MSBuild Binary and Structured Log Viewer - <a href="http://msbuildlog.com/">http://msbuildlog.com/</a></li> <li>MSBuildEditor <ul> <li>The MSBuild Editor extension provides improved support for editing MSBuild files in Visual Studio and Visual Studio for Mac.</li> <li><a href="https://github.com/mhutch/MonoDevelop.MSBuildEditor">https://github.com/mhutch/MonoDevelop.MSBuildEditor</a></li> </ul> </li> <li>Build Tools <ul> <li>build native and managed MSBuild-based applications without requiring the Visual Studio IDE. There are options to install the Visual C++ compilers and libraries, MFC, ATL, and C++/CLI support, and .NET and .NET Core support.</li> <li><a href="http://aka.ms/buildtools">http://aka.ms/buildtools</a></li> <li>GoingNative 63: C++ Build Tools - <a href="https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-63-CPP-Build-Tools">https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-63-CPP-Build-Tools</a></li> </ul> </li> <li><a href="http://www.reedbeta.com/blog/custom-toolchain-with-msbuild/">Using A Custom Toolchain In Visual Studio With MSBuild</a></li> <li><a href="http://miken-1gam.blogspot.com/2013/01/visual-studio-and-custom-build-rules.html">Visual Studio and Custom Build Rules</a></li> <li><a href="https://github.com/BenVillalobos/msbuildism">"MSBuild is complicated. Let's lift the veil on it."</a></li> </ul> https://tedneward.github.io/Research/tools/n8n/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/n8n/index.html <p><a href="https://n8n.io">Website</a> | <a href="https://github.com/n8n-io/n8n">Source</a> | <a href="https://docs.n8n.io/">Docs</a> | <a href="https://n8n.io/integrations">400+ Integrations</a> | <a href="https://n8n.io/workflows">Example Workflows</a> | <a href="https://docs.n8n.io/advanced-ai/">AI &amp; LangChain Guide</a> | <a href="https://community.n8n.io/">Community Forum</a> | <a href="https://community.n8n.io/c/tutorials/28">Community Tutorials</a></p> <p>n8n is a workflow automation platform that gives technical teams the flexibility of code with the speed of no-code. With 400+ integrations, native AI capabilities, and a fair-code license, n8n lets you build powerful automations while maintaining full control over your data and deployments.</p> <h2>Quick Start</h2> <p>Try n8n instantly with <a href="https://docs.n8n.io/hosting/installation/npm/">npx</a> (requires <a href="https://nodejs.org/en/">Node.js</a>):</p> <pre><code>npx n8n </code></pre> <p>Or deploy with <a href="https://docs.n8n.io/hosting/installation/docker/">Docker</a>:</p> <pre><code>docker volume create n8n_data docker run -it --rm --name n8n -p 5678:5678 -v n8n_data:/home/node/.n8n docker.n8n.io/n8nio/n8n </code></pre> <p>Access the editor at <a href="http://localhost:5678/">http://localhost:5678</a></p> https://tedneward.github.io/Research/tools/nirlauncher/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/nirlauncher/index.html <p><a href="https://launcher.nirsoft.net/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.makeuseof.com/portable-windows-toolkit-fits-on-usb-replaces-apps/">https://www.makeuseof.com/portable-windows-toolkit-fits-on-usb-replaces-apps/</a></li> </ul> https://tedneward.github.io/Research/tools/notebooklm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/notebooklm/index.html <p><a href="https://notebooklm.google.com/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/learn-how-to-self-host-llm-with-notebooklm/">How NotebookLM made self-hosting an LLM easier than I ever expected</a></li> <li><a href="https://www.fastcompany.com/91467915/notebooklm-guide-google-docs">The complete guide to NotebookLM</a></li> <li><a href="https://www.makeuseof.com/i-tried-audio-video-overviews-notebooklm/">I gave in and tried NotebookLM's popular features and it forever changed how I learn</a></li> <li><a href="https://pub.towardsai.net/the-notebooklm-workflow-that-changed-how-i-learn-any-technology-373f430a17e5">The NotebookLM Workflow That Changed How I Learn Any Technology</a></li> </ul> https://tedneward.github.io/Research/tools/gnu/flex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gnu/flex/index.html <p><a href="https://github.com/westes/flex/">GNU Website</a></p> https://tedneward.github.io/Research/tools/gollum/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gollum/index.html <p><a href="https://github.com/gollum/gollum">Source</a></p> https://tedneward.github.io/Research/tools/grav/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/grav/index.html <p><a href="https://getgrav.org/">Website</a> | <a href="https://github.com/getgrav/grav">Source</a></p> https://tedneward.github.io/Research/tools/hatch/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hatch/index.html <p><a href="https://hatch.pypa.io/latest">Website</a></p> https://tedneward.github.io/Research/tools/hoarder/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hoarder/index.html <p><a href="https://hoarder.app/">Website</a><br> <a href="https://github.com/hoarder-app/hoarder">Source</a></p> https://tedneward.github.io/Research/tools/html-tools/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/html-tools/index.html <p><a href="https://simonwillison.net/2025/Dec/10/html-tools/">Article</a> | <a href="https://tools.simonwillison.net/">"over 150 of them"</a></p> <p><em>There's... a lot to unpack here. In a good way.</em></p> https://tedneward.github.io/Research/tools/ida-pro/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ida-pro/index.html <p><a href="https://hex-rays.com/ida-pro/">Website</a> | Commercial with free download</p> https://tedneward.github.io/Research/tools/imagor/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/imagor/index.html <p><a href="https://imagor.net/">Website</a> | <a href="https://github.com/cshum/imagor-studio">Source</a></p> <h2>Getting Started</h2> <p>Docker: <code>docker run -p 8000:8000 --rm -v $(pwd)/imagor-studio-data:/app/data -v ~/Pictures:/app/gallery -e DATABASE_URL="sqlite:///app/data/imagor-studio.db" shumc/imagor-studio</code></p> <h2>Reading</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://www.xda-developers.com/imagor-self-hosted-image-gallery-and-editor-running-browser/">Imagor is a self-hosted image gallery and editor, running entirely in your browser</a></li> </ul> https://tedneward.github.io/Research/tools/instrew/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/instrew/index.html <p><a href="https://github.com/aengelke/instrew">Source</a> | <a href="https://conf.researchr.org/details/vee-2020/vee-2020-papers/5/Instrew-Leveraging-LLVM-for-High-Performance-Dynamic-Binary-Instrumentation">Paper</a> (by Alexis Engelke, Martin Schulz; VEE 2020)</p> https://tedneward.github.io/Research/tools/jarchitect/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jarchitect/index.html <p><a href="https://www.jarchitect.com/">Website</a> | Commercial (free trial)</p> https://tedneward.github.io/Research/tools/jc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jc/index.html <p><a href="https://github.com/kellyjonbrazil/jc">Source</a> | <a href="https://blog.kellybrazil.com/2019/11/26/bringing-the-unix-philosophy-to-the-21st-century/">"Bringing the Unix Philosophy to the 21st Century"</a></p> <p>Ironically, this is what <a href="/languages/powershell">PowerShell</a> took as a fundamental principle: a pipeline/stream of objects, rather than just JSON.</p> https://tedneward.github.io/Research/tools/jekyll/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jekyll/index.html <p><a href="https://jekyllrb.com/">Website</a></p> https://tedneward.github.io/Research/tools/jqassistant/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jqassistant/index.html <p><a href="https://jqassistant.org/">Website</a> | <a href="https://github.com/jqassistant">Source</a></p> <p>It is centered around three main use cases:</p> <ul> <li>Easily carry out <a href="https://en.wikipedia.org/wiki/Software_analytics">Software Analytics</a> to gain insights into your software systems</li> <li>Continuously verify the correct implementation of targeted design and architecture using provided or user-defined rules to reach defined quality goals</li> <li>Automatically enable <a href="https://medium.com/geekculture/living-documentation-brief-history-and-evolution-of-the-concept-4492fafb5d7">Living Documentation</a> by generating documentation from the code and validating documentation against the implementation to avoid the documentation-code-gap and confusion</li> </ul> https://tedneward.github.io/Research/tools/kalk-dev/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/kalk-dev/index.html <p><a href="https://kalk.dev/">Website</a> | <a href="https://github.com/xoofx/kalk/">Source</a> | <a href="https://kalk.dev/doc/">Docs</a></p> https://tedneward.github.io/Research/tools/komodo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/komodo/index.html <p><a href="http://www.komodoide.com/komodo-edit">Website</a> | <a href="https://github.com/Komodo/KomodoEdit">Source</a></p> https://tedneward.github.io/Research/tools/lazydocker/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lazydocker/index.html <p><a href="https://github.com/jesseduffield/lazydocker">Source</a></p> https://tedneward.github.io/Research/tools/lincheck/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lincheck/index.html <p><a href="https://kotlinlang.org/docs/lincheck-guide.html">Website</a> | JetBrains (no source?)</p> https://tedneward.github.io/Research/tools/llvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/llvm/index.html <p><a href="https://llvm.org">Website</a> | <a href="https://github.com/llvm/llvm-project/">Source</a></p> <ul> <li>Clang documentation - <a href="http://clang.llvm.org/docs/">http://clang.llvm.org/docs/</a></li> <li>LLVM documentation - <a href="http://llvm.org/docs/">http://llvm.org/docs/</a></li> </ul> https://tedneward.github.io/Research/tools/gnu/make/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gnu/make/index.html <p>The original build tool for C/C++ source systems.</p> <h2><a href="https://learnxinyminutes.com/docs/make/">Learn X in Y Minutes: make</a></h2> <p>A Makefile defines a graph of rules for creating a target (or targets). Its purpose is to do the minimum amount of work needed to update a target to the most recent version of the source.</p> <p>There are many varieties of make in existence, however this article assumes that we are using GNU make which is the standard on Linux.</p> <pre><code class="language-make"> # Comments can be written like this. # File should be named Makefile and then can be run as `make &lt;target&gt;`. # Otherwise we use `make -f "filename" &lt;target&gt;`. # Warning - only use TABS to indent in Makefiles, never spaces! #----------------------------------------------------------------------- # Basics #----------------------------------------------------------------------- # Rules are of the format # target: &lt;prerequisite&gt; # where prerequisites are optional. # A rule - this rule will only run if file0.txt doesn't exist. file0.txt: echo "foo" &gt; file0.txt # Even comments in these 'recipe' sections get passed to the shell. # Try `make file0.txt` or simply `make` - first rule is the default. # This rule will only run if file0.txt is newer than file1.txt. file1.txt: file0.txt cat file0.txt &gt; file1.txt # use the same quoting rules as in the shell. @cat file0.txt &gt;&gt; file1.txt # @ stops the command from being echoed to stdout. -@echo 'hello' # - means that make will keep going in the case of an error. # Try `make file1.txt` on the commandline. # A rule can have multiple targets and multiple prerequisites file2.txt file3.txt: file0.txt file1.txt touch file2.txt touch file3.txt # Make will complain about multiple recipes for the same rule. Empty # recipes don't count though and can be used to add new dependencies. #----------------------------------------------------------------------- # Phony Targets #----------------------------------------------------------------------- # A phony target. Any target that isn't a file. # It will never be up to date so make will always try to run it. all: maker process # We can declare things out of order. maker: touch ex0.txt ex1.txt # Can avoid phony rules breaking when a real file has the same name by .PHONY: all maker process # This is a special target. There are several others. # A rule with a dependency on a phony target will always run ex0.txt ex1.txt: maker # Common phony targets are: all make clean install ... #----------------------------------------------------------------------- # Automatic Variables &amp; Wildcards #----------------------------------------------------------------------- process: file*.txt #using a wildcard to match filenames @echo $^ # $^ is a variable containing the list of prerequisites @echo $@ # prints the target name #(for multiple target rules, $@ is whichever caused the rule to run) @echo $&lt; # the first prerequisite listed @echo $? # only the dependencies that are out of date @echo $+ # all dependencies including duplicates (unlike normal) #@echo $| # all of the 'order only' prerequisites # Even if we split up the rule dependency definitions, $^ will find them process: ex1.txt file0.txt # ex1.txt will be found but file0.txt will be deduplicated. #----------------------------------------------------------------------- # Patterns #----------------------------------------------------------------------- # Can teach make how to convert certain files into other files. %.png: %.svg inkscape --export-png $^ # Pattern rules will only do anything if make decides to create the # target. # Directory paths are normally ignored when matching pattern rules. But # make will try to use the most appropriate rule available. small/%.png: %.svg inkscape --export-png --export-dpi 30 $^ # make will use the last version for a pattern rule that it finds. %.png: %.svg @echo this rule is chosen # However make will use the first pattern rule that can make the target %.png: %.ps @echo this rule is not chosen if *.svg and *.ps are both present # make already has some pattern rules built-in. For instance, it knows # how to turn *.c files into *.o files. # Older makefiles might use suffix rules instead of pattern rules .png.ps: @echo this rule is similar to a pattern rule. # Tell make about the suffix rule .SUFFIXES: .png #----------------------------------------------------------------------- # Variables #----------------------------------------------------------------------- # aka. macros # Variables are basically all string types name = Ted name2="Sarah" echo: @echo $(name) @echo ${name2} @echo $name # This won't work, treated as $(n)ame. @echo $(name3) # Unknown variables are treated as empty strings. # There are 4 places to set variables. # In order of priority from highest to lowest: # 1: commandline arguments # 2: Makefile # 3: shell environment variables - make imports these automatically. # 4: make has some predefined variables name4 ?= Jean # Only set the variable if environment variable is not already defined. override name5 = David # Stops commandline arguments from changing this variable. name4 +=grey # Append values to variable (includes a space). # Pattern-specific variable values (GNU extension). echo: name2 = Sara # True within the matching rule # and also within its remade recursive dependencies # (except it can break when your graph gets too complicated!) # Some variables defined automatically by make. echo_inbuilt: echo $(CC) echo ${CXX} echo $(FC) echo ${CFLAGS} echo $(CPPFLAGS) echo ${CXXFLAGS} echo $(LDFLAGS) echo ${LDLIBS} #----------------------------------------------------------------------- # Variables 2 #----------------------------------------------------------------------- # The first type of variables are evaluated each time they are used. # This can be expensive, so a second type of variable exists which is # only evaluated once. (This is a GNU make extension) var := hello var2 ::= $(var) hello #:= and ::= are equivalent. # These variables are evaluated procedurally (in the order that they # appear), thus breaking with the rest of the language ! # This doesn't work var3 ::= $(var4) and good luck var4 ::= good night #----------------------------------------------------------------------- # Functions #----------------------------------------------------------------------- # make has lots of functions available. sourcefiles = $(wildcard *.c */*.c) objectfiles = $(patsubst %.c,%.o,$(sourcefiles)) # Format is $(func arg0,arg1,arg2...) # Some examples ls: * src/* @echo $(filter %.txt, $^) @echo $(notdir $^) @echo $(join $(dir $^),$(notdir $^)) #----------------------------------------------------------------------- # Directives #----------------------------------------------------------------------- # Include other makefiles, useful for platform specific code include foo.mk sport = tennis # Conditional compilation report: ifeq ($(sport),tennis) @echo 'game, set, match' else @echo "They think it's all over; it is now" endif # There are also ifneq, ifdef, ifndef foo = true ifdef $(foo) bar = 'hello' endif </code></pre> <hr> <ul> <li><a href="http://swcarpentry.github.io/make-novice/">software carpentry tutorial</a></li> <li>A more helpful makefile <ul> <li><a href="https://jakemccrary.com/blog/2018/12/27/a-more-helpful-makefile/">https://jakemccrary.com/blog/2018/12/27/a-more-helpful-makefile/</a></li> </ul> </li> <li>A Super-Simple Makefile for Medium-Sized C/C++ Projects <ul> <li><a href="https://spin.atomicobject.com/2016/08/26/makefile-c-projects/">https://spin.atomicobject.com/2016/08/26/makefile-c-projects/</a></li> </ul> </li> <li>A Tutorial on Portable Makefiles <ul> <li><a href="http://nullprogram.com/blog/2017/08/20/">http://nullprogram.com/blog/2017/08/20/</a></li> </ul> </li> <li>Makefile Tutorial <ul> <li><a href="https://gist.github.com/isaacs/62a2d1825d04437c6f08">https://gist.github.com/isaacs/62a2d1825d04437c6f08</a></li> </ul> </li> <li>Notes for new Make users <ul> <li><a href="http://gromnitsky.users.sourceforge.net/articles/notes-for-new-make-users/">http://gromnitsky.users.sourceforge.net/articles/notes-for-new-make-users/</a></li> </ul> </li> <li>Propositions as Filenames, Builds as Proofs: The Essence of Make <ul> <li><a href="https://bentnib.org/posts/2015-04-17-propositions-as-filenames-essence-of-make.html">https://bentnib.org/posts/2015-04-17-propositions-as-filenames-essence-of-make.html</a></li> </ul> </li> <li>Well documented Makefiles (available via <code>make help</code>) <ul> <li><a href="https://suva.sh/posts/well-documented-makefiles/">https://suva.sh/posts/well-documented-makefiles/</a></li> </ul> </li> </ul> <h4>GNU Make</h4> <ul> <li>GNU Make - <a href="http://www.gnu.org/software/make/">http://www.gnu.org/software/make/</a> <ul> <li>Manual - <a href="https://www.gnu.org/software/make/manual/">https://www.gnu.org/software/make/manual/</a></li> </ul> </li> <li>An Empirical Analysis of GNU Make in Open Source Projects <ul> <li>2017 PhD Thesis; <a href="http://research.cs.queensu.ca/~doug/research.html">Douglas Martin</a></li> <li><a href="https://qspace.library.queensu.ca/handle/1974/15767">https://qspace.library.queensu.ca/handle/1974/15767</a></li> <li>Makefile Corpus - <a href="http://research.cs.queensu.ca/~doug/files/MakeEgs.tar.gz">http://research.cs.queensu.ca/~doug/files/MakeEgs.tar.gz</a></li> <li>Makefile Framework - <a href="http://research.cs.queensu.ca/~doug/files/MakeFramework.zip">http://research.cs.queensu.ca/~doug/files/MakeFramework.zip</a></li> </ul> </li> <li>An Empirical Study of Unspecified Dependencies in Make-Based Build Systems <ul> <li>Empirical Software Engineering (2017)</li> <li>Cor-Paul Bezemer, Shane McIntosh, Bram Adams, Daniel M. German, Ahmed E. Hassan</li> <li><a href="https://doi.org/10.1007/s10664-017-9510-8">https://doi.org/10.1007/s10664-017-9510-8</a></li> <li><a href="http://rebels.ece.mcgill.ca/journalpaper/2017/03/07/an-empirical-study-of-unspecified-dependencies-in-make-based-build-systems.html">http://rebels.ece.mcgill.ca/journalpaper/2017/03/07/an-empirical-study-of-unspecified-dependencies-in-make-based-build-systems.html</a></li> <li><a href="http://sail.cs.queensu.ca/Downloads/EMSE2017_AnEmpiricalStudyOfUnspecifiedDependenciesInMake-BasedBuildSystems.pdf">http://sail.cs.queensu.ca/Downloads/EMSE2017_AnEmpiricalStudyOfUnspecifiedDependenciesInMake-BasedBuildSystems.pdf</a></li> <li>Slides: <a href="https://www.slideshare.net/corpaulbezemer/an-empirical-study-of-unspecified-dependencies-in-makebased-build-systems">https://www.slideshare.net/corpaulbezemer/an-empirical-study-of-unspecified-dependencies-in-makebased-build-systems</a></li> </ul> </li> <li>GNU make - Paul D. Smith - <a href="http://make.mad-scientist.net/">http://make.mad-scientist.net/</a> <ul> <li>Paul Smith's Rules of Makefiles - <a href="http://make.mad-scientist.net/papers/rules-of-makefiles/">http://make.mad-scientist.net/papers/rules-of-makefiles/</a></li> <li>Deferred Simple Variable Expansion - <a href="http://make.mad-scientist.net/deferred-simple-variable-expansion/">http://make.mad-scientist.net/deferred-simple-variable-expansion/</a></li> </ul> </li> <li>GNU Make articles - John Graham-Cumming <ul> <li><a href="http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html">http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-articles.html</a></li> <li><a href="https://www.cmcrossroads.com/keyword/type/3037?type=article">https://www.cmcrossroads.com/keyword/type/3037?type=article</a></li> <li><a href="https://www.cmcrossroads.com/keyword/type/3169?type=article">https://www.cmcrossroads.com/keyword/type/3169?type=article</a></li> </ul> </li> <li>GNU Make meets file names with spaces in them - <a href="https://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them">https://www.cmcrossroads.com/article/gnu-make-meets-file-names-spaces-them</a></li> <li>GNU Make Standard Library - <a href="http://gmsl.sourceforge.net/">http://gmsl.sourceforge.net/</a></li> <li>HOWTO: Intro to GNU make variables - <a href="https://blog.melski.net/2015/01/07/howto-intro-to-gnu-make-variables/">https://blog.melski.net/2015/01/07/howto-intro-to-gnu-make-variables/</a></li> <li>Make is (probably) fine <ul> <li><a href="https://blog.yossarian.net/2019/04/23/Make-is-probably-fine">https://blog.yossarian.net/2019/04/23/Make-is-probably-fine</a></li> </ul> </li> <li>Make it simple: An empirical analysis of GNU Make feature use in open source projects <ul> <li>International Conference on Program Comprehension 2015</li> <li>Douglas H Martin, James R Cordy, Bram Adams, Giulio Antoniol</li> <li><a href="http://maroon.cs.queensu.ca/home/cordy/Papers/MCAA_ICPC15_Makefiles.pdf">http://maroon.cs.queensu.ca/home/cordy/Papers/MCAA_ICPC15_Makefiles.pdf</a></li> </ul> </li> <li>Managing Projects with GNU Make <ul> <li><a href="http://www.wanderinghorse.net/computing/make/">http://www.wanderinghorse.net/computing/make/</a></li> <li><a href="http://www.oreilly.com/openbook/make3/book/index.csp">http://www.oreilly.com/openbook/make3/book/index.csp</a></li> <li><a href="https://notendur.hi.is/jonasson/software/make-book/">https://notendur.hi.is/jonasson/software/make-book/</a></li> </ul> </li> <li>Properly using GNU-Make - <a href="https://slashvar.github.io/2017/02/13/using-gnu-make.html">https://slashvar.github.io/2017/02/13/using-gnu-make.html</a></li> <li>Remake – GNU Make with comprehensible tracing and a debugger <ul> <li><a href="http://bashdb.sourceforge.net/remake/">http://bashdb.sourceforge.net/remake/</a></li> <li><a href="https://github.com/rocky/remake">https://github.com/rocky/remake</a></li> </ul> </li> <li>learn C the hard way <a href="http://c.learncodethehardway.org/book/ex2.html">ex2</a> <a href="http://c.learncodethehardway.org/book/ex28.html">ex28</a></li> </ul> <h4>NMake</h4> <p><a href="https://docs.microsoft.com/en-us/cpp/build/nmake-reference">The Microsoft Program Maintenance Utility (NMAKE.EXE)</a></p> https://tedneward.github.io/Research/tools/gradle/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gradle/index/index.html <p><a href="https://gradle.org/">Website</a> | <a href="https://github.com/gradle/gradle">Source</a> | <a href="https://plugins.gradle.org/">Plugin Portal</a></p> <p>Docs: <a href="https://docs.gradle.org/current/userguide/userguide.html">User Guide</a> | <a href="https://docs.gradle.org/current/dsl/">DSL Reference</a> | <a href="https://discuss.gradle.org/">Forums</a></p> <p><a href="https://thoughts-on-cpp.com/2019/04/10/introduction-into-c-builds-with-gradle/">Introduction into C++ Builds with Gradle</a></p> <p><a href="https://imperceptiblethoughts.com/shadow/introduction/">Shadow plugin</a>: "Shadow is a Gradle plugin for combining a project's dependency classes and resources into a single output Jar. The combined Jar is often referred to a fat-jar or uber-jar. ... Shadowing a project output has 2 major use cases: (1) Creating an executable JAR distribution, <a href="2">and</a> Bundling and relocating common dependencies in libraries to avoid classpath conflicts."</p> <h3>Reading</h3> <ul> <li><a href="http://spring.io/guides/gs/gradle/">Building Java Projects with Gradle</a></li> <li><a href="https://www.syncfusion.com/ebooks/gradle_succinctly">Gradle Succinctly</a> - José Roberto Olivas Mendoza</li> <li><a href="https://docs.gradle.org/current/userguide/userguide.html">Gradle User Guide</a> - Hans Dockter, Adam Murdoch (<a href="https://docs.gradle.org/current/userguide/userguide.pdf">PDF</a>)</li> </ul> <h3>Notes</h3> <p>Gradle uses a 1:1 relationship of a build script to a Project instance. Most "magic" comes from the use of plugins (listed below). Thus, the world's simplest Gradle build:</p> <pre><code>// build.gradle.kts println("Hello Gradle") </code></pre> <p>... when run with <code>gradle -b build.gradle.kts</code>:</p> <pre><code>~/Projects/Exploration.git/Gradle % gradle -b build.gradle.kts &gt; Configure project : Hello Gradle &gt; Task :help Welcome to Gradle 8.10. To run a build, run gradle &lt;task&gt; ... To see a list of available tasks, run gradle tasks To see more detail about a task, run gradle help --task &lt;task&gt; To see a list of command-line options, run gradle --help For more detail on using Gradle, see https://docs.gradle.org/8.10/userguide/command_line_interface.html For troubleshooting, visit https://help.gradle.org Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0. You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins. For more on this, please refer to https://docs.gradle.org/8.10/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation. BUILD SUCCESSFUL in 843ms 1 actionable task: 1 executed </code></pre> <h2>Common tasks</h2> <h3>Execute "run" with command-line arguments</h3> <p><code>gradle run --args="one two three \"four is a collection of words\" five"</code> (must have all arguments in a single double-quoted string)</p> <h2>Customizing</h2> <h3>Keyword Expansion</h3> <p>Found in Chapter 1 of <em>Gradle Beyond the Basics</em> (OReilly, Tim Berglund)</p> <pre><code>// I think this is Groovy syntax versionId = '1.6' task copyProductionConfig(type: Copy) { from 'source' include 'config.properties' into 'build/war/WEB-INF/config' expand([ databaseHostname: 'db.company.com', version: versionId, buildNumber: (int)(Math.random() * 1000), date: new Date() ]) } </code></pre> <p>This will replace placeholders like the following:</p> <pre><code># # Application configuration file # hostname: ${databaseHostname} appVersion: ${version} locale: en_us initialConnections: 10 transferThrottle: 5400 queueTimeout: 30000 buildNumber: ${buildNumber} buildDate: ${date.format("yyyyMMdd'T'HHmmssZ")} </code></pre> <h3>Custom Task inside the build script</h3> <p>Wrote a custom task that invokes <code>javap</code> on compiled class files (<a href="https://github.com/tedneward/Demo-JVM-Bytecode/blob/main/JavaExamples/app/build.gradle.kts">here</a>):</p> <pre><code>open class Javap: DefaultTask() { @Internal var workingDir = "./build/classes/java/main" @Internal var outputDir: String = "" @Internal lateinit var classFiles: List&lt;String&gt; @Internal lateinit var javapArguments: List&lt;String&gt; @Internal var projectDir = project.projectDir @TaskAction fun runCommand() { logger.warn("Running 'disasm' replaces the marked javap files with newly-generated content!") for (classFile in classFiles) { val commandLine = ("javap " + javapArguments.joinToString(" ") + " " + classFile) try { val workingDir = File(projectDir, workingDir) val outputDir = File(projectDir, outputDir) val parts = commandLine.split("\\s".toRegex()) // Transform the classFile string/path to a dotted name val dottedClassFile = classFile.split("/").joinToString(".") val outputFile = File(outputDir, dottedClassFile + ".bytecode") val proc = ProcessBuilder(*parts.toTypedArray()) .directory(workingDir) .redirectOutput(outputFile) .redirectError(ProcessBuilder.Redirect.PIPE) .start() proc.waitFor(60, TimeUnit.MINUTES) println("Success processing ${classFile}") } catch(e: Exception) { e.printStackTrace() println("Failure processing ${classFile}: ${e.message}") } } } } tasks.register&lt;Javap&gt;("disasm") { group = "build" description = "Produce disassembly listings of Java code" javapArguments = listOf("-v") outputDir = "" classFiles = listOf( "com/newardassociates/demo/App.class", "com/newardassociates/demo/Greeter.class", "com/newardassociates/demo/Math.class", "com/newardassociates/demo/Outer.class", "com/newardassociates/demo/Outer\$Inner.class", "com/newardassociates/demo/StringConcat.class", "com/newardassociates/demo/Varargs.class", ) dependsOn("compileJava") } </code></pre> <ul> <li>@TaskAction definitely is the magic to make Gradle invoke the task</li> <li>@Internal is required on the properties inside the task--Gradle apparently really wants to know whether it's an input, output, or internal property. I still need to figure out input and output properties on a custom task.</li> <li>I think <code>group</code> and <code>description</code> are inherited from <code>DefaultTask</code>, so I should probably figure out how to pass those in a constructor or something</li> <li>There's probably a better way of dealing with multiple files, but this was sufficient for the Bytecode demos project.</li> <li>I should probably move those "println" statements over to use the logger instead</li> </ul> <h3>Custom Task compiled as part of the build</h3> <h3>Custom Task in a standalone JAR</h3> <hr> <h2>Boilerplates</h2> <ul> <li><a href="https://github.com/nenick/android-gradle-template">android-gradle-template</a> - Template project for developing Android app.</li> <li><a href="https://github.com/vert-x/vertx-gradle-template">vertx-gradle-template</a> - Template project for developing Vert.x module.</li> <li><a href="https://github.com/int128/gradle-plugin-starter">gradle-plugin-starter</a> - Template project for developing Gradle plugin.</li> <li><a href="https://github.com/RallySoftware/gatling-with-gradle">gatling-with-gradle</a> - Sample project that demonstrates how to automate load testing with <a href="http://gatling.io/">Gatling</a>.</li> </ul> <h2>Resources</h2> https://tedneward.github.io/Research/tools/grimoirelab/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/grimoirelab/index.html <p><a href="https://chaoss.github.io/grimoirelab/">Website</a> | <a href="http://github.com/chaoss/grimoirelab">Source</a> | <a href="https://chaoss.github.io/grimoirelab-tutorial/">Tutorial</a></p> <p>GrimoireLab is a CHAOSS toolset for software development analytics. It includes a coordinated set of tools to retrieve data from systems used to support software development (repositories), store it in databases, enrich it by computing relevant metrics, and make it easy to run analytics and visualizations on it.</p> <p>Currently, GrimoireLab toolkit is organized in the following repositories:</p> <ul> <li>Data retrieval related components: <ul> <li><a href="https://github.com/chaoss/grimoirelab-perceval">Perceval</a>: retrieval of data from data sources <ul> <li><a href="https://github.com/chaoss/grimoirelab-perceval-opnfv">Perceval (bundle for OPNFV)</a></li> <li><a href="https://github.com/chaoss/grimoirelab-perceval-mozilla">Perceval (bundle for Mozilla)</a></li> <li><a href="https://github.com/chaoss/grimoirelab-perceval-puppet">Perceval (bundle for Puppet)</a></li> <li><a href="https://github.com/chaoss/grimoirelab-perceval-weblate">Perceval (bundle for Weblate)</a></li> </ul> </li> <li><a href="https://github.com/chaoss/grimoirelab-graal">Graal</a>: source data analysis with external tools</li> <li><a href="https://github.com/chaoss/grimoirelab-kingarthur">KingArthur</a>: batch processing for massive retrieval</li> </ul> </li> <li>Data enrichment related components: <ul> <li><a href="https://github.com/chaoss/grimoirelab-elk">GrimoireElk</a>: storage and enrichment of data</li> <li><a href="https://github.com/chaoss/grimoirelab-cereslib">Cereslib</a>: generic data processor</li> <li><a href="https://github.com/chaoss/grimoirelab-sortinghat">SortingHat</a>: identity management</li> </ul> </li> <li>Data consumption related components: <ul> <li><a href="https://github.com/chaoss/grimoirelab-sigils">Sigils</a>: visualizations and dashboards</li> <li><a href="https://github.com/chaoss/grimoirelab-kidash">Kidash</a>: visualizations and dashboards manager</li> <li><a href="https://github.com/chaoss/grimoirelab-manuscripts">Manuscripts</a>: reporting</li> </ul> </li> <li>Platform management, orchestration, and common utils: <ul> <li><a href="https://github.com/chaoss/grimoirelab-mordred">Mordred</a>: orchestration</li> <li><a href="https://github.com/chaoss/grimoirelab-toolkit">GrimoireLab Toolkit</a>: common utilities</li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/hevea/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hevea/index.html <p><a href="https://hevea.inria.fr/">Website</a> | <a href="https://github.com/maranget/hevea">Source</a> | <a href="https://hevea.inria.fr/distri/hevea-2.37-manual.pdf">Docs</a></p> https://tedneward.github.io/Research/tools/homebrew/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/homebrew/index.html <p><a href="https://brew.sh/">Website</a> | <a href="https://github.com/Homebrew/brew">Source</a></p> <p>Install script: <code>/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"</code></p> <h2>Notes</h2> <p>Generate a Brewfile (all the Homebrew-installed packages) by running:</p> <pre><code>brew bundle dump --force </code></pre> <p><code>brew bundle dump</code> also includes installed VS Code extensions!</p> <p>Install everything in the Brewfile by running:</p> <pre><code>brew bundle install --file=./Brewfile </code></pre> https://tedneward.github.io/Research/tools/huginn/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/huginn/index.html <p><a href="https://github.com/huginn">Source</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/huginn-free-open-source-ifttt-alternative/">"Huginn is a free, open-source IFTTT alternative that's much more powerful"</a></li> </ul> https://tedneward.github.io/Research/tools/ikos/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ikos/index.html <p><a href="https://github.com/NASA-SW-VnV/ikos">Source</a></p> https://tedneward.github.io/Research/tools/imhex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/imhex/index.html <p><a href="https://imhex.werwolv.net/">Website</a> | <a href="https://github.com/WerWolv/ImHex">Source</a> | <a href="https://imhex.werwolv.net/docs/">Pattern Language Docs</a></p> https://tedneward.github.io/Research/tools/jacoco/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jacoco/index.html <p><a href="https://www.jacoco.org/jacoco/">Website</a> | <a href="https://github.com/jacoco/jacoco/">Source</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://blog.picnic.nl/how-to-find-dead-code-in-your-java-services-d167c8f22838">https://blog.picnic.nl/how-to-find-dead-code-in-your-java-services-d167c8f22838</a></li> </ul> https://tedneward.github.io/Research/tools/java-decompiler/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/java-decompiler/index.html <p><a href="https://java-decompiler.github.io/">Website</a> | <a href="https://github.com/java-decompiler">Source</a></p> <p>Features:</p> <ul> <li>JD-Core and JD-GUI are written in Java.</li> <li>JD-Core works with most current compilers including the following: <ul> <li>JDK 1.1.8, JDK 1.3.1, JDK 1.4.2, JDK 1.5.0, JDK 1.6.0, JDK 1.7.0, JDK 1.8.0, JDK 9.0.1, JDK 10.0.2</li> <li>jrockit90_150_06</li> <li>jikes-1.22</li> <li>harmony-jdk-r533500</li> <li>Eclipse Java Compiler v_677_R32x, 3.2.1 release</li> </ul> </li> <li> <p>JD-GUI supports Drag and Drop.</p> </li> <li>JD-GUI supports CLASS, JAR, WAR, EAR, AAR, KAR, JMOD &amp; ZIP files.</li> <li>JD-GUI displays color coded Java source code.</li> <li>JD-GUI allows you to browse the CLASS files and Java modules hierarchy.</li> <li>JD-GUI lets you drag and drop LOG files, decompile CLASS files, and display the line of code that appears in Java stack traces.</li> </ul> https://tedneward.github.io/Research/tools/jcpp/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jcpp/index.html <p><a href="https://github.com/shevek/jcpp">Source</a></p> https://tedneward.github.io/Research/tools/jflex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jflex/index.html <p>"A lexical analyzer generator takes as input a specification with a set of regular expressions and corresponding actions. It generates a program (a lexer) that reads input, matches the input against the regular expressions in the spec file, and runs the corresponding action if a regular expression matched. Lexers usually are the first front-end step in compilers, matching keywords, comments, operators, etc, and generating an input token stream for parsers. Lexers can also be used for many other purposes.</p> <p>"JFlex lexers are based on deterministic finite automata (DFAs). They are fast, without expensive backtracking.</p> <p>"JFlex is designed to work together with the LALR parser generator CUP by Scott Hudson, and the Java modification of Berkeley Yacc BYacc/J by Bob Jamison. It can also be used together with other parser generators like ANTLR or as a standalone tool."</p> <p><a href="https://jflex.de/">Website</a> | <a href="https://github.com/jflex-de/">Github</a></p> https://tedneward.github.io/Research/tools/jrnl/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jrnl/index.html <p><a href="https://jrnl.sh/en/stable/">Website</a> | <a href="https://github.com/jrnl-org/jrnl">Source</a></p> <p>Command-line composing and viewing of a "journal file" or files, which may be encrypted if desired.</p> <p>Config file default is <code>$HOME/.config/jrnl/jrnl.yaml</code> (macOS) or <code>%USERPROFILE%\.config\jrnl\jrnl.yaml</code> (Windows).</p> <h1>Basic usage</h1> <p><em>(from the GitHub documentation page)</em></p> <p><code>jrnl</code> has two modes: <strong>composing</strong> and <strong>viewing</strong>. Whenever you don't enter any arguments that start with a dash (<code>-</code>) or double-dash (<code>--</code>), you're in composing mode, meaning that you can write your entry on the command line.</p> <p>We intentionally break a convention on command line arguments: all arguments starting with a <em>single dash</em> (<code>-</code>) will <em>filter</em> your journal before viewing it. Filter arguments can be combined arbitrarily. Arguments with a <em>double dash</em> (<code>--</code>) will <em>control</em> how your journal is displayed or exported. Control arguments are mutually exclusive (i.e., you can only specify one way to display or export your journal at a time).</p> <p>For a list of commands, enter <code>jrnl --help</code>.</p> <h2>Composing Entries</h2> <p>Composing mode is entered by either starting <code>jrnl</code> without any arguments -- which will launch an external editor -- or by just writing an entry on the command line:</p> <pre><code class="language-text">jrnl today at 3am: I just met Steve Buscemi in a bar! What a nice guy. </code></pre> <p>!!! note<br> Most shells contain a certain number of reserved characters, such as <code>#</code> and<br> <code>*</code>. These characters, as well as unbalanced single or double quotation<br> marks, parentheses, and others, likely will cause problems. Although<br> reserved characters can be escaped using <code>\</code>, this is not ideal for<br> long-form writing. The solution: first enter <code>jrnl</code> and hit <code>return</code>. You<br> can then enter the text of your journal entry. Alternatively, you can <a href="./advanced.md">use<br> an external editor</a>.</p> <p>You can also import an entry directly from a file:</p> <pre><code class="language-sh">jrnl &lt; my_entry.txt </code></pre> <h3>Specifying Date and Time</h3> <p>If you don't specify a date and time (e.g., <code>jrnl finished writing letter to brother</code>), <code>jrnl</code> will create an entry using the current date and time. For retrospective entries, you can use a timestamp to tell <code>jrnl</code> where to put the entry. Timestamps can be entered using a variety of formats. Here are some that work:</p> <ul> <li>at 6am</li> <li>yesterday</li> <li>last monday</li> <li>sunday at noon</li> <li>2 march 2012</li> <li>7 apr</li> <li>5/20/1998 at 23:42</li> <li>2020-05-22T15:55-04:00</li> </ul> <p>If you don't use a timestamp, <code>jrnl</code> will create an entry using the current time. If you use a date only (no time), <code>jrnl</code> will use the default time specified in your <a href="./reference-config-file.md#default_hour-and-default_minute">configuration file</a>. Behind the scenes, <code>jrnl</code> reorganizes entries in chronological order.</p> <h3>Using Tags</h3> <p><code>jrnl</code> supports tags. The default tag symbol is <code>@</code> (largely because <code>#</code> is a reserved character). You can specify your own tag symbol in the <a href="./reference-config-file.md#tagsymbols">configuration file</a>. To use tags, preface the desired tag with the symbol:</p> <pre><code class="language-sh">jrnl Had a wonderful day at the @beach with @Tom and @Anna. </code></pre> <p>Although you can use capitals while tagging an entry, searches by tag are case-insensitive.</p> <p>There is no limit to how many tags you can use in an entry.</p> <h3>Starring Entries</h3> <p>To mark an entry as a favorite, simply "star" it using an asterisk (<code>*</code>):</p> <pre><code class="language-sh">jrnl last sunday *: Best day of my life. </code></pre> <p>If you don't want to add a date (i.e., you want the date to be entered as <em>now</em>), the following options are equivalent:</p> <ul> <li><code>jrnl *: Best day of my life.</code></li> <li><code>jrnl *Best day of my life.</code></li> <li><code>jrnl Best day of my life.*</code></li> </ul> <p>!!! note<br> Make sure that the asterisk (<code>*</code>) is <strong>not</strong> surrounded by whitespaces.<br> <code>jrnl Best day of my life! *</code> will not work because the <code>*</code> character has a<br> special meaning in most shells.</p> <h2>Viewing and Searching Entries</h2> <p><code>jrnl</code> can display entries in a variety of ways.</p> <p>To view all entries, enter:</p> <pre><code class="language-sh">jrnl -to today </code></pre> <p><code>jrnl</code> provides several filtering commands, prefaced by a single dash (<code>-</code>), that allow you to find a more specific range of entries. For example,</p> <pre><code class="language-sh">jrnl -n 10 </code></pre> <p>lists the ten most recent entries. <code>jrnl -10</code> is even more concise and works the same way. If you want to see all of the entries you wrote from the beginning of last year until the end of this past March, you would enter</p> <pre><code class="language-sh">jrnl -from "last year" -to march </code></pre> <p>Filter criteria that use more than one word require surrounding quotes (<code>""</code>).</p> <p>To see entries on a particular date, use <code>-on</code>:</p> <pre><code class="language-sh">jrnl -on yesterday </code></pre> <h3>Text Search</h3> <p>The <code>-contains</code> command displays all entries containing the text you enter after it. This may be helpful when you're searching for entries and you can't remember if you tagged any words when you wrote them.</p> <p>You may realize that you use a word a lot and want to turn it into a tag in all<br> of your previous entries.</p> <pre><code class="language-sh">jrnl -contains "dogs" --edit </code></pre> <p>opens your external editor so that you can add a tag symbol (<code>@</code> by default) to all instances of the word "dogs."</p> <h3>Filtering by Tag</h3> <p>You can filter your journal entries by tag. For example,</p> <pre><code class="language-sh">jrnl @pinkie @WorldDomination </code></pre> <p>displays all entries in which either <code>@pinkie</code> or <code>@WorldDomination</code> occurred. Tag filters can be combined with other filters:</p> <pre><code class="language-sh">jrnl -n 5 @pinkie -and @WorldDomination </code></pre> <p>displays the last five entries containing <em>both</em> <code>@pinkie</code> <em>and</em> <code>@worldDomination</code>. You can change which symbols you'd like to use for tagging in the <a href="./reference-config-file.md#tagsymbols">configuration file</a>.</p> <p>!!! note<br> Entering <code>jrnl @pinkie @WorldDomination</code> will display entries in which both<br> tags are present because, although no command line arguments are given, all<br> of the input strings look like tags. <code>jrnl</code> will assume you want to filter<br> by tag, rather than create a new entry that consists only of tags.</p> <p>To view a list of all tags in the journal, enter:</p> <pre><code class="language-sh">jrnl --tags </code></pre> <h3>Viewing Starred Entries</h3> <p>To display only your favorite (starred) entries, enter</p> <pre><code class="language-sh">jrnl -starred </code></pre> <h2>Editing Entries</h2> <p>You can edit entries after writing them. This is particularly useful when your journal file is encrypted. To use this feature, you need to have an external editor configured in your <a href="./reference-config-file.md#editor">configuration file</a>. You can also edit only the entries that match specific search criteria. For example,</p> <pre><code class="language-sh">jrnl -to 1950 @texas -and @history --edit </code></pre> <p>opens your external editor displaying all entries tagged with <code>@texas</code> and <code>@history</code> that were written before 1950. After making changes, save and close the file, and only those entries will be modified (and encrypted, if applicable).</p> <p>If you are using multiple journals, it's easy to edit specific entries from specific journals. Simply prefix the filter string with the name of the journal. For example,</p> <pre><code class="language-sh">jrnl work -n 1 --edit </code></pre> <p>opens the most recent entry in the 'work' journal in your external editor.</p> <h2>Deleting Entries</h2> <p>The <code>--delete</code> command opens an interactive interface for deleting entries. The date and title of each entry in the journal are presented one at a time, and you can choose whether to keep or delete each entry.</p> <p>If no filters are specified, <code>jrnl</code> will ask you to keep or delete each entry in the entire journal, one by one. If there are a lot of entries in the journal, it may be more efficient to filter entries before passing the <code>--delete</code> command.</p> <p>Here's an example. Say you have a journal into which you've imported the last 12 years of blog posts. You use the <code>@book</code> tag a lot, and for some reason you want to delete some, but not all, of the entries in which you used that tag, but only the ones you wrote at some point in 2004 or earlier. You're not sure which entries you want to keep, and you want to look through them before deciding. This is what you might enter:</p> <pre><code class="language-sh">jrnl -to 2004 @book --delete </code></pre> <p><code>jrnl</code> will show you only the relevant entries, and you can choose the ones you want to delete.</p> <p>You may want to delete <em>all</em> of the entries containing <code>@book</code> that you wrote in 2004 or earlier. If there are dozens or hundreds, the easiest way would be to use an external editor. Open an editor with the entries you want to delete...</p> <pre><code class="language-sh">jrnl -to 2004 @book --edit </code></pre> <p>...select everything, delete it, save and close, and all of those entries are removed from the journal.</p> <h2>Listing Journals</h2> <p>To list all of your journals:</p> <pre><code class="language-sh">jrnl --list </code></pre> <p>The journals displayed correspond to those specified in the <code>jrnl</code> <a href="./reference-config-file.md#journals">configuration file</a>.</p> https://tedneward.github.io/Research/tools/kapersky-rescue-disk/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/kapersky-rescue-disk/index.html <p><a href="https://www.kaspersky.co.in/downloads/free-rescue-disk">Download</a> but this URL doesn't seem to display anything?</p> https://tedneward.github.io/Research/tools/krakatau/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/krakatau/index.html <p><a href="https://github.com/Storyyeller/Krakatau">Source</a></p> https://tedneward.github.io/Research/tools/ledger/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ledger/index.html <p><a href="https://ledger-cli.org/index.html">Website</a> | <a href="https://git.ledger-cli.org/ledger">Source</a></p> https://tedneward.github.io/Research/tools/litmuskt/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/litmuskt/index.html <p><a href="https://github.com/Jetbrains-Research/litmuskt">Source</a></p> <h2>Reading</h2> <h3>Articles/Papers</h3> <ul> <li><a href="https://blog.jetbrains.com/research/2025/10/litmuskt-concurrency-testing/">Finding Order in the Mayhem: A Novel Concurrency Testing Tool that Improved the Kotlin Compiler</a></li> <li><a href="https://media.githubusercontent.com/media/anlun/publicFiles/master/papers/Moiseenko-al-Programming21-full.pdf">A Survey of Programming Language Memory Models</a></li> </ul> https://tedneward.github.io/Research/tools/lmstudio/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lmstudio/index.html <p><a href="https://lmstudio.ai/">Website</a> | <a href="https://github.com/lmstudio-ai/lms">Source</a> | <a href="https://lmstudio.ai/docs/developer">Docs</a></p> https://tedneward.github.io/Research/tools/gnu/stow/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gnu/stow/index.html <h2>Articles</h2> <ul> <li><a href="https://tamerlan.dev/how-i-manage-my-dotfiles-using-gnu-stow/">How I manage my dotfiles using GNU stow</a></li> </ul> https://tedneward.github.io/Research/tools/gradle/plugins/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gradle/plugins/index.html <blockquote> <p>"Official plugin" means that it's provided as a builtin plugin by Gradle.</p> </blockquote> <h3>Language</h3> <ul> <li><a href="https://docs.gradle.org/current/userguide/java_plugin.html">java</a> - Official plugin that adds Java compilation, testing and bundling capabilities.</li> <li><a href="https://docs.gradle.org/current/userguide/groovy_plugin.html">groovy</a> - Official plugin that adds support for building Groovy projects.</li> <li><a href="https://docs.gradle.org/current/userguide/scala_plugin.html">scala</a> - Official plugin that adds support for building Scala projects.</li> <li><a href="https://docs.gradle.org/current/userguide/antlr_plugin.html">antlr</a> - Official plugin that adds support for generating parsers using <a href="http://www.antlr.org/">Antlr</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">assembler</a> - Official plugin that adds native assembly language capabilities to a project.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">c</a> - Official plugin that adds C source compilation capabilities to a project.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">cpp</a> - Official plugin that adds C++ source compilation capabilities to a project.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">objective-c</a> - Official plugin that adds Objective-C source compilation capabilities to a project.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">objective-cpp</a> - Official plugin that adds Objective-C++ source compilation capabilities to a project.</li> <li><a href="https://github.com/evant/gradle-retrolambda">gradle-retrolambda</a> - Get Java lambda support in Java 6, 7 and Android.</li> <li><a href="https://github.com/clojurephant/clojurephant">clojurephant</a> - Clojure/ClojureScript support for Gradle</li> </ul> <h3>Code quality</h3> <ul> <li><a href="https://docs.gradle.org/current/userguide/findbugs_plugin.html">findbugs</a> - Official plugin that performs quality checks on Java source files using <a href="http://findbugs.sourceforge.net/">FindBugs</a>.</li> <li><a href="https://plugins.gradle.org/plugin/com.github.spotbugs">spotbugs</a> - Official plugin that performs quality checks on Java source files using <a href="https://spotbugs.github.io/">SpotBugs</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/checkstyle_plugin.html">checkstyle</a> - Official plugin that performs quality checks on Java source files using <a href="http://checkstyle.sourceforge.net/index.html">Checkstyle</a>.</li> <li><a href="https://github.com/diffplug/spotless/tree/master/plugin-gradle">spotless</a> - Checks and applies formatting rules using the Eclipse, google-java-format, ktlint, scalafmt, and also user-defined rules.</li> <li><a href="https://docs.gradle.org/current/userguide/pmd_plugin.html">pmd</a> - Official plugin that performs quality checks on your project's Java source files using <a href="http://pmd.sourceforge.net/">PMD</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/jdepend_plugin.html">jdepend</a> - Official plugin that performs quality checks on your project's source files using <a href="http://clarkware.com/software/JDepend.html">JDepend</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/codenarc_plugin.html">codenarc</a> - Official plugin that Performs quality checks on Groovy source files using <a href="http://codenarc.sourceforge.net/index.html">CodeNarc</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/jacoco_plugin.html">jacoco</a> - Official plugin that provides integration with the <a href="http://www.eclemma.org/jacoco/">JaCoCo</a> code coverage library for Java.</li> <li><a href="https://github.com/stevesaliman/gradle-cobertura-plugin">gradle-cobertura-plugin</a> - Use cobertura.</li> <li><a href="https://github.com/kt3k/coveralls-gradle-plugin">coveralls-gradle-plugin</a> - Send coverage data to <a href="https://coveralls.io/">coveralls.io</a>.</li> <li><a href="https://github.com/scoverage/gradle-scoverage">gradle-scoverage</a> - Enable the use of Scoverage in a Gradle Scala project.</li> <li><a href="https://github.com/tbroyer/gradle-errorprone-plugin">gradle-errorprone-plugin</a> - Use the <a href="https://github.com/google/error-prone">error-prone</a> compiler for Java.</li> <li><a href="https://github.com/ksoichiro/gradle-spelling-plugin">gradle-spelling-plugin</a> - Inspect spelling using custom blacklist.</li> <li><a href="https://github.com/simonharrer/gradle-modernizer-plugin">gradle-modernizer-plugin</a> - Detect uses of legacy Java APIs.</li> </ul> <h3>Code generation</h3> <ul> <li><a href="https://github.com/tcawley/gradle-protobuf-plugin">gradle-protobuf-plugin</a> - Compile <a href="https://developers.google.com/protocol-buffers/">Google Protocol Buffers</a> files.</li> <li><a href="https://github.com/ewerk/gradle-plugins/tree/master/querydsl-plugin">querydsl-plugin</a> - Generate <a href="http://www.querydsl.com/">Querydsl</a> classes.</li> </ul> <h3>Java application development</h3> <ul> <li><a href="https://github.com/tbroyer/gradle-apt-plugin">gradle-apt-plugin</a> - Make it easier/safer to use Java annotation processors.</li> <li><a href="https://github.com/darylteo/vertx-gradle-plugin">vertx-gradle-plugin</a> - Unofficial plugin for starting Vert.x projects.</li> </ul> <h3>Web application development</h3> <ul> <li><a href="http://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#build-tool-plugins-gradle-plugin">spring-boot</a> - Provide Spring Boot support.</li> <li><a href="https://github.com/akhikhl/gretty">gretty</a> - Run web apps on jetty and tomcat.</li> <li><a href="https://github.com/bmuschko/gradle-tomcat-plugin">gradle-tomcat-plugin</a> - Support deployment of your web application to an embedded Tomcat web container.</li> <li><a href="https://github.com/zeroturnaround/gradle-jrebel-plugin">gradle-jrebel-plugin</a> - Generate rebel.xml configuration file.</li> <li><a href="https://github.com/eriwen/gradle-js-plugin">gradle-js-plugin</a> - Manage JavaScript.</li> <li><a href="https://github.com/kellyrob99/gradle-jslint-plugin">gradle-jslint-plugin</a> - Run JSLint static analysis against JavaScipt code.</li> <li><a href="https://github.com/srs/gradle-node-plugin">gradle-node-plugin</a> - Run NodeJS scripts.</li> <li><a href="https://github.com/srs/gradle-gulp-plugin">gradle-gulp-plugin</a> - Run Gulp tasks.</li> <li><a href="https://github.com/srs/gradle-grunt-plugin">gradle-grunt-plugin</a> - Run Grunt tasks.</li> <li><a href="https://github.com/craigburke/bower-installer-gradle">bower-installer-plugin</a> - Manage client-side dependencies.</li> <li><a href="https://github.com/EvidentSolutions/apina">apina</a> - Creates client-side TypeScript from server-side APIs.</li> <li><a href="https://github.com/bertramdev/asset-pipeline">asset-pipeline</a> - Manage and process static assets in JVM applications.</li> <li><a href="https://github.com/ksoichiro/gradle-web-resource-plugin">gradle-web-resource-plugin</a> - Use CoffeeScript, LESS and Bower libraries without Node.js/npm.</li> <li><a href="https://github.com/robfletcher/gradle-compass">gradle-compass</a> - Compile and watche SASS files.</li> <li><a href="https://github.com/noamt/rest-gradle-plugin">rest-gradle-plugin</a> - Perform REST requests.</li> <li><a href="https://plugins.gradle.org/plugin/org.gretty">Gretty</a> - Advanced gradle plugin for running web-apps on jetty and tomcat. <a href="https://gretty-gradle-plugin.github.io/gretty-doc/">https://gretty-gradle-plugin.github.io/gretty-doc/</a> Add this plugin to your build using the plugins DSL: <pre><code>plugins { id("org.gretty") version "4.1.7" } </code></pre> </li> </ul> <h3>Android application development</h3> <ul> <li>Dependency management</li> <li><a href="https://github.com/JakeWharton/sdk-manager-plugin">sdk-manager-plugin</a> - Download and manage Android SDK.</li> <li><a href="https://github.com/nhachicha/android-native-dependencies">android-native-dependencies</a> - Gradle plugin for resolving and downloading Android native dependencies (.so)</li> <li>Alternative language</li> <li><a href="https://github.com/groovy/groovy-android-gradle-plugin">groovy-android-gradle-plugin</a> - Support the Groovy language for building Android apps.</li> <li>APK handling</li> <li><a href="https://github.com/KeepSafe/dexcount-gradle-plugin">dexcount-gradle-plugin</a> - Report the number of method references in APK.</li> <li><a href="https://github.com/ihrthk/android-gradle-mulchannel-plugin">android-gradle-mulchannel-plugin</a> - Generate multiple apks from different channel.</li> <li><a href="https://github.com/vanniktech/gradle-android-apk-size-plugin">gradle-android-apk-size-plugin</a> - Gradle plugin that generates CSV files with apk size per output and variant of an apk.</li> <li>Build variant handling</li> <li><a href="https://github.com/moallemi/gradle-advanced-build-version">gradle-advanced-build-version</a> - Generate the Android version code and version name automatically.</li> <li><a href="https://github.com/tmiyamon/gradle-config">gradle-config</a> - Handle variant specific settings with yaml format.</li> <li>Icons</li> <li><a href="https://github.com/gfx/gradle-android-ribbonizer-plugin">gradle-android-ribbonizer-plugin</a> - Add a ribbon to launcher icons of Android apps.</li> <li><a href="https://github.com/splatte/gradle-android-appiconoverlay">gradle-android-appiconoverlay</a> - Automatically overlay the app icon with the current git commit SHA1.</li> <li><a href="https://github.com/tmiyamon/gradle-mdicons">gradle-mdicons</a> - Manage material design icons.</li> <li>Releasing</li> <li><a href="https://github.com/deploygate/gradle-deploygate-plugin">gradle-deploygate-plugin</a> - Build and deploy apps to DeployGate.</li> <li><a href="https://github.com/testfairy/testfairy-gradle-plugin">testfairy-gradle-plugin</a> - Official plugin to upload signed builds to TestFairy.</li> <li><a href="https://github.com/Triple-T/gradle-play-publisher">gradle-play-publisher</a> - Manage your complete Play Store presence in your repository: Listing, Release Notes, APKs and App Bundles.</li> <li>Testing</li> <li><a href="https://github.com/Centril/gradle-plugin-robospock">gradle-plugin-robospock</a> - Configure robospock (gradle + spock + roboelectric) easily.</li> <li><a href="https://github.com/bjoernQ/unmock-plugin">unmock-plugin</a> - Allow you to use selected classes from a real Android-Jarfile for Android unit testing.</li> <li><a href="https://github.com/AutoScout24/gradle-monkey-plugin">gradle-monkey-plugin</a> - Run Android monkey tests.</li> <li>Miscellaneous</li> <li><a href="https://github.com/stephanenicolas/ormlite-android-gradle-plugin">ormlite-android-gradle-plugin</a> - Generate an ORMLite configuration file and boost DAOs creations.</li> <li><a href="https://github.com/ksoichiro/gradle-eclipse-aar-plugin">gradle-eclipse-aar-plugin</a> - Use Android AAR libraries on Eclipse.</li> <li><a href="https://github.com/ksoichiro/gradle-android-git">gradle-android-git</a> - Manage Git dependency for Android apps.</li> <li><a href="https://github.com/Karumi/Shot">Shot</a> - Shot is a Gradle plugin that simplifies the execution of screenshot tests using <a href="http://facebook.github.io/screenshot-tests-for-android/">Screenshot Tests For Android by Facebook</a>.</li> </ul> <h3>iOS and Mac application development</h3> <ul> <li><a href="https://github.com/openbakery/gradle-xcodePlugin">gradle-xcodePlugin</a> - Build iOS and Mac projects.</li> <li><a href="https://github.com/j2objc-contrib/j2objc-gradle">j2objc-gradle</a> - Enable Java source to be part of an iOS application's build.</li> </ul> <h3>Editor and IDE integration</h3> <ul> <li><a href="https://docs.gradle.org/current/userguide/eclipse_plugin.html">eclipse</a> - Official plugin that generates files that are used by <a href="http://eclipse.org/">Eclipse IDE</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/idea_plugin.html">idea</a> - Official plugin that generates files that are used by <a href="http://www.jetbrains.com/idea/index.html">Intellij IDEA IDE</a>.</li> <li><a href="https://docs.gradle.org/current/userguide/native_software.html">visual-studio</a> - Official plugin that adds integration with Visual Studio.</li> <li><a href="https://github.com/diffplug/goomph">goomph</a> - Downloads an Eclipse IDE with all required plugins and creates a workspace with specified settings and projects.</li> <li><a href="https://github.com/phildopus/gradle-sublimetext-plugin">gradle-sublimetext-plugin</a> - Generate Sublime Text 2 project file.</li> <li><a href="https://github.com/Scuilion/gradle-syntastic-plugin">gradle-syntastic-plugin</a> - Integrate Java project with Vim and Syntastic.</li> </ul> <h3>Templating</h3> <ul> <li><a href="https://github.com/aalmiray/markdown-gradle-plugin">markdown-gradle-plugin</a> - Convert Markdown to HTML.</li> <li><a href="https://github.com/67726e/gradle-twirl">gradle-twirl</a> - Provide <a href="https://github.com/playframework/twirl">Twirl</a> template compilation and integration.</li> </ul> <h3>Database</h3> <ul> <li><a href="http://flywaydb.org/documentation/gradle/">flyway-gradle-plugin</a> - <a href="http://flywaydb.org/">Flyway</a> database migration tasks.</li> <li><a href="https://github.com/liquibase/liquibase-gradle-plugin">liquibase-gradle-plugin</a> - Use <a href="http://liquibase.org/">Liquibase</a> to manage your database upgrades.</li> <li><a href="https://github.com/marceloemanoel/gradle-migrations-plugin">gradle-migrations-plugin</a> - Provide gradle build integration with <a href="https://code.google.com/p/mybatis/wiki/Migration">mybatis migrations</a>.</li> <li><a href="https://github.com/marklogic-community/ml-gradle">ml-gradle</a> - Automate everything involving <a href="https://developer.marklogic.com/">MarkLogic</a>.</li> </ul> <h3>Dependency management</h3> <ul> <li><a href="https://github.com/ben-manes/gradle-versions-plugin">gradle-versions-plugin</a> - Provide a task to determine which dependencies have updates.</li> <li><a href="https://github.com/Ullink/gradle-nuget-plugin">gradle-nuget-plugin</a> - Execute NuGet.exe from Gradle.</li> <li><a href="https://github.com/wfhartford/gradle-dependency-analyze">gradle-dependency-analyze</a> - Dependency analysis plugin for gradle.</li> <li><a href="https://github.com/nebula-plugins/gradle-dependency-lock-plugin">gradle-dependency-lock-plugin</a> - Allow people using dynamic dependency versions to lock them to specific versions.</li> <li><a href="https://github.com/layerhq/gradle-git-repo-plugin">gradle-git-repo-plugin</a> - Use a private git repo as a Maven repository.</li> <li><a href="https://github.com/jmfayard/buildSrcVersions">buildSrcVersions</a> - Painless dependencies management.</li> </ul> <h3>Debugging</h3> <ul> <li><a href="https://github.com/tkruse/gradle-groovysh-plugin">gradle-groovysh-plugin</a> - Start an interactive groovy shells.</li> </ul> <h3>Testing</h3> <ul> <li><a href="https://github.com/alphagov/gradle-gatling-plugin">gradle-gatling-plugin</a> - Run <a href="http://gatling.io/">Gatling</a> scenarios.</li> <li><a href="https://github.com/ksoichiro/gradle-console-reporter">gradle-console-reporter</a> - Report test result to console.</li> <li><a href="https://github.com/radarsh/gradle-test-logger-plugin">gradle-test-logger-plugin</a> - A Gradle plugin for printing beautiful logs on the console while running tests.</li> <li><a href="https://github.com/Softeq/itest-gradle-plugin">gradle-itest-plugin</a> - This plugin adds integration testing support to the project</li> </ul> <h3>Building</h3> <ul> <li><a href="https://github.com/passy/build-time-tracker-plugin">build-time-tracker-plugin</a> - Continuously track and report your build times.</li> <li><a href="https://github.com/nebula-plugins/gradle-metrics-plugin">gradle-metrics-plugin</a> - Collect Gradle build metrics and persist them to an external datastore.</li> <li><a href="https://nebula-plugins.github.io/">nebula-plugin</a> - A collection of Gradle plugins providing repeatable builds, immutable deployments and helping eliminate boilerplate code.</li> </ul> <h3>Packaging</h3> <ul> <li><a href="https://github.com/rholder/gradle-one-jar">gradle-one-jar</a> - Build self-contained executable jars that include all dependencies.</li> <li><a href="https://github.com/ksoichiro/gradle-build-info-plugin">gradle-build-info-plugin</a> - Include build information such as Git commit ID to your JAR.</li> <li><a href="https://github.com/ksoichiro/gradle-replacer">gradle-replacer</a> - Provide a minimalistic template engine feature.</li> </ul> <h3>Releasing</h3> <ul> <li><a href="https://github.com/int128/gradle-ssh-plugin">gradle-ssh-plugin</a> - Provide SSH facilities for continuous delivery.</li> <li><a href="https://docs.gradle.org/current/userguide/maven_plugin.html">maven</a> - Official plugin that adds support for publishing artifacts to Maven repositories.</li> <li><a href="https://plugins.gradle.org/plugin/com.gradle.plugin-publish">plugin-publish</a> - Publish plugins to the Gradle Plugin Portal.</li> <li><a href="https://github.com/bintray/gradle-bintray-plugin">gradle-bintray-plugin</a> - Publish artifacts to Bintray.</li> <li><a href="https://github.com/bmuschko/gradle-nexus-plugin">gradle-nexus-plugin</a> - Configure and upload artifacts to Sonatype Nexus.</li> <li><a href="https://github.com/researchgate/gradle-release">gradle-release</a> - Automate releasing process. Similar to the Maven release plugin.</li> <li><a href="https://github.com/diffplug/spotless-changelog">spotless-changelog</a> - Parses changelog to calculate next version, then updates changelog on publish.</li> </ul> <h3>Notification</h3> <ul> <li><a href="https://docs.gradle.org/current/userguide/announce_plugin.html">announce</a> - Official plugin that publishes messages to platforms such as Twitter or Growl.</li> <li><a href="https://github.com/Mindera/gradle-slack-plugin">gradle-slack-plugin</a> - Send messages to Slack after each build.</li> </ul> <h3>Cloud services</h3> <ul> <li><a href="https://github.com/classmethod-aws/gradle-aws-plugin">gradle-aws-plugin</a> - Manage AWS resouces.</li> <li><a href="https://github.com/skhatri/gradle-s3-plugin">gradle-s3-plugin</a> - Upload files to / download files from S3.</li> <li><a href="https://github.com/nebula-plugins/gradle-stash-plugin">gradle-stash-plugin</a> - Interact with the Stash SCM.</li> <li><a href="https://github.com/melix/gradle-cf-plugin">gradle-cf-plugin</a> - Interact with CloudFoundry.</li> </ul> <h3>SCM</h3> <ul> <li><a href="https://github.com/ajoberstar/gradle-git">gradle-git</a> - Set of plugin to interact with Git repositories, publish files to gh-page, etc.</li> <li><a href="https://github.com/martoe/gradle-svntools-plugin">gradle-svntools-plugin</a> - Provide various SVN-related tasks.</li> <li><a href="https://github.com/novabyte/gradle-snapshot-plugin">gradle-snapshot-plugin</a> - Generate build metadata from SCM tools.</li> </ul> <h3>CI</h3> <ul> <li><a href="https://github.com/jfrogdev/build-info">build-info</a> - Artifactory's open integration layer for the CI servers and build tools.</li> </ul> <h3>VM and container</h3> <ul> <li><a href="https://github.com/bmuschko/gradle-vagrant-plugin">gradle-vagrant-plugin</a> - Manage Vagrant boxes.</li> <li><a href="https://github.com/bmuschko/gradle-docker-plugin">bmuschko/gradle-docker-plugin</a> - Gradle plugin for managing Docker images and containers.</li> <li><a href="https://github.com/nebula-plugins/nebula-docker-plugin">nebula-docker-plugin</a> - Nebula gradle plugin for reducing boilerplate in creating docker images.</li> <li><a href="https://github.com/palantir/gradle-docker">palantir/gradle-docker</a> - Build and push Docker images.</li> <li><a href="https://github.com/Transmode/gradle-docker">Transmode/gradle-docker</a> - Build Docker images.</li> </ul> https://tedneward.github.io/Research/tools/grunt/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/grunt/index.html <p><a href="https://gruntjs.com/">Website</a></p> https://tedneward.github.io/Research/tools/hexfiend/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hexfiend/index.html <p><a href="https://hexfiend.com/">Website</a> | <a href="https://github.com/ridiculousfish/HexFiend/">Source</a></p> https://tedneward.github.io/Research/tools/honkit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/honkit/index.html <p><a href="https://honkit.netlify.app/">Website</a> | <a href="https://github.com/honkit/honkit">Source</a></p> <p>A fork of Gitbook. Can generate PDF, ePub, or Mobi files.</p> https://tedneward.github.io/Research/tools/hugo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hugo/index.html <p><a href="https://gohugo.io/">Website</a> | <a href="https://github.com/gohugoio/hugo">Source</a></p> https://tedneward.github.io/Research/tools/ilograph/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ilograph/index.html <p>Ilographs are advanced, interactive diagrams that allow users to see systems from multiple perspectives and levels of detail. Ilograph is a diagramming platform for system, software, network, cloud and other engineers.</p> <p><a href="https://www.ilograph.com/docs/editing/idl/">Website</a> | <a href="https://www.ilograph.com/docs/">Ilograph Diagramming Website</a></p> https://tedneward.github.io/Research/tools/infer/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/infer/index.html <p><a href="http://fbinfer.com/">Website</a> | <a href="https://github.com/facebook/infer">Source</a></p> <h2>Articles</h2> <ul> <li><a href="https://neelbhatt.com/2021/03/07/first-look-at-infersharp-a-c-version-of-facebooks-infer/">First look at InferSharp: A C# version of Facebook’s Infer</a> <a href="https://github.com/microsoft/infersharp">InferSharp</a></li> </ul> https://tedneward.github.io/Research/tools/jad/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jad/index.html <p>Not updated since Java5, not open-source, written in C++ (and it was <em>fast</em>). <a href="http://www.javadecompilers.com/jad">Last known available download site</a></p> https://tedneward.github.io/Research/tools/javacc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/javacc/index.html <p><a href="https://javacc.github.io/javacc/">Website</a> | <a href="https://github.com/javacc/javacc">Source</a></p> <p>Features:</p> <ul> <li> <p>JavaCC generates top-down (recursive descent) parsers as opposed to bottom-up parsers generated by YACC-like tools. This allows the use of more general grammars, although left-recursion is disallowed. Top-down parsers have a number of other advantages (besides more general grammars) such as being easier to debug, having the ability to parse to any non-terminal in the grammar, and also having the ability to pass values (attributes) both up and down the parse tree during parsing.</p> </li> <li> <p>By default, JavaCC generates an LL(1) parser. However, there may be portions of grammar that are not LL(1). JavaCC offers the capabilities of syntactic and semantic lookahead to resolve shift-shift ambiguities locally at these points. For example, the parser is LL(k) only at such points, but remains LL(1) everywhere else for better performance. Shift-reduce and reduce-reduce conflicts are not an issue for top-down parsers.</p> </li> <li> <p>JavaCC generates parsers that are 100% pure Java, so there is no runtime dependency on JavaCC and no special porting effort required to run on different machine platforms.</p> </li> <li> <p>JavaCC allows extended BNF specifications - such as (A)<em>, (A)+ etc - within the lexical and the grammar specifications. Extended BNF relieves the need for left-recursion to some extent. In fact, extended BNF is often easier to read as in A ::= y(x)</em> versus A ::= Ax|y.</p> </li> <li> <p>The lexical specifications (such as regular expressions, strings) and the grammar specifications (the BNF) are both written together in the same file. It makes grammars easier to read since it is possible to use regular expressions inline in the grammar specification, and also easier to maintain.</p> </li> <li> <p>The lexical analyzer of JavaCC can handle full Unicode input, and lexical specifications may also include any Unicode character. This facilitates descriptions of language elements such as Java identifiers that allow certain Unicode characters (that are not ASCII), but not others.</p> </li> <li> <p>JavaCC offers Lex-like lexical state and lexical action capabilities. Specific aspects in JavaCC that are superior to other tools are the first class status it offers concepts such as TOKEN, MORE, SKIP and state changes. This allows cleaner specifications as well as better error and warning messages from JavaCC.</p> </li> <li> <p>Tokens that are defined as special tokens in the lexical specification are ignored during parsing, but these tokens are available for processing by the tools. A useful application of this is in the processing of comments.</p> </li> <li> <p>Lexical specifications can define tokens not to be case-sensitive either at the global level for the entire lexical specification, or on an individual lexical specification basis.</p> </li> <li> <p>JavaCC comes with JJTree, an extremely powerful tree building pre-processor.</p> </li> <li> <p>JavaCC also includes JJDoc, a tool that converts grammar files to documentation files, optionally in HTML.</p> </li> <li> <p>JavaCC offers many options to customize its behavior and the behavior of the generated parsers. Examples of such options are the kinds of Unicode processing to perform on the input stream, the number of tokens of ambiguity checking to perform etc.</p> </li> <li> <p>JavaCC error reporting is among the best in parser generators. JavaCC generated parsers are able to clearly point out the location of parse errors with complete diagnostic information.</p> </li> <li> <p>Using options DEBUG_PARSER, DEBUG_LOOKAHEAD, and DEBUG_TOKEN_MANAGER, users can get in-depth analysis of the parsing and the token processing steps.</p> </li> <li> <p>The JavaCC release includes a wide range of examples including Java and HTML grammars. The examples, along with their documentation, are a great way to get acquainted with JavaCC.</p> </li> </ul> https://tedneward.github.io/Research/tools/jcstress/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jcstress/index.html <p><a href="https://openjdk.org/projects/code-tools/jcstress/">Website</a> | <a href="https://github.com/openjdk/jcstress">Source</a></p> <p>In order to understand jcstress tests and maybe write your own, it might be useful to work through the jcstress-samples. The samples come in three groups:</p> <ul> <li>APISample target to explain the jcstress API;</li> <li>JMMSample target to explain the basics of Java Memory Model;</li> <li>ConcurrencySample show the interesting concurrent behaviors of standard library.</li> </ul> https://tedneward.github.io/Research/tools/jintegra/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jintegra/index.html <p><a href="https://j-integra.intrinsyc.com/">Website</a> | Commercial</p> https://tedneward.github.io/Research/tools/jsdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jsdb/index.html <p>"JSDB® is a JavaScript shell designed with network-centric programming in mind. It can be used as a database interface, XML processor, and internet-oriented scripting language. JSDB can do nearly any batch processing task that Visual Basic can do, and it has drivers to let you read and send email, query a MySQL database, and run regular expression searches, without having to learn a specific SQL syntax or install a bunch of ActiveX components. JSDB has everything you need to write a web server or database middleware, with a graphical debugger, itself written in JSDB."</p> <p><a href="http://www.jsdb.org/">Website</a></p> https://tedneward.github.io/Research/tools/kiota/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/kiota/index.html <p><a href="https://learn.microsoft.com/en-us/openapi/kiota/overview">Website</a> | <a href="https://github.com/microsoft/kiota">Source</a></p> <p>The goal is to eliminate the need to take a dependency on a different API SDK for every API that you need to call. Kiota API clients provide a strongly typed experience with all the features you expect from a high quality API SDK, but without having to learn a new library for every HTTP API.</p> <p>This library builds on top of the Microsoft.OpenAPI.NET library to ensure comprehensive support for APIs that use OpenAPI descriptions. One of the goals of the project is to provide the best code generator support possible for OpenAPI and JSON Schema features. The conceptual documentation describes how kiota works and the high level concepts, this readme documents how to get started with Kiota.</p> https://tedneward.github.io/Research/tools/kubernetes/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/kubernetes/index.html <p><a href="https://kubernetes.io/">Website</a> | <a href="https://kubernetes.io/docs/home/">Docs</a> | <a href="https://github.com/kubernetes/kubernetes">Source</a></p> <p>Kubernetes is a portable, extensible, open-source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.</p> <ul> <li><a href="https://resources.linuxfoundation.org/LF+Projects/CNCF/TheNewStack_Book2_KubernetesDeploymentAndSecurityPatterns.pdf">Kubernetes Deployment &amp; Security Patterns</a> - Alex Williams (PDF)</li> <li><a href="https://www.digitalocean.com/community/curriculums/kubernetes-for-full-stack-developers">Kubernetes for Full-Stack Developers</a> - Jamon Camisso, Hanif Jetha, Katherine Juell (PDF, EPUB)</li> <li><a href="https://livro.descomplicandokubernetes.com.br/en/">Uncomplicating Kubernetes</a> - Jeferson Fernando</li> </ul> <p>Features:</p> <ul> <li>Automated rollouts and rollbacks: Kubernetes progressively rolls out changes to your application or its configuration, while monitoring application health to ensure it doesn't kill all your instances at the same time. If something goes wrong, Kubernetes will rollback the change for you. Take advantage of a growing ecosystem of deployment solutions.</li> <li>Service discovery and load balancing: No need to modify your application to use an unfamiliar service discovery mechanism. Kubernetes gives Pods their own IP addresses and a single DNS name for a set of Pods, and can load-balance across them.</li> <li>Storage orchestration: Automatically mount the storage system of your choice, whether from local storage, a public cloud provider such as GCP or AWS, or a network storage system such as NFS, iSCSI, Gluster, Ceph, Cinder, or Flocker.</li> <li>Secret and configuration management: Deploy and update secrets and application configuration without rebuilding your image and without exposing secrets in your stack configuration.</li> <li>Automatic bin packing: Automatically places containers based on their resource requirements and other constraints, while not sacrificing availability. Mix critical and best-effort workloads in order to drive up utilization and save even more resources.</li> <li>Batch execution: In addition to services, Kubernetes can manage your batch and CI workloads, replacing containers that fail, if desired.</li> <li>IPv4/IPv6 dual-stack: Allocation of IPv4 and IPv6 addresses to Pods and Services</li> <li>Horizontal scaling: Scale your application up and down with a simple command, with a UI, or automatically based on CPU usage.</li> <li>Self-healing: Restarts containers that fail, replaces and reschedules containers when nodes die, kills containers that don't respond to your user-defined health check, and doesn't advertise them to clients until they are ready to serve.</li> <li>Designed for extensibility: Add features to your Kubernetes cluster without changing upstream source code.</li> </ul> <hr> <p>Kubernetes scope levels:</p> <p><img src="https://media-exp1.licdn.com/dms/image/C4D22AQFGGnID0n3O7w/feedshare-shrink_800/0/1646199387172?e=1649289600&amp;v=beta&amp;t=tOpWwoqDw4YOWpwnKN-OqSM_M53vSTPWdyQnyo1jlik" alt=""></p> <p>Project (Strongest network/data/metadata isolation)<br> ... contains Cluster(s) (Strongest control plane isolation, stronger network/data isolation, strong metadata isolation)<br> ... contains Node(s) (stronger resource/network/data isolation)<br> ... contains Pod(s) (Some network isolation, some more resource isolation)<br> ... contains Container(s) (Some resource isolation, kernel security isolation)<br> ... with Namespaces (Some control plane isolation, service account isolation) being inside Cluster and encircling Pods</p> <hr> <h2>Articles</h2> <ul> <li><a href="https://medium.com/@mikhail_advani/kubernetes-in-cluster-traffic-encryption-using-cert-manager-b70c2101a12d">Kubernetes in-cluster traffic encryption using cert manager</a></li> </ul> <hr> <h2>Related Projects</h2> <p><a href="https://github.com/kyverno/kyverno">kyverno</a>: Kubernetes Native Policy Management</p> https://tedneward.github.io/Research/tools/leo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/leo/index.html <p><a href="http://www.leoeditor.com/">Website</a> | <a href="https://github.com/leo-editor">Source</a></p> https://tedneward.github.io/Research/tools/live-boot/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/live-boot/index.html <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://support.system76.com/articles/live-disk/">Live Disk Creation</a></li> </ul> https://tedneward.github.io/Research/tools/lobehub/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lobehub/index.html <p><a href="https://lobehub.com/">Website</a> | <a href="https://github.com/lobehub/lobehub">Source</a></p> <p>LobeHub is the ultimate space for work and life: to find, build, and collaborate with agent teammates that grow with you. We’re building the world’s largest human–agent co-evolving network.</p> <h2>👋🏻 Getting Started &amp; Join Our Community</h2> <p>We are a group of e/acc design-engineers, hoping to provide modern design components and tools for AIGC.<br> By adopting the Bootstrapping approach, we aim to provide developers and users with a more open, transparent, and user-friendly product ecosystem.</p> <h2>✨ Features</h2> <p>Today’s agents are one-off, task-driven tools. They lack context, live in isolation, and require manual hand-offs between different windows and models. While some maintain memory, it is often global, shallow, and impersonal. In this mode, users are forced to toggle between fragmented conversations, making it difficult to form structured productivity.</p> <p>LobeHub is a work-and-lifestyle space to find, build, and collaborate with agent teammates that grow with you. In LobeHub, we treat <strong>Agents as the unit of work</strong>, providing an infrastructure where humans and agents co-evolve.</p> <h3>Create: Agents as the Unit of Work</h3> <p>Building a personalized AI team starts with the <strong>Agent Builder</strong>. You can describe what you need once, and the agent setup starts right away, applying auto-configurations so you can use it instantly.</p> <ul> <li><strong>Unified Intelligence</strong>: Seamlessly access any model and any modality—all under your control.</li> <li><strong>10,000+ Skills</strong>: Connect your agents to the skills you use every day with a library of over 10,000 tools and MCP-compatible plugins.</li> </ul> <h3>Collaborate: Scale New Forms of Collaboration Networks</h3> <p>LobeHub introduces <strong>Agent Groups</strong>, allowing you to work with agents like real teammates. The system assembles the right agents for the task, enabling parallel collaboration and iterative improvement.</p> <ul> <li><strong>Pages</strong>: Write and refine content with multiple agents in one place with a shared context.</li> <li><strong>Schedule</strong>: Schedule runs and let agents do the work at the right time, even while you are away.</li> <li><strong>Project</strong>: Organize work by project to keep everything structured and easy to track.</li> <li><strong>Workspace</strong>: A shared space for teams to collaborate with agents, ensuring clear ownership and visibility across the organization.</li> </ul> <h3>Evolve: Co-evolution of Humans and Agents</h3> <p>The best AI is one that understands you deeply. LobeHub features <strong>Personal Memory</strong> that builds a clear understanding of your needs.</p> <ul> <li><strong>Continual Learning</strong>: Your agents learn from how you work, adapting their behavior to act at the right moment.</li> <li><strong>White-Box Memory</strong>: We believe in transparency. Your agents use structured, editable memory, giving you full control over what they remember.</li> </ul> <h3>MCP Plugin One-Click Installation</h3> <p><strong>Seamlessly Connect Your AI to the World</strong></p> <p>Unlock the full potential of your AI by enabling smooth, secure, and dynamic interactions with external tools, data sources, and services. LobeHub's MCP (Model Context Protocol) plugin system breaks down the barriers between your AI and the digital ecosystem, allowing for unprecedented connectivity and functionality.</p> <p>Transform your conversations into powerful workflows by connecting to databases, APIs, file systems, and more. Experience the freedom of AI that truly understands and interacts with your world.</p> <h3>MCP Marketplace</h3> <p><strong>Discover, Connect, Extend</strong></p> <p>Browse a growing library of MCP plugins to expand your AI's capabilities and streamline your workflows effortlessly. Visit <a href="https://lobehub.com/mcp">lobehub.com/mcp</a> to explore the MCP Marketplace, which offers a curated collection of integrations that enhance your AI's ability to work with various tools and services.</p> <p>From productivity tools to development environments, discover new ways to extend your AI's reach and effectiveness. Connect with the community and find the perfect plugins for your specific needs.</p> <h3>Desktop App</h3> <p><strong>Peak Performance, Zero Distractions</strong></p> <p>Get the full LobeHub experience without browser limitations—comprehensive, focused, and always ready to go. Our desktop application provides a dedicated environment for your AI interactions, ensuring optimal performance and minimal distractions.</p> <p>Experience faster response times, better resource management, and a more stable connection to your AI assistant. The desktop app is designed for users who demand the best performance from their AI tools.</p> <h3>Smart Internet Search</h3> <p><strong>Online Knowledge On Demand</strong></p> <p>With real-time internet access, your AI keeps up with the world—news, data, trends, and more. Stay informed and get the most current information available, enabling your AI to provide accurate and up-to-date responses.</p> <p>Access live information, verify facts, and explore current events without leaving your conversation. Your AI becomes a gateway to the world's knowledge, always current and comprehensive.</p> <h3><a href="https://lobehub.com/docs/usage/features/cot">Chain of Thought</a></h3> <p>Experience AI reasoning like never before. Watch as complex problems unfold step by step through our innovative Chain of Thought (CoT) visualization. This breakthrough feature provides unprecedented transparency into AI's decision-making process, allowing you to observe how conclusions are reached in real-time.</p> <p>By breaking down complex reasoning into clear, logical steps, you can better understand and validate the AI's problem-solving approach. Whether you're debugging, learning, or simply curious about AI reasoning, CoT visualization transforms abstract thinking into an engaging, interactive experience.</p> <h3><a href="https://lobehub.com/docs/usage/features/branching-conversations">Branching Conversations</a></h3> <p>Introducing a more natural and flexible way to chat with AI. With Branch Conversations, your discussions can flow in multiple directions, just like human conversations do. Create new conversation branches from any message, giving you the freedom to explore different paths while preserving the original context.</p> <p>Choose between two powerful modes:</p> <ul> <li><strong>Continuation Mode:</strong> Seamlessly extend your current discussion while maintaining valuable context</li> <li><strong>Standalone Mode:</strong> Start fresh with a new topic based on any previous message</li> </ul> <p>This groundbreaking feature transforms linear conversations into dynamic, tree-like structures, enabling deeper exploration of ideas and more productive interactions.</p> <h3><a href="https://lobehub.com/docs/usage/features/artifacts">Artifacts Support</a></h3> <p>Experience the power of Claude Artifacts, now integrated into LobeHub. This revolutionary feature expands the boundaries of AI-human interaction, enabling real-time creation and visualization of diverse content formats.</p> <p>Create and visualize with unprecedented flexibility:</p> <ul> <li>Generate and display dynamic SVG graphics</li> <li>Build and render interactive HTML pages in real-time</li> <li>Produce professional documents in multiple formats</li> </ul> <h3><a href="https://lobehub.com/blog/knowledge-base">File Upload /Knowledge Base</a></h3> <p>LobeHub supports file upload and knowledge base functionality. You can upload various types of files including documents, images, audio, and video, as well as create knowledge bases, making it convenient for users to manage and search for files. Additionally, you can utilize files and knowledge base features during conversations, enabling a richer dialogue experience.</p> <h3><a href="https://lobehub.com/docs/usage/features/multi-ai-providers">Multi-Model Service Provider Support</a></h3> <p>In the continuous development of LobeHub, we deeply understand the importance of diversity in model service providers for meeting the needs of the community when providing AI conversation services. Therefore, we have expanded our support to multiple model service providers, rather than being limited to a single one, in order to offer users a more diverse and rich selection of conversations.</p> <p>In this way, LobeHub can more flexibly adapt to the needs of different users, while also providing developers with a wider range of choices.</p> <h3><a href="https://lobehub.com/docs/usage/features/local-llm">Local Large Language Model (LLM) Support</a></h3> <p>To meet the specific needs of users, LobeHub also supports the use of local models based on <a href="https://ollama.ai">Ollama</a>, allowing users to flexibly use their own or third-party models.</p> <h3><a href="https://lobehub.com/docs/usage/features/vision">Model Visual Recognition</a></h3> <p>LobeHub now supports OpenAI's latest <a href="https://platform.openai.com/docs/guides/vision"><code>gpt-4-vision</code></a> model with visual recognition capabilities,<br> a multimodal intelligence that can perceive visuals. Users can easily upload or drag and drop images into the dialogue box,<br> and the agent will be able to recognize the content of the images and engage in intelligent conversation based on this,<br> creating smarter and more diversified chat scenarios.</p> <p>This feature opens up new interactive methods, allowing communication to transcend text and include a wealth of visual elements.<br> Whether it's sharing images in daily use or interpreting images within specific industries, the agent provides an outstanding conversational experience.</p> <h3><a href="https://lobehub.com/docs/usage/features/tts">TTS &amp; STT Voice Conversation</a></h3> <p>LobeHub supports Text-to-Speech (TTS) and Speech-to-Text (STT) technologies, enabling our application to convert text messages into clear voice outputs,<br> allowing users to interact with our conversational agent as if they were talking to a real person. Users can choose from a variety of voices to pair with the agent.</p> <p>Moreover, TTS offers an excellent solution for those who prefer auditory learning or desire to receive information while busy.<br> In LobeHub, we have meticulously selected a range of high-quality voice options (OpenAI Audio, Microsoft Edge Speech) to meet the needs of users from different regions and cultural backgrounds.<br> Users can choose the voice that suits their personal preferences or specific scenarios, resulting in a personalized communication experience.</p> <h3><a href="https://lobehub.com/docs/usage/features/text-to-image">Text to Image Generation</a></h3> <p>With support for the latest text-to-image generation technology, LobeHub now allows users to invoke image creation tools directly within conversations with the agent. By leveraging the capabilities of AI tools such as <a href="https://openai.com/dall-e-3"><code>DALL-E 3</code></a>, <a href="https://www.midjourney.com/"><code>MidJourney</code></a>, and <a href="https://pollinations.ai/"><code>Pollinations</code></a>, the agents are now equipped to transform your ideas into images.</p> <p>This enables a more private and immersive creative process, allowing for the seamless integration of visual storytelling into your personal dialogue with the agent.</p> <h3><a href="https://lobehub.com/docs/usage/features/plugin-system">Plugin System (Function Calling)</a></h3> <p>The plugin ecosystem of LobeHub is an important extension of its core functionality, greatly enhancing the practicality and flexibility of the LobeHub assistant.</p> <p> <video controls src="https://github.com/lobehub/lobehub/assets/28616219/f29475a3-f346-4196-a435-41a6373ab9e2" muted="false"></video></p> <p>By utilizing plugins, LobeHub assistants can obtain and process real-time information, such as searching for web information and providing users with instant and relevant news.</p> <p>In addition, these plugins are not limited to news aggregation, but can also extend to other practical functions, such as quickly searching documents, generating images, obtaining data from various platforms like Bilibili, Steam, and interacting with various third-party services.</p> <h3><a href="https://lobehub.com/docs/usage/features/agent-market">Agent Market (GPTs)</a></h3> <p>In LobeHub Agent Marketplace, creators can discover a vibrant and innovative community that brings together a multitude of well-designed agents,<br> which not only play an important role in work scenarios but also offer great convenience in learning processes.<br> Our marketplace is not just a showcase platform but also a collaborative space. Here, everyone can contribute their wisdom and share the agents they have developed.</p> <blockquote> <p>[!TIP]</p> <p>By <a href="https://github.com/lobehub/lobe-chat-agents">🤖/🏪 Submit Agents</a>, you can easily submit your agent creations to our platform.<br> Importantly, LobeHub has established a sophisticated automated internationalization (i18n) workflow,<br> capable of seamlessly translating your agent into multiple language versions.<br> This means that no matter what language your users speak, they can experience your agent without barriers.</p> <p>[!IMPORTANT]</p> <p>We welcome all users to join this growing ecosystem and participate in the iteration and optimization of agents.<br> Together, we can create more interesting, practical, and innovative agents, further enriching the diversity and practicality of the agent offerings.</p> </blockquote> <!-- AGENT LIST --> <table> <thead> <tr> <th> Recent Submits </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <a href="https://lobechat.com/discover/assistant/lateral-thinking-puzzle">Turtle Soup Host</a><br>^{By <strong><a href="https://github.com/CSY2022">CSY2022</a></strong> on <strong>2025-06-19</strong>} </td> <td> A turtle soup host needs to provide the scenario, the complete story (truth of the event), and the key point (the condition for guessing correctly).<br><code>turtle-soup</code> <code>reasoning</code> <code>interaction</code> <code>puzzle</code> <code>role-playing</code> </td> </tr> <tr> <td> <a href="https://lobechat.com/discover/assistant/academic-writing-assistant">Academic Writing Assistant</a><br>^{By <strong><a href="https://github.com/swarfte">swarfte</a></strong> on <strong>2025-06-17</strong>} </td> <td> Expert in academic research paper writing and formal documentation<br><code>academic-writing</code> <code>research</code> <code>formal-style</code> </td> </tr> <tr> <td> <a href="https://lobechat.com/discover/assistant/food-reviewer">Gourmet Reviewer🍟</a><br>^{By <strong><a href="https://github.com/renhai-lab">renhai-lab</a></strong> on <strong>2025-06-17</strong>} </td> <td> Food critique expert<br><code>gourmet</code> <code>review</code> <code>writing</code> </td> </tr> <tr> <td> <a href="https://lobechat.com/discover/assistant/java-development">Minecraft Senior Developer</a><br>^{By <strong><a href="https://github.com/iamyuuk">iamyuuk</a></strong> on <strong>2025-06-17</strong>} </td> <td> Expert in advanced Java development and Minecraft mod and server plugin development<br><code>development</code> <code>programming</code> <code>minecraft</code> <code>java</code> </td> </tr> </tbody> </table> <blockquote> <p>📊 Total agents: <a href="https://lobechat.com/discover/assistants"><kbd><strong>505</strong></kbd></a></p> </blockquote> <!-- AGENT LIST --> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <p><a href="https://lobehub.com/docs/usage/features/database"><img src="https://github.com/user-attachments/assets/f1697c8b-d1fb-4dac-ba05-153c6295d91d" alt=""></a></p> <h3><a href="https://lobehub.com/docs/usage/features/database">Support Local / Remote Database</a></h3> <p>LobeHub supports the use of both server-side and local databases. Depending on your needs, you can choose the appropriate deployment solution:</p> <ul> <li><strong>Local database</strong>: suitable for users who want more control over their data and privacy protection. LobeHub uses CRDT (Conflict-Free Replicated Data Type) technology to achieve multi-device synchronization. This is an experimental feature aimed at providing a seamless data synchronization experience.</li> <li><strong>Server-side database</strong>: suitable for users who want a more convenient user experience. LobeHub supports PostgreSQL as a server-side database. For detailed documentation on how to configure the server-side database, please visit <a href="https://lobehub.com/docs/self-hosting/advanced/server-database">Configure Server-side Database</a>.</li> </ul> <p>Regardless of which database you choose, LobeHub can provide you with an excellent user experience.</p> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <p><a href="https://lobehub.com/docs/usage/features/auth"><img src="https://github.com/user-attachments/assets/80bb232e-19d1-4f97-98d6-e291f3585e6d" alt=""></a></p> <h3><a href="https://lobehub.com/docs/usage/features/auth">Support Multi-User Management</a></h3> <p>LobeHub supports multi-user management and provides flexible user authentication solutions:</p> <ul> <li><strong>Better Auth</strong>: LobeHub integrates <code>Better Auth</code>, a modern and flexible authentication library that supports multiple authentication methods, including OAuth, email login, credential login, magic links, and more. With <code>Better Auth</code>, you can easily implement user registration, login, session management, social login, multi-factor authentication (MFA), and other functions to ensure the security and privacy of user data.</li> </ul> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <p><a href="https://lobehub.com/docs/usage/features/pwa"><img src="https://github.com/user-attachments/assets/9647f70f-b71b-43b6-9564-7cdd12d1c24d" alt=""></a></p> <h3><a href="https://lobehub.com/docs/usage/features/pwa">Progressive Web App (PWA)</a></h3> <p>We deeply understand the importance of providing a seamless experience for users in today's multi-device environment.<br> Therefore, we have adopted Progressive Web Application (<a href="https://support.google.com/chrome/answer/9658361">PWA</a>) technology,<br> a modern web technology that elevates web applications to an experience close to that of native apps.</p> <p>Through PWA, LobeHub can offer a highly optimized user experience on both desktop and mobile devices while maintaining high-performance characteristics.<br> Visually and in terms of feel, we have also meticulously designed the interface to ensure it is indistinguishable from native apps,<br> providing smooth animations, responsive layouts, and adapting to different device screen resolutions.</p> <blockquote> <p>[!NOTE]</p> <p>If you are unfamiliar with the installation process of PWA, you can add LobeHub as your desktop application (also applicable to mobile devices) by following these steps:</p> <ul> <li>Launch the Chrome or Edge browser on your computer.</li> <li>Visit the LobeHub webpage.</li> <li>In the upper right corner of the address bar, click on the <kbd>Install</kbd> icon.</li> <li>Follow the instructions on the screen to complete the PWA Installation.</li> </ul> </blockquote> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <p><a href="https://lobehub.com/docs/usage/features/mobile"><img src="https://github.com/user-attachments/assets/32cf43c4-96bd-4a4c-bfb6-59acde6fe380" alt=""></a></p> <h3><a href="https://lobehub.com/docs/usage/features/mobile">Mobile Device Adaptation</a></h3> <p>We have carried out a series of optimization designs for mobile devices to enhance the user's mobile experience. Currently, we are iterating on the mobile user experience to achieve smoother and more intuitive interactions. If you have any suggestions or ideas, we welcome you to provide feedback through GitHub Issues or Pull Requests.</p> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <p><a href="https://lobehub.com/docs/usage/features/theme"><img src="https://github.com/user-attachments/assets/b47c39f1-806f-492b-8fcb-b0fa973937c1" alt=""></a></p> <h3><a href="https://lobehub.com/docs/usage/features/theme">Custom Themes</a></h3> <p>As a design-engineering-oriented application, LobeHub places great emphasis on users' personalized experiences,<br> hence introducing flexible and diverse theme modes, including a light mode for daytime and a dark mode for nighttime.<br> Beyond switching theme modes, a range of color customization options allow users to adjust the application's theme colors according to their preferences.<br> Whether it's a desire for a sober dark blue, a lively peach pink, or a professional gray-white, users can find their style of color choices in LobeHub.</p> <blockquote> <p>[!TIP]</p> <p>The default configuration can intelligently recognize the user's system color mode and automatically switch themes to ensure a consistent visual experience with the operating system.<br> For users who like to manually control details, LobeHub also offers intuitive setting options and a choice between chat bubble mode and document mode for conversation scenarios.</p> </blockquote> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <h3><code>*</code> What's more</h3> <p>Beside these features, LobeHub also have much better basic technique underground:</p> <ul> <li>[x] 💨 <strong>Quick Deployment</strong>: Using the Vercel platform or docker image, you can deploy with just one click and complete the process within 1 minute without any complex configuration.</li> <li>[x] 🌐 <strong>Custom Domain</strong>: If users have their own domain, they can bind it to the platform for quick access to the dialogue agent from anywhere.</li> <li>[x] 🔒 <strong>Privacy Protection</strong>: All data is stored locally in the user's browser, ensuring user privacy.</li> <li>[x] 💎 <strong>Exquisite UI Design</strong>: With a carefully designed interface, it offers an elegant appearance and smooth interaction. It supports light and dark themes and is mobile-friendly. PWA support provides a more native-like experience.</li> <li>[x] 🗣️ <strong>Smooth Conversation Experience</strong>: Fluid responses ensure a smooth conversation experience. It fully supports Markdown rendering, including code highlighting, LaTex formulas, Mermaid flowcharts, and more.</li> </ul> <blockquote> <p>✨ more features will be added when LobeHub evolve.</p> </blockquote> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <h2>🛳 Self Hosting</h2> <p>LobeHub provides Self-Hosted Version with Vercel, Alibaba Cloud, and <a href="https://hub.docker.com/r/lobehub/lobehub">Docker Image</a>. This allows you to deploy your own chatbot within a few minutes without any prior knowledge.</p> <blockquote> <p>[!TIP]</p> <p>Learn more about <a href="https://lobehub.com/docs/self-hosting/start">📘 Build your own LobeHub</a> by checking it out.</p> </blockquote> <h3><code>A</code> Deploying with Vercel, Zeabur , Sealos or Alibaba Cloud</h3> <p>"If you want to deploy this service yourself on Vercel, Zeabur or Alibaba Cloud, you can follow these steps:</p> <ul> <li>Prepare your <a href="https://platform.openai.com/account/api-keys">OpenAI API Key</a>.</li> <li>Click the button below to start deployment: Log in directly with your GitHub account, and remember to fill in the <code>OPENAI_API_KEY</code>(required) on the environment variable section.</li> <li>After deployment, you can start using it.</li> <li>Bind a custom domain (optional): The DNS of the domain assigned by Vercel is polluted in some areas; binding a custom domain can connect directly.</li> </ul> <div align="center"> <table> <thead> <tr> <th align="center"> Deploy with Vercel </th> <th align="center"> Deploy with Zeabur </th> <th align="center"> Deploy with Sealos </th> <th align="center"> Deploy with RepoCloud </th> <th align="center"> Deploy with Alibaba Cloud </th> </tr> </thead> <tbody> <tr> <td align="center"> <a href="https://vercel.com/new/clone?repository-url=https%3A%2F%2Fgithub.com%2Flobehub%2Flobehub&amp;env=OPENAI_API_KEY&amp;envDescription=Find%20your%20OpenAI%20API%20Key%20by%20click%20the%20right%20Learn%20More%20button.&amp;envLink=https%3A%2F%2Fplatform.openai.com%2Faccount%2Fapi-keys&amp;project-name=lobehub&amp;repository-name=lobehub"><img src="https://vercel.com/button" alt=""></a> </td> <td align="center"> <a href="https://zeabur.com/templates/VZGGTI"><img src="https://zeabur.com/button.svg" alt=""></a> </td> <td align="center"> <a href="https://template.usw.sealos.io/deploy?templateName=lobehub-db"><img src="https://raw.githubusercontent.com/labring-actions/templates/main/Deploy-on-Sealos.svg" alt=""></a> </td> <td align="center"> <a href="https://repocloud.io/details/?app_id=248"><img src="https://d16t0pc4846x52.cloudfront.net/deploylobe.svg" alt=""></a> </td> <td align="center"> <a href="https://computenest.console.aliyun.com/service/instance/create/default?type=user&amp;ServiceName=LobeHub%E7%A4%BE%E5%8C%BA%E7%89%88"><img src="https://service-info-public.oss-cn-hangzhou.aliyuncs.com/computenest-en.svg" alt=""></a> </td> </tr> </tbody> </table> </div> <h4>After Fork</h4> <p>After fork, only retain the upstream sync action and disable other actions in your repository on GitHub.</p> <h4>Keep Updated</h4> <p>If you have deployed your own project following the one-click deployment steps in the README, you might encounter constant prompts indicating "updates available." This is because Vercel defaults to creating a new project instead of forking this one, resulting in an inability to detect updates accurately.</p> <blockquote> <p>[!TIP]</p> <p>We suggest you redeploy using the following steps, <a href="https://lobehub.com/docs/self-hosting/advanced/upstream-sync">📘 Auto Sync With Latest</a></p> </blockquote> <p><br></p> <h3><code>B</code> Deploying with Docker</h3> <p><a href="https://hub.docker.com/r/lobehub/lobehub"><img src="https://img.shields.io/docker/v/lobehub/lobehub?color=369eff&amp;label=docker&amp;labelColor=black&amp;logo=docker&amp;logoColor=white&amp;style=flat-square&amp;sort=semver" alt=""></a><br> <a href="https://hub.docker.com/r/lobehub/lobehub"><img src="https://img.shields.io/docker/image-size/lobehub/lobehub?color=369eff&amp;labelColor=black&amp;style=flat-square&amp;sort=semver" alt=""></a><br> <a href="https://hub.docker.com/r/lobehub/lobehub"><img src="https://img.shields.io/docker/pulls/lobehub/lobehub?color=45cc11&amp;labelColor=black&amp;style=flat-square&amp;sort=semver" alt=""></a></p> <p>We provide a Docker image for deploying the LobeHub service on your own private device. Use the following command to start the LobeHub service:</p> <ol> <li>create a folder to for storage files</li> </ol> <pre><code class="language-fish">$ mkdir lobehub-db &amp;&amp; cd lobehub-db </code></pre> <ol> <li>init the LobeHub infrastructure</li> </ol> <pre><code class="language-fish">bash &lt;(curl -fsSL https://lobe.li/setup.sh) </code></pre> <ol> <li>Start the LobeHub service</li> </ol> <pre><code class="language-fish">docker compose up -d </code></pre> <blockquote> <p>[!NOTE]</p> <p>For detailed instructions on deploying with Docker, please refer to the <a href="https://lobehub.com/docs/self-hosting/server-database/docker-compose">📘 Docker Deployment Guide</a></p> </blockquote> <p><br></p> <h3>Environment Variable</h3> <p>This project provides some additional configuration items set with environment variables:</p> <table> <thead> <tr> <th> Environment Variable </th> <th> Required </th> <th> Description </th> <th> Example </th> </tr> </thead> <tbody> <tr> <td> <code>OPENAI_API_KEY</code> </td> <td> Yes </td> <td> This is the API key you apply on the OpenAI account page </td> <td> <code>sk-xxxxxx...xxxxxx</code> </td> </tr> <tr> <td> <code>OPENAI_PROXY_URL</code> </td> <td> No </td> <td> If you manually configure the OpenAI interface proxy, you can use this configuration item to override the default OpenAI API request base URL </td> <td> <code>https://api.chatanywhere.cn</code> or <code>https://aihubmix.com/v1</code> <br>The default value is<br><code>https://api.openai.com/v1</code> </td> </tr> <tr> <td> <code>OPENAI_MODEL_LIST</code> </td> <td> No </td> <td> Used to control the model list. Use <code>+</code> to add a model, <code>-</code> to hide a model, and <code>model_name=display_name</code> to customize the display name of a model, separated by commas. </td> <td> <code>qwen-7b-chat,+glm-6b,-gpt-3.5-turbo</code> </td> </tr> </tbody> </table> <blockquote> <p>[!NOTE]</p> <p>The complete list of environment variables can be found in the <a href="https://lobehub.com/docs/self-hosting/environment-variables">📘 Environment Variables</a></p> </blockquote> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <h2>📦 Ecosystem</h2> <table> <thead> <tr> <th> NPM </th> <th> Repository </th> <th> Description </th> <th> Version </th> </tr> </thead> <tbody> <tr> <td> <a href="https://www.npmjs.com/package/@lobehub/ui">@lobehub/ui</a> </td> <td> <a href="https://github.com/lobehub/lobe-ui">lobehub/lobe-ui</a> </td> <td> Open-source UI component library dedicated to building AIGC web applications. </td> <td> <a href="https://www.npmjs.com/package/@lobehub/ui"><img src="https://img.shields.io/npm/v/@lobehub/ui?color=369eff&amp;labelColor=black&amp;logo=npm&amp;logoColor=white&amp;style=flat-square" alt=""></a> </td> </tr> <tr> <td> <a href="https://www.npmjs.com/package/@lobehub/icons">@lobehub/icons</a> </td> <td> <a href="https://github.com/lobehub/lobe-icons">lobehub/lobe-icons</a> </td> <td> Popular AI / LLM Model Brand SVG Logo and Icon Collection. </td> <td> <a href="https://www.npmjs.com/package/@lobehub/icons"><img src="https://img.shields.io/npm/v/@lobehub/icons?color=369eff&amp;labelColor=black&amp;logo=npm&amp;logoColor=white&amp;style=flat-square" alt=""></a> </td> </tr> <tr> <td> <a href="https://www.npmjs.com/package/@lobehub/tts">@lobehub/tts</a> </td> <td> <a href="https://github.com/lobehub/lobe-tts">lobehub/lobe-tts</a> </td> <td> High-quality &amp; reliable TTS/STT React Hooks library </td> <td> <a href="https://www.npmjs.com/package/@lobehub/tts"><img src="https://img.shields.io/npm/v/@lobehub/tts?color=369eff&amp;labelColor=black&amp;logo=npm&amp;logoColor=white&amp;style=flat-square" alt=""></a> </td> </tr> <tr> <td> <a href="https://www.npmjs.com/package/@lobehub/lint">@lobehub/lint</a> </td> <td> <a href="https://github.com/lobehub/lobe-lint">lobehub/lobe-lint</a> </td> <td> Configurations for ESlint, Stylelint, Commitlint, Prettier, Remark, and Semantic Release for LobeHub. </td> <td> <a href="https://www.npmjs.com/package/@lobehub/lint"><img src="https://img.shields.io/npm/v/@lobehub/lint?color=369eff&amp;labelColor=black&amp;logo=npm&amp;logoColor=white&amp;style=flat-square" alt=""></a> </td> </tr> </tbody> </table> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <h2>🧩 Plugins</h2> <p>Plugins provide a means to extend the <a href="https://lobehub.com/blog/openai-function-call">Function Calling</a> capabilities of LobeHub. They can be used to introduce new function calls and even new ways to render message results. If you are interested in plugin development, please refer to our <a href="https://lobehub.com/docs/usage/plugins/development">📘 Plugin Development Guide</a> in the Wiki.</p> <ul> <li><a href="https://github.com/lobehub/lobe-chat-plugins">lobe-chat-plugins</a>: This is the plugin index for LobeHub. It accesses index.json from this repository to display a list of available plugins for LobeHub to the user.</li> <li><a href="https://github.com/lobehub/chat-plugin-template">chat-plugin-template</a>: This is the plugin template for LobeHub plugin development.</li> <li><a href="https://github.com/lobehub/chat-plugin-sdk">@lobehub/chat-plugin-sdk</a>: The LobeHub Plugin SDK assists you in creating exceptional chat plugins for LobeHub.</li> <li><a href="https://github.com/lobehub/chat-plugins-gateway">@lobehub/chat-plugins-gateway</a>: The LobeHub Plugins Gateway is a backend service that provides a gateway for LobeHub plugins. We deploy this service using Vercel. The primary API POST /api/v1/runner is deployed as an Edge Function.</li> </ul> <blockquote> <p>[!NOTE]</p> <p>The plugin system is currently undergoing major development. You can learn more in the following issues:</p> <ul> <li>[x] <a href="https://github.com/lobehub/lobehub/issues/73"><strong>Plugin Phase 1</strong></a>: Implement separation of the plugin from the main body, split the plugin into an independent repository for maintenance, and realize dynamic loading of the plugin.</li> <li>[x] <a href="https://github.com/lobehub/lobehub/issues/97"><strong>Plugin Phase 2</strong></a>: The security and stability of the plugin's use, more accurately presenting abnormal states, the maintainability of the plugin architecture, and developer-friendly.</li> <li>[x] <a href="https://github.com/lobehub/lobehub/issues/149"><strong>Plugin Phase 3</strong></a>: Higher-level and more comprehensive customization capabilities, support for plugin authentication, and examples.</li> </ul> </blockquote> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <h2>🔗 More Products</h2> <ul> <li><strong><a href="https://github.com/lobehub/sd-webui-lobe-theme">🅰️ Lobe SD Theme</a>:</strong> Modern theme for Stable Diffusion WebUI, exquisite interface design, highly customizable UI, and efficiency-boosting features.</li> <li><strong><a href="https://github.com/lobehub/lobe-midjourney-webui">⛵️ Lobe Midjourney WebUI</a>:</strong> WebUI for Midjourney, leverages AI to quickly generate a wide array of rich and diverse images from text prompts, sparking creativity and enhancing conversations.</li> <li><strong><a href="https://github.com/lobehub/lobe-commit/tree/master/packages/lobe-i18n">🌏 Lobe i18n</a> :</strong> Lobe i18n is an automation tool for the i18n (internationalization) translation process, powered by ChatGPT. It supports features such as automatic splitting of large files, incremental updates, and customization options for the OpenAI model, API proxy, and temperature.</li> <li><strong><a href="https://github.com/lobehub/lobe-commit/tree/master/packages/lobe-commit">💌 Lobe Commit</a>:</strong> Lobe Commit is a CLI tool that leverages Langchain/ChatGPT to generate Gitmoji-based commit messages.</li> </ul> <!-- LINK GROUP --> https://tedneward.github.io/Research/tools/gnu/bash/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gnu/bash/index.html <p><a href="https://www.gnu.org/software/bash/">GNU bash Website</a> | <a href="https://savannah.gnu.org/projects/bash/">Source</a> | <a href="https://www.gnu.org/software/bash/manual/bash.pdf">GNU Bash manual</a></p> <h2>Reading, Places</h2> <ul> <li><a href="http://tldp.org/LDP/abs/html/">Advanced Bash-Scripting Guide</a></li> <li><a href="http://mywiki.wooledge.org/BashGuide">Bash Guide</a> - Maarten Billemont <a href="http://s.ntnu.no/bashguide.pdf">(PDF)</a></li> <li><a href="http://www.tldp.org/LDP/Bash-Beginners-Guide/html/">Bash Guide for Beginners (2008)</a></li> <li><a href="http://wiki.bash-hackers.org/">Bash-hackers wiki</a></li> <li><a href="http://goalkicker.com/BashBook/">Bash Notes for Professionals</a></li> <li><a href="http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO.html">BASH Programming (2000)</a></li> <li><a href="http://www.gnu.org/software/bash/manual/bashref.html">Bash Reference Manual</a></li> <li><a href="https://web.archive.org/web/20180328183806/http://gdrcorelec.ups-tlse.fr/files/bash.pdf">Bash tutorial</a></li> <li><a href="https://magpi.raspberrypi.org/books/command-line-second-edition/pdf/download">Conquer the Command Line</a></li> <li><a href="https://www.softcover.io/read/fc6c09de/unix_commands">Conquering the Command Line</a></li> <li><a href="https://devhints.io/bash">Devhints</a></li> <li><a href="http://www.hypexr.org/bash_tutorial.php">Getting Started with BASH</a></li> <li><a href="https://google.github.io/styleguide/shell.xml">Google Shell Style Guide</a></li> <li><a href="https://github.com/bobbyiliev/introduction-to-bash-scripting">Introduction to Bash Scripting</a></li> <li><a href="https://launchschool.com/books/command_line">Introduction to the Command Line</a></li> <li><a href="https://learnxinyminutes.com/docs/bash/">Learn bash in y minutes</a></li> <li><a href="http://www.freeos.com/guides/lsst/">Linux Shell Scripting Tutorial - A Beginner's Handbook (2002)</a></li> <li><a href="https://bash.cyberciti.biz/guide/Main_Page">Linux Shell Scripting Tutorial (LSST) v2.0</a></li> <li><a href="https://dev.to/dhupee/make-your-linux-terminal-enjoyable-to-use-3j47">"Make Your Linux Terminal Enjoyable to Use"</a></li> <li><a href="http://wiki.bash-hackers.org/syntax/shellvars">Shell vars</a></li> <li><a href="https://www.shellcheck.net/">ShellCheck</a></li> <li><a href="https://devmanual.gentoo.org/tools-reference/bash/index.html">shell - Standard Shell</a></li> <li><a href="http://slackbook.org">Slackbook (2005)</a></li> <li><a href="http://guide.bash.academy">The Bash Academy</a></li> <li><a href="http://linuxcommand.org/tlcl.php">The Linux Command Line</a></li> <li><a href="http://linuxcommand.org/lc3_writing_shell_scripts.php">Writing Shell Scripts</a></li> </ul> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://www.howtogeek.com/bash-variables-i-use-in-almost-every-script/">7 Bash variables I use in almost every script</a>:</li> </ul> <h2>Supporting tools/projects</h2> <ul> <li><a href="https://direnv.net/">direnv</a> (<a href="https://github.com/direnv/direnv">Source</a>): unclutter your .profile</li> <li><a href="https://www.gnu.org/software/parallel/parallel.html">parallel</a>: build and execute shell command lines from standard input in parallel</li> </ul> <h2>"Dotfiles"</h2> <ul> <li>Seems like <code>.bash_profile</code> gets loaded first if present, and if found, doesn't look for <code>.bashrc</code> any further; <code>.bash_profile</code> thus would need to directly source <code>.bashrc</code></li> </ul> <p><code>.bashrc</code>:</p> <pre><code># see /usr/share/doc/bash/examples/startup-files (in the package bash-doc) # for examples # If not running interactively, don't do anything case $- in *i*) ;; *) return;; esac # don't put duplicate lines or lines starting with space in the history. # See bash(1) for more options HISTCONTROL=ignoreboth # append to the history file, don't overwrite it shopt -s histappend # for setting history length see HISTSIZE and HISTFILESIZE in bash(1) HISTSIZE=1000 HISTFILESIZE=2000 # check the window size after each command and, if necessary, # update the values of LINES and COLUMNS. shopt -s checkwinsize # If set, the pattern "**" used in a pathname expansion context will # match all files and zero or more directories and subdirectories. #shopt -s globstar # make less more friendly for non-text input files, see lesspipe(1) [ -x /usr/bin/lesspipe ] &amp;&amp; eval "$(SHELL=/bin/sh lesspipe)" # set variable identifying the chroot you work in (used in the prompt below) if [ -z "${debian_chroot:-}" ] &amp;&amp; [ -r /etc/debian_chroot ]; then debian_chroot=$(cat /etc/debian_chroot) fi # set a fancy prompt (non-color, unless we know we "want" color) case "$TERM" in xterm-color|*-256color) color_prompt=yes;; esac # uncomment for a colored prompt, if the terminal has the capability; turned # off by default to not distract the user: the focus in a terminal window # should be on the output of commands, not on the prompt #force_color_prompt=yes if [ -n "$force_color_prompt" ]; then if [ -x /usr/bin/tput ] &amp;&amp; tput setaf 1 &gt;&amp;/dev/null; then # We have color support; assume it's compliant with Ecma-48 # (ISO/IEC-6429). (Lack of such support is extremely rare, and such # a case would tend to support setf rather than setaf.) color_prompt=yes else color_prompt= fi fi if [ "$color_prompt" = yes ]; then PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]\$ ' else PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ ' fi unset color_prompt force_color_prompt # If this is an xterm set the title to user@host:dir case "$TERM" in xterm*|rxvt*) PS1="\[\e]0;${debian_chroot:+($debian_chroot)}\u@\h: \w\a\]$PS1" ;; *) ;; esac # enable color support of ls and also add handy aliases if [ -x /usr/bin/dircolors ]; then test -r ~/.dircolors &amp;&amp; eval "$(dircolors -b ~/.dircolors)" || eval "$(dircolors -b)" alias ls='ls --color=auto' #alias dir='dir --color=auto' #alias vdir='vdir --color=auto' alias grep='grep --color=auto' alias fgrep='fgrep --color=auto' alias egrep='egrep --color=auto' fi # colored GCC warnings and errors #export GCC_COLORS='error=01;31:warning=01;35:note=01;36:caret=01;32:locus=01:quote=01' # some more ls aliases alias ll='ls -alF' alias la='ls -A' alias l='ls -CF' # Add an "alert" alias for long running commands. Use like so: # sleep 10; alert alias alert='notify-send --urgency=low -i "$([ $? = 0 ] &amp;&amp; echo terminal || echo error)" "$(history|tail -n1|sed -e '\''s/^\s*[0-9]\+\s*//;s/[;&amp;|]\s*alert$//'\'')"' # Alias definitions. # You may want to put all your additions into a separate file like # ~/.bash_aliases, instead of adding them here directly. # See /usr/share/doc/bash-doc/examples in the bash-doc package. if [ -f ~/.bash_aliases ]; then . ~/.bash_aliases fi # enable programmable completion features (you don't need to enable # this, if it's already enabled in /etc/bash.bashrc and /etc/profile # sources /etc/bash.bashrc). if ! shopt -oq posix; then if [ -f /usr/share/bash-completion/bash_completion ]; then . /usr/share/bash-completion/bash_completion elif [ -f /etc/bash_completion ]; then . /etc/bash_completion fi fi LS_COLORS=$LS_COLORS:'di=0;35:ln=36' ; export LS_COLORS </code></pre> <p><code>.dircolors</code>: I think we can generate this, but here's the color scheme I've found best (personally) for WSL and native-Linux bash.</p> <pre><code># Configuration file for dircolors, a utility to help you set the # LS_COLORS environment variable used by GNU ls with the --color option. # Copyright (C) 1996-2023 Free Software Foundation, Inc. # Copying and distribution of this file, with or without modification, # are permitted provided the copyright notice and this notice are preserved. # # The keywords COLOR, OPTIONS, and EIGHTBIT (honored by the # slackware version of dircolors) are recognized but ignored. # Global config options can be specified before TERM or COLORTERM entries # =================================================================== # Terminal filters # =================================================================== # Below are TERM or COLORTERM entries, which can be glob patterns, which # restrict following config to systems with matching environment variables. COLORTERM ?* TERM Eterm TERM ansi TERM *color* TERM con[0-9]*x[0-9]* TERM cons25 TERM console TERM cygwin TERM *direct* TERM dtterm TERM gnome TERM hurd TERM jfbterm TERM konsole TERM kterm TERM linux TERM linux-c TERM mlterm TERM putty TERM rxvt* TERM screen* TERM st TERM terminator TERM tmux* TERM vt100 TERM xterm* # =================================================================== # Basic file attributes # =================================================================== # Below are the color init strings for the basic file types. # One can use codes for 256 or more colors supported by modern terminals. # The default color codes use the capabilities of an 8 color terminal # with some additional attributes as per the following codes: # Attribute codes: # 00=none 01=bold 04=underscore 05=blink 07=reverse 08=concealed # Text color codes: # 30=black 31=red 32=green 33=yellow 34=blue 35=magenta 36=cyan 37=white # Background color codes: # 40=black 41=red 42=green 43=yellow 44=blue 45=magenta 46=cyan 47=white #NORMAL 00 # no color code at all #FILE 00 # regular file: use no color at all RESET 0 # reset to "normal" color DIR 01;34 # directory LINK 01;36 # symbolic link. (If you set this to 'target' instead of a # numerical value, the color is as for the file pointed to.) MULTIHARDLINK 00 # regular file with more than one link FIFO 40;33 # pipe SOCK 01;35 # socket DOOR 01;35 # door BLK 40;33;01 # block device driver CHR 40;33;01 # character device driver ORPHAN 40;31;01 # symlink to nonexistent file, or non-stat'able file ... MISSING 00 # ... and the files they point to SETUID 37;41 # file that is setuid (u+s) SETGID 30;43 # file that is setgid (g+s) CAPABILITY 00 # file with capability (very expensive to lookup) STICKY_OTHER_WRITABLE 30;42 # dir that is sticky and other-writable (+t,o+w) OTHER_WRITABLE 34;107 # dir that is other-writable (o+w) and not sticky STICKY 37;44 # dir with the sticky bit set (+t) and not other-writable # This is for files with execute permission: EXEC 01;32 # =================================================================== # File extension attributes # =================================================================== # List any file extensions like '.gz' or '.tar' that you would like ls # to color below. Put the suffix, a space, and the color init string. # (and any comments you want to add after a '#'). # Suffixes are matched case insensitively, but if you define different # init strings for separate cases, those will be honored. # # If you use DOS-style suffixes, you may want to uncomment the following: #.cmd 01;32 # executables (bright green) #.exe 01;32 #.com 01;32 #.btm 01;32 #.bat 01;32 # Or if you want to color scripts even if they do not have the # executable bit actually set. #.sh 01;32 #.csh 01;32 # archives or compressed (bright red) .tar 01;31 .tgz 01;31 .arc 01;31 .arj 01;31 .taz 01;31 .lha 01;31 .lz4 01;31 .lzh 01;31 .lzma 01;31 .tlz 01;31 .txz 01;31 .tzo 01;31 .t7z 01;31 .zip 01;31 .z 01;31 .dz 01;31 .gz 01;31 .lrz 01;31 .lz 01;31 .lzo 01;31 .xz 01;31 .zst 01;31 .tzst 01;31 .bz2 01;31 .bz 01;31 .tbz 01;31 .tbz2 01;31 .tz 01;31 .deb 01;31 .rpm 01;31 .jar 01;31 .war 01;31 .ear 01;31 .sar 01;31 .rar 01;31 .alz 01;31 .ace 01;31 .zoo 01;31 .cpio 01;31 .7z 01;31 .rz 01;31 .cab 01;31 .wim 01;31 .swm 01;31 .dwm 01;31 .esd 01;31 # image formats .avif 01;35 .jpg 01;35 .jpeg 01;35 .mjpg 01;35 .mjpeg 01;35 .gif 01;35 .bmp 01;35 .pbm 01;35 .pgm 01;35 .ppm 01;35 .tga 01;35 .xbm 01;35 .xpm 01;35 .tif 01;35 .tiff 01;35 .png 01;35 .svg 01;35 .svgz 01;35 .mng 01;35 .pcx 01;35 .mov 01;35 .mpg 01;35 .mpeg 01;35 .m2v 01;35 .mkv 01;35 .webm 01;35 .webp 01;35 .ogm 01;35 .mp4 01;35 .m4v 01;35 .mp4v 01;35 .vob 01;35 .qt 01;35 .nuv 01;35 .wmv 01;35 .asf 01;35 .rm 01;35 .rmvb 01;35 .flc 01;35 .avi 01;35 .fli 01;35 .flv 01;35 .gl 01;35 .dl 01;35 .xcf 01;35 .xwd 01;35 .yuv 01;35 .cgm 01;35 .emf 01;35 # https://wiki.xiph.org/MIME_Types_and_File_Extensions .ogv 01;35 .ogx 01;35 # audio formats .aac 00;36 .au 00;36 .flac 00;36 .m4a 00;36 .mid 00;36 .midi 00;36 .mka 00;36 .mp3 00;36 .mpc 00;36 .ogg 00;36 .ra 00;36 .wav 00;36 # https://wiki.xiph.org/MIME_Types_and_File_Extensions .oga 00;36 .opus 00;36 .spx 00;36 .xspf 00;36 # backup files *~ 00;90 *# 00;90 .bak 00;90 .crdownload 00;90 .dpkg-dist 00;90 .dpkg-new 00;90 .dpkg-old 00;90 .dpkg-tmp 00;90 .old 00;90 .orig 00;90 .part 00;90 .rej 00;90 .rpmnew 00;90 .rpmorig 00;90 .rpmsave 00;90 .swp 00;90 .tmp 00;90 .ucf-dist 00;90 .ucf-new 00;90 .ucf-old 00;90 # # Subsequent TERM or COLORTERM entries, can be used to add / override # config specific to those matching environment variables. </code></pre> <h2><a href="https://github.com/Randy8080/reference/blob/main/bash.md">Cheat sheet</a></h2> <h3>Getting started</h3> <h3>hello.sh</h3> <pre><code class="language-bash">#!/bin/bash VAR="world" echo "Hello $VAR!" # =&gt; Hello world! </code></pre> <p>Execute the script</p> <pre><code class="language-shell">$ bash hello.sh </code></pre> <h3>Variables</h3> <pre><code class="language-bash">NAME="John" echo ${NAME} # =&gt; John echo $NAME # =&gt; John echo "$NAME" # =&gt; John echo '$NAME' # =&gt; $NAME echo "${NAME}!" # =&gt; John! NAME = "John" # =&gt; Error (about space) </code></pre> <h3>Comments</h3> <pre><code class="language-bash"># This is an inline Bash comment. </code></pre> <pre><code class="language-bash">: ' This is a very neat comment in bash ' </code></pre> <p>Multi-line comments use <code>:'</code> to open and <code>'</code> to close</p> <h3>Arguments</h3> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>$1</code> … <code>$9</code></td> <td> Parameter 1 ... 9 </td> </tr> <tr> <td> <code>$0</code> </td> <td> Name of the script itself </td> </tr> <tr> <td> <code>$1</code> </td> <td> First argument </td> </tr> <tr> <td> <code>${10}</code> </td> <td> Positional parameter 10 </td> </tr> <tr> <td> <code>$#</code> </td> <td> Number of arguments </td> </tr> <tr> <td> <code>$$</code> </td> <td> Process id of the shell </td> </tr> <tr> <td> <code>$*</code> </td> <td> All arguments </td> </tr> <tr> <td> <code>$@</code> </td> <td> All arguments, starting from first </td> </tr> <tr> <td> <code>$-</code> </td> <td> Current options </td> </tr> <tr> <td> <code>$_</code> </td> <td> Last argument of the previous command </td> </tr> </tbody> </table> <p>See: <a href="http://wiki.bash-hackers.org/syntax/shellvars#special_parameters_and_shell_variables">Special parameters</a>.</p> <h3>Functions</h3> <pre><code class="language-bash">get_name() { echo "John" } echo "You are $(get_name)" </code></pre> <p>See: <a href="#functions-2">Functions</a></p> <h3>Conditionals {#conditionals-example}</h3> <pre><code class="language-bash">if [[ -z "$string" ]]; then echo "String is empty" elif [[ -n "$string" ]]; then echo "String is not empty" fi </code></pre> <p>See: <a href="#conditionals-2">Conditionals</a></p> <h3>Brace expansion</h3> <pre><code class="language-bash">echo {A,B}.js </code></pre> <hr> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>{A,B}</code> </td> <td> Same as <code>A B</code> </td> </tr> <tr> <td> <code>{A,B}.js</code> </td> <td> Same as <code>A.js B.js</code> </td> </tr> <tr> <td> <code>{1..5}</code> </td> <td> Same as <code>1 2 3 4 5</code> </td> </tr> </tbody> </table> <p>See: <a href="http://wiki.bash-hackers.org/syntax/expansion/brace">Brace expansion</a></p> <h3>Shell execution</h3> <pre><code class="language-bash">echo "I'm in $(PWD)" # Same echo "I'm in `pwd`" </code></pre> <p>See: <a href="http://wiki.bash-hackers.org/syntax/expansion/cmdsubst">Command substitution</a></p> <h2>Parameter expansions {.cols-3}</h2> <h3>Syntax {.row-span-2}</h3> <table> <thead> <tr> <th> Code </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>${FOO%suffix}</code> </td> <td> Remove suffix </td> </tr> <tr> <td> <code>${FOO#prefix}</code> </td> <td> Remove prefix </td> </tr> <tr> <td> <code>${FOO%%suffix}</code> </td> <td> Remove long suffix </td> </tr> <tr> <td> <code>${FOO##prefix}</code> </td> <td> Remove long prefix </td> </tr> <tr> <td> <code>${FOO/from/to}</code> </td> <td> Replace first match </td> </tr> <tr> <td> <code>${FOO//from/to}</code> </td> <td> Replace all </td> </tr> <tr> <td> <code>${FOO/%from/to}</code> </td> <td> Replace suffix </td> </tr> <tr> <td> <code>${FOO/#from/to}</code> </td> <td> Replace prefix </td> </tr> </tbody> </table> <h4>Substrings</h4> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>${FOO:0:3}</code> </td> <td> Substring <em>(position, length)</em> </td> </tr> <tr> <td> <code>${FOO:(-3):3}</code> </td> <td> Substring from the right </td> </tr> </tbody> </table> <h4>Length</h4> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>${#FOO}</code> </td> <td> Length of <code>$FOO</code> </td> </tr> </tbody> </table> <h4>Default values</h4> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>${FOO:-val}</code> </td> <td> <code>$FOO</code>, or <code>val</code> if unset </td> </tr> <tr> <td> <code>${FOO:=val}</code> </td> <td> Set <code>$FOO</code> to <code>val</code> if unset </td> </tr> <tr> <td> <code>${FOO:+val}</code> </td> <td> <code>val</code> if <code>$FOO</code> is set </td> </tr> <tr> <td> <code>${FOO:?message}</code> </td> <td> Show message and exit if <code>$FOO</code> is unset </td> </tr> </tbody> </table> <h3>Substitution</h3> <pre><code class="language-bash">echo ${food:-Cake} #=&gt; $food or "Cake" </code></pre> <pre><code class="language-bash">STR="/path/to/foo.cpp" echo ${STR%.cpp} # /path/to/foo echo ${STR%.cpp}.o # /path/to/foo.o echo ${STR%/*} # /path/to echo ${STR##*.} # cpp (extension) echo ${STR##*/} # foo.cpp (basepath) echo ${STR#*/} # path/to/foo.cpp echo ${STR##*/} # foo.cpp echo ${STR/foo/bar} # /path/to/bar.cpp </code></pre> <h3>Slicing</h3> <pre><code class="language-bash">name="John" echo ${name} # =&gt; John echo ${name:0:2} # =&gt; Jo echo ${name::2} # =&gt; Jo echo ${name::-1} # =&gt; Joh echo ${name:(-1)} # =&gt; n echo ${name:(-2)} # =&gt; hn echo ${name:(-2):2} # =&gt; hn length=2 echo ${name:0:length} # =&gt; Jo </code></pre> <p>See: <a href="http://wiki.bash-hackers.org/syntax/pe">Parameter expansion</a></p> <h3>basepath &amp; dirpath</h3> <pre><code class="language-bash">SRC="/path/to/foo.cpp" </code></pre> <pre><code class="language-bash">BASEPATH=${SRC##*/} echo $BASEPATH # =&gt; "foo.cpp" DIRPATH=${SRC%$BASEPATH} echo $DIRPATH # =&gt; "/path/to/" </code></pre> <h3>Transform</h3> <pre><code class="language-bash">STR="HELLO WORLD!" echo ${STR,} # =&gt; hELLO WORLD! echo ${STR,,} # =&gt; hello world! STR="hello world!" echo ${STR^} # =&gt; Hello world! echo ${STR^^} # =&gt; HELLO WORLD! ARR=(hello World) echo "${ARR[@],}" # =&gt; hello world echo "${ARR[@]^}" # =&gt; Hello World </code></pre> <h2>Arrays {.cols-3}</h2> <h3>Defining arrays</h3> <pre><code class="language-bash">Fruits=('Apple' 'Banana' 'Orange') Fruits[0]="Apple" Fruits[1]="Banana" Fruits[2]="Orange" ARRAY2=(foo{1..2}) # =&gt; foo1 foo2 ARRAY3=({A..D}) # =&gt; A B C D # declare construct declare -a Numbers=(1 2 3 4 5 6) </code></pre> <h3>Indexing</h3> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>${Fruits[0]}</code> </td> <td> First element </td> </tr> <tr> <td> <code>${Fruits[-1]}</code> </td> <td> Last element </td> </tr> <tr> <td> <code>${Fruits[*]}</code> </td> <td> All elements </td> </tr> <tr> <td> <code>${Fruits[@]}</code> </td> <td> All elements </td> </tr> <tr> <td> <code>${#Fruits[@]}</code> </td> <td> Number of all </td> </tr> <tr> <td> <code>${#Fruits}</code> </td> <td> Length of 1st </td> </tr> <tr> <td> <code>${#Fruits[3]}</code> </td> <td> Length of nth </td> </tr> <tr> <td> <code>${Fruits[@]:3:2}</code> </td> <td> Range </td> </tr> <tr> <td> <code>${!Fruits[@]}</code> </td> <td> Keys of all </td> </tr> </tbody> </table> <h3>Iteration</h3> <pre><code class="language-bash">Fruits=('Apple' 'Banana' 'Orange') for e in "${Fruits[@]}"; do echo $e done </code></pre> <h4>With index</h4> <pre><code class="language-bash">for i in "${!Fruits[@]}"; do printf "%s\t%s\n" "$i" "${Fruits[$i]}" done </code></pre> <h3>Operations {.col-span-2}</h3> <pre><code class="language-bash">Fruits=("${Fruits[@]}" "Watermelon") # Push Fruits+=('Watermelon') # Also Push Fruits=( ${Fruits[@]/Ap*/} ) # Remove by regex match unset Fruits[2] # Remove one item Fruits=("${Fruits[@]}") # Duplicate Fruits=("${Fruits[@]}" "${Veggies[@]}") # Concatenate lines=(`cat "logfile"`) # Read from file </code></pre> <h3>Arrays as arguments</h3> <pre><code class="language-bash">function extract() { local -n myarray=$1 local idx=$2 echo "${myarray[$idx]}" } Fruits=('Apple' 'Banana' 'Orange') extract Fruits 2 # =&gt; Orangle </code></pre> <h2>Dictionaries {.cols-3}</h2> <h3>Defining</h3> <pre><code class="language-bash">declare -A sounds </code></pre> <pre><code class="language-bash">sounds[dog]="bark" sounds[cow]="moo" sounds[bird]="tweet" sounds[wolf]="howl" </code></pre> <h3>Working with dictionaries</h3> <pre><code class="language-bash">echo ${sounds[dog]} # Dog's sound echo ${sounds[@]} # All values echo ${!sounds[@]} # All keys echo ${#sounds[@]} # Number of elements unset sounds[dog] # Delete dog </code></pre> <h3>Iteration</h3> <pre><code class="language-bash">for val in "${sounds[@]}"; do echo $val done </code></pre> <hr> <pre><code class="language-bash">for key in "${!sounds[@]}"; do echo $key done </code></pre> <h2>Conditionals {.cols-3}</h2> <h3>Integer conditions</h3> <table> <thead> <tr> <th> Condition </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>[[ NUM -eq NUM ]]</code> </td> <td> <yel> Eq </yel>ual </td> </tr> <tr> <td> <code>[[ NUM -ne NUM ]]</code> </td> <td> <yel> N </yel>ot <yel> e </yel>qual </td> </tr> <tr> <td> <code>[[ NUM -lt NUM ]]</code> </td> <td> <yel> L </yel>ess <yel> t </yel>han </td> </tr> <tr> <td> <code>[[ NUM -le NUM ]]</code> </td> <td> <yel> L </yel>ess than or <yel> e </yel>qual </td> </tr> <tr> <td> <code>[[ NUM -gt NUM ]]</code> </td> <td> <yel> G </yel>reater <yel> t </yel>han </td> </tr> <tr> <td> <code>[[ NUM -ge NUM ]]</code> </td> <td> <yel> G </yel>reater than or <yel> e </yel>qual </td> </tr> <tr> <td> <code>(( NUM &lt; NUM ))</code> </td> <td> Less than </td> </tr> <tr> <td> <code>(( NUM &lt;= NUM ))</code> </td> <td> Less than or equal </td> </tr> <tr> <td> <code>(( NUM &gt; NUM ))</code> </td> <td> Greater than </td> </tr> <tr> <td> <code>(( NUM &gt;= NUM ))</code> </td> <td> Greater than or equal </td> </tr> </tbody> </table> <h3>String conditions</h3> <table> <thead> <tr> <th> Condition </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>[[ -z STR ]]</code> </td> <td> Empty string </td> </tr> <tr> <td> <code>[[ -n STR ]]</code> </td> <td> <yel> N </yel>ot empty string </td> </tr> <tr> <td> <code>[[ STR == STR ]]</code> </td> <td> Equal </td> </tr> <tr> <td> <code>[[ STR = STR ]]</code> </td> <td> Equal (Same above) </td> </tr> <tr> <td> <code>[[ STR &lt; STR ]]</code> </td> <td> Less than <em>(ASCII)</em> </td> </tr> <tr> <td> <code>[[ STR &gt; STR ]]</code> </td> <td> Greater than <em>(ASCII)</em> </td> </tr> <tr> <td> <code>[[ STR != STR ]]</code> </td> <td> Not Equal </td> </tr> <tr> <td> <code>[[ STR =~ STR ]]</code> </td> <td> Regexp </td> </tr> </tbody> </table> <h3>Example {.row-span-3}</h3> <h4>String</h4> <pre><code class="language-bash">if [[ -z "$string" ]]; then echo "String is empty" elif [[ -n "$string" ]]; then echo "String is not empty" else echo "This never happens" fi </code></pre> <h4>Combinations</h4> <pre><code class="language-bash">if [[ X &amp;&amp; Y ]]; then ... fi </code></pre> <h4>Equal</h4> <pre><code class="language-bash">if [[ "$A" == "$B" ]]; then ... fi </code></pre> <h4>Regex</h4> <pre><code class="language-bash">if [[ '1. abc' =~ ([a-z]+) ]]; then echo ${BASH_REMATCH[1]} fi </code></pre> <h4>Smaller</h4> <pre><code class="language-bash">if (( $a &lt; $b )); then echo "$a is smaller than $b" fi </code></pre> <h4>Exists</h4> <pre><code class="language-bash">if [[ -e "file.txt" ]]; then echo "file exists" fi </code></pre> <h3>File conditions {.row-span-2}</h3> <table> <thead> <tr> <th> Condition </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>[[ -e FILE ]]</code> </td> <td> <yel> E </yel>xists </td> </tr> <tr> <td> <code>[[ -d FILE ]]</code> </td> <td> <yel> D </yel>irectory </td> </tr> <tr> <td> <code>[[ -f FILE ]]</code> </td> <td> <yel> F </yel>ile </td> </tr> <tr> <td> <code>[[ -h FILE ]]</code> </td> <td> Symlink </td> </tr> <tr> <td> <code>[[ -s FILE ]]</code> </td> <td> Size is &gt; 0 bytes </td> </tr> <tr> <td> <code>[[ -r FILE ]]</code> </td> <td> <yel> R </yel>eadable </td> </tr> <tr> <td> <code>[[ -w FILE ]]</code> </td> <td> <yel> W </yel>ritable </td> </tr> <tr> <td> <code>[[ -x FILE ]]</code> </td> <td> Executable </td> </tr> <tr> <td> <code>[[ f1 -nt f2 ]]</code> </td> <td> f1 <yel> n </yel>ewer <yel> t </yel>han f2 </td> </tr> <tr> <td> <code>[[ f1 -ot f2 ]]</code> </td> <td> f2 <yel> o </yel>lder <yel> t </yel>han f1 </td> </tr> <tr> <td> <code>[[ f1 -ef f2 ]]</code> </td> <td> Same files </td> </tr> </tbody> </table> <h3>More conditions</h3> <table> <thead> <tr> <th> Condition </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>[[ -o noclobber ]]</code> </td> <td> If OPTION is enabled </td> </tr> <tr> <td> <code>[[ ! EXPR ]]</code> </td> <td> Not </td> </tr> <tr> <td> <code>[[ X &amp;&amp; Y ]]</code> </td> <td> And </td> </tr> <tr> <td> <code>[[ X || Y ]]</code> </td> <td> Or </td> </tr> </tbody> </table> <h3>logical and, or</h3> <pre><code class="language-bash">if [ "$1" = 'y' -a $2 -gt 0 ]; then echo "yes" fi if [ "$1" = 'n' -o $2 -lt 0 ]; then echo "no" fi </code></pre> <h2>Loops {.cols-3}</h2> <h3>Basic for loop</h3> <pre><code class="language-bash">for i in /etc/rc.*; do echo $i done </code></pre> <h3>C-like for loop</h3> <pre><code class="language-bash">for ((i = 0 ; i &lt; 100 ; i++)); do echo $i done </code></pre> <h3>Ranges {.row-span-2}</h3> <pre><code class="language-bash">for i in {1..5}; do echo "Welcome $i" done </code></pre> <h4>With step size</h4> <pre><code class="language-bash">for i in {5..50..5}; do echo "Welcome $i" done </code></pre> <h3>Auto increment</h3> <pre><code class="language-bash">i=1 while [[ $i -lt 4 ]]; do echo "Number: $i" ((i++)) done </code></pre> <h3>Auto decrement</h3> <pre><code class="language-bash">i=3 while [[ $i -gt 0 ]]; do echo "Number: $i" ((i--)) done </code></pre> <h3>Continue</h3> <pre><code class="language-bash">for number in $(seq 1 3); do if [[ $number == 2 ]]; then continue; fi echo "$number" done </code></pre> <h3>Break</h3> <pre><code class="language-bash">for number in $(seq 1 3); do if [[ $number == 2 ]]; then # Skip entire rest of loop. break; fi # This will only print 1 echo "$number" done </code></pre> <h3>Until</h3> <pre><code class="language-bash">count=0 until [ $count -gt 10 ]; do echo "$count" ((count++)) done </code></pre> <h3>Forever</h3> <pre><code class="language-bash">while true; do # here is some code. done </code></pre> <h3>Forever (shorthand)</h3> <pre><code class="language-bash">while :; do # here is some code. done </code></pre> <h3>Reading lines</h3> <pre><code class="language-bash">cat file.txt | while read line; do echo $line done </code></pre> <h2>Functions {.cols-3}</h2> <h3>Defining functions</h3> <pre><code class="language-bash">myfunc() { echo "hello $1" } </code></pre> <pre><code class="language-bash"># Same as above (alternate syntax) function myfunc() { echo "hello $1" } </code></pre> <pre><code class="language-bash">myfunc "John" </code></pre> <h3>Returning values</h3> <pre><code class="language-bash">myfunc() { local myresult='some value' echo $myresult } </code></pre> <pre><code class="language-bash">result="$(myfunc)" </code></pre> <h3>Raising errors</h3> <pre><code class="language-bash">myfunc() { return 1 } </code></pre> <pre><code class="language-bash">if myfunc; then echo "success" else echo "failure" fi </code></pre> <h2>Options {.cols-2}</h2> <h3>Options</h3> <pre><code class="language-bash"># Avoid overlay files # (echo "hi" &gt; foo) set -o noclobber # Used to exit upon error # avoiding cascading errors set -o errexit # Unveils hidden failures set -o pipefail # Exposes unset variables set -o nounset </code></pre> <h3>Glob options</h3> <pre><code class="language-bash"># Non-matching globs are removed # ('*.foo' =&gt; '') shopt -s nullglob # Non-matching globs throw errors shopt -s failglob # Case insensitive globs shopt -s nocaseglob # Wildcards match dotfiles # ("*.sh" =&gt; ".foo.sh") shopt -s dotglob # Allow ** for recursive matches # ('lib/**/*.rb' =&gt; 'lib/a/b/c.rb') shopt -s globstar </code></pre> <h2>History {.cols-2}</h2> <h3>Commands</h3> <table> <thead> <tr> <th> Command </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>history</code> </td> <td> Show history </td> </tr> <tr> <td> <code>shopt -s histverify</code> </td> <td> Don't execute expanded result immediately </td> </tr> </tbody> </table> <h3>Expansions</h3> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>!$</code> </td> <td> Expand last parameter of most recent command </td> </tr> <tr> <td> <code>!*</code> </td> <td> Expand all parameters of most recent command </td> </tr> <tr> <td> <code>!-n</code> </td> <td> Expand <code>n</code>th most recent command </td> </tr> <tr> <td> <code>!n</code> </td> <td> Expand <code>n</code>th command in history </td> </tr> <tr> <td> <code>!&lt;command&gt;</code> </td> <td> Expand most recent invocation of command <code>&lt;command&gt;</code> </td> </tr> </tbody> </table> <h3>Operations</h3> <table> <thead> <tr> <th> Code </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>!!</code> </td> <td> Execute last command again </td> </tr> <tr> <td> <code>!!:s/&lt;FROM&gt;/&lt;TO&gt;/</code> </td> <td> Replace first occurrence of <code>&lt;FROM&gt;</code> to <code>&lt;TO&gt;</code> in most recent command </td> </tr> <tr> <td> <code>!!:gs/&lt;FROM&gt;/&lt;TO&gt;/</code> </td> <td> Replace all occurrences of <code>&lt;FROM&gt;</code> to <code>&lt;TO&gt;</code> in most recent command </td> </tr> <tr> <td> <code>!$:t</code> </td> <td> Expand only basename from last parameter of most recent command </td> </tr> <tr> <td> <code>!$:h</code> </td> <td> Expand only directory from last parameter of most recent command </td> </tr> </tbody> </table> <p><code>!!</code> and <code>!$</code> can be replaced with any valid expansion.</p> <h3>Slices</h3> <table> <thead> <tr> <th> Code </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>!!:n</code> </td> <td> Expand only <code>n</code>th token from most recent command (command is <code>0</code>; first argument is <code>1</code>) </td> </tr> <tr> <td> <code>!^</code> </td> <td> Expand first argument from most recent command </td> </tr> <tr> <td> <code>!$</code> </td> <td> Expand last token from most recent command </td> </tr> <tr> <td> <code>!!:n-m</code> </td> <td> Expand range of tokens from most recent command </td> </tr> <tr> <td> <code>!!:n-$</code> </td> <td> Expand <code>n</code>th token to last from most recent command </td> </tr> </tbody> </table> <p><code>!!</code> can be replaced with any valid expansion i.e. <code>!cat</code>, <code>!-2</code>, <code>!42</code>, etc.</p> <h2>Miscellaneous {.cols-3}</h2> <h3>Numeric calculations</h3> <pre><code class="language-bash">$((a + 200)) # Add 200 to $a </code></pre> <pre><code class="language-bash">$(($RANDOM%200)) # Random number 0..199 </code></pre> <h3>Subshells</h3> <pre><code class="language-bash">(cd somedir; echo "I'm now in $PWD") pwd # still in first directory </code></pre> <h3>Inspecting commands</h3> <pre><code class="language-bash">command -V cd #=&gt; "cd is a function/alias/whatever" </code></pre> <h3>Redirection {.row-span-2 .col-span-2}</h3> <pre><code class="language-bash">python hello.py &gt; output.txt # stdout to (file) python hello.py &gt;&gt; output.txt # stdout to (file), append python hello.py 2&gt; error.log # stderr to (file) python hello.py 2&gt;&amp;1 # stderr to stdout python hello.py 2&gt;/dev/null # stderr to (null) python hello.py &amp;&gt;/dev/null # stdout and stderr to (null) </code></pre> <pre><code class="language-bash">python hello.py &lt; foo.txt # feed foo.txt to stdin for python </code></pre> <h3>Source relative</h3> <pre><code class="language-bash">source "${0%/*}/../share/foo.sh" </code></pre> <h3>Directory of script</h3> <pre><code class="language-bash">DIR="${0%/*}" </code></pre> <h3>Case/switch</h3> <pre><code class="language-bash">case "$1" in start | up) vagrant up ;; *) echo "Usage: $0 {start|stop|ssh}" ;; esac </code></pre> <h3>Trap errors {.col-span-2}</h3> <pre><code class="language-bash">trap 'echo Error at about $LINENO' ERR </code></pre> <p>or</p> <pre><code class="language-bash">traperr() { echo "ERROR: ${BASH_SOURCE[1]} at about ${BASH_LINENO[0]}" } set -o errtrace trap traperr ERR </code></pre> <h3>printf</h3> <pre><code class="language-bash">printf "Hello %s, I'm %s" Sven Olga #=&gt; "Hello Sven, I'm Olga printf "1 + 1 = %d" 2 #=&gt; "1 + 1 = 2" printf "Print a float: %f" 2 #=&gt; "Print a float: 2.000000" </code></pre> <h3>Getting options {.col-span-2}</h3> <pre><code class="language-bash">while [[ "$1" =~ ^- &amp;&amp; ! "$1" == "--" ]]; do case $1 in -V | --version ) echo $version exit ;; -s | --string ) shift; string=$1 ;; -f | --flag ) flag=1 ;; esac; shift; done if [[ "$1" == '--' ]]; then shift; fi </code></pre> <h3>Check for command's result {.col-span-2}</h3> <pre><code class="language-bash">if ping -c 1 google.com; then echo "It appears you have a working internet connection" fi </code></pre> <h3>Special variables {.row-span-2}</h3> <table> <thead> <tr> <th> Expression </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>$?</code> </td> <td> Exit status of last task </td> </tr> <tr> <td> <code>$!</code> </td> <td> PID of last background task </td> </tr> <tr> <td> <code>$$</code> </td> <td> PID of shell </td> </tr> <tr> <td> <code>$0</code> </td> <td> Filename of the shell script </td> </tr> </tbody> </table> <p>See <a href="http://wiki.bash-hackers.org/syntax/shellvars#special_parameters_and_shell_variables">Special parameters</a>.</p> <h3>Grep check {.col-span-2}</h3> <pre><code class="language-bash">if grep -q 'foo' ~/.bash_history; then echo "You appear to have typed 'foo' in the past" fi </code></pre> <h3>Backslash escapes {.row-span-2}</h3> <ul> <li>&nbsp;</li> <li>!</li> <li>"</li> <li>#</li> <li>&amp;</li> <li>'</li> <li>(</li> <li>)</li> <li>,</li> <li>;</li> <li>&lt;</li> <li>&gt;</li> <li>[</li> <li>|</li> <li>\</li> <li>]</li> <li>^</li> <li>{</li> <li>}</li> <li>`</li> <li>$</li> <li>*</li> <li>?<br> {.cols-4 .style-none}</li> </ul> <p>Escape these special characters with <code>\</code></p> <h3>Heredoc</h3> <pre><code class="language-sh">cat &lt;&lt;END hello world END </code></pre> <h3>Go to previous directory</h3> <pre><code class="language-bash">pwd # /home/user/foo cd bar/ pwd # /home/user/foo/bar cd - pwd # /home/user/foo </code></pre> <h3>Reading input</h3> <pre><code class="language-bash">echo -n "Proceed? [y/n]: " read ans echo $ans </code></pre> <pre><code class="language-bash">read -n 1 ans # Just one character </code></pre> <h3>Conditional execution</h3> <pre><code class="language-bash">git commit &amp;&amp; git push git commit || echo "Commit failed" </code></pre> <h3>Strict mode</h3> <pre><code class="language-bash">set -euo pipefail IFS=$'\n\t' </code></pre> <p>See: <a href="http://redsymbol.net/articles/unofficial-bash-strict-mode/">Unofficial bash strict mode</a></p> https://tedneward.github.io/Research/tools/goblintools/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/goblintools/index.html <p><a href="https://goblin.tools/">Website</a> | Android and iOS versions available | Closed source?</p> <p>Tools: <a href="https://goblin.tools/">Magic ToDo</a> | <a href="https://goblin.tools/Formalizer">Formalizer</a> - Turn the spicy thoughts into classy ones, or vice versa | <a href="https://goblin.tools/Judge">Judge</a> - am I misreading the tone of this? | <a href="https://goblin.tools/Estimator">Estimator</a> - Just tell me how long this is probably gonna take | <a href="https://goblin.tools/Compiler">Compiler</a> - Compile my braindump into a list of tasks | <a href="https://goblin.tools/Chef">Chef</a> - What do I want for lunch today?</p> https://tedneward.github.io/Research/tools/graphviz/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/graphviz/index.html <p><a href="https://www.graphviz.org/">Website</a> | <a href="https://gitlab.com/graphviz/graphviz">Source</a> | <a href="https://www.graphviz.org/documentation/">Docs</a></p> https://tedneward.github.io/Research/tools/hammerspoon/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hammerspoon/index.html <p><a href="https://www.hammerspoon.org/">Website</a> | <a href="https://github.com/Hammerspoon/hammerspoon">Source</a> | <a href="https://www.hammerspoon.org/docs/index.html">API Documentation</a></p> <p>Basically wires up a <a href="/languages/lua">Lua</a> engine with the various APIs exposed by macOS applications.</p> <p><strong>Example: Hello World</strong></p> <p>In your init.lua place the following:</p> <pre><code>hs.hotkey.bind({"cmd", "alt", "ctrl"}, "W", function() hs.alert.show("Hello World!") end) </code></pre> <p>Then save the file, click on the Hammerspoon menubar icon and choose Reload Config. You should now find that pressing Cmd+Option+Ctrl+W will display a Hello World notification on your screen.</p> <p>What is happening here is that we’re telling Hammerspoon to bind an anonymous function to a particular hotkey. The hotkey is specified by a table of modifier keys (Cmd, Option, and Ctrl in this case) and a normal key (W). An anonymous function is simply one that doesn’t have a name. We could have defined the alert function separately with a name and passed that name to <code>hs.hotkey.bind()</code>, but Lua makes it easy to define the functions inline.</p> https://tedneward.github.io/Research/tools/hexo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hexo/index.html <p><a href="https://hexo.io/">Website</a> | <a href="https://github.com/hexojs/hexo">Source</a></p> https://tedneward.github.io/Research/tools/hopper/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/hopper/index.html <p><a href="https://www.hopperapp.com/">Website</a> | Commercial (free limited trial)</p> https://tedneward.github.io/Research/tools/iced/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/iced/index.html <p>It can be used for static analysis of x86/x64 binaries, to rewrite code (eg. remove garbage instructions), to relocate code or as a disassembler.</p> <p>✔️Supports all Intel and AMD instructions<br> ✔️Supports .NET, Rust, JavaScript (WebAssembly)<br> ✔️The formatter supports masm, nasm, gas (AT&amp;T), Intel (XED) and there are many options to customize the output<br> ✔️The decoder is 4x+ faster than other similar libraries and doesn't allocate any memory<br> ✔️Small decoded instructions, only 32 bytes<br> ✔️High level Assembler (.NET) providing a simple and lean syntax (e.g asm.mov(eax, edx)))<br> ✔️The encoder can be used to re-encode decoded instructions at any address<br> ✔️API to get instruction info, eg. read/written registers, memory and rflags bits; CPUID feature flag, flow control info, etc<br> ✔️All instructions are tested (decode, encode, format, instruction info)</p> <p><a href="https://github.com/0xd4d/iced">Source</a></p> https://tedneward.github.io/Research/tools/ilspy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ilspy/index.html <p><a href="http://www.ilspy.net/">Website</a> | <a href="https://github.com/icsharpcode/ILSpy">Source</a></p> https://tedneward.github.io/Research/tools/injectdso/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/injectdso/index.html <p>It currently consists of injectdll (for Microsoft Windows), injectdylib (for MacOS X) and injectoid (for Android). Support for Linux (injectso) is planned for the near future as well.</p> <p>injectdso may be useful for tasks like:</p> <ul> <li> <p>Discovering and testing sandbox escape exploits</p> </li> <li> <p>Testing properties of heap allocators on closed source applications</p> </li> <li> <p>Hooking higher-level programming language runtimes</p> </li> </ul> <p>...and probably many more.</p> <p><a href="https://github.com/huku-/injectdso">Github</a></p> https://tedneward.github.io/Research/tools/jandex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jandex/index.html <p><a href="https://smallrye.io/jandex/">Website</a> | <a href="https://github.com/smallrye/jandex/">Source</a></p> https://tedneward.github.io/Research/tools/jbake/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jbake/index.html <p>JBake is a Java based open source static site/blog generator for developers.<br> * Supports AsciiDoc, Markdown and good old HTML formatted content.<br> * Structure your content any way you see fit.<br> * RSS feed, archive and tag support.<br> * View draft content before publishing it and making it available to the world.<br> * Freemarker, Groovy, Thymeleaf &amp; Jade based templates &amp; scripting support.<br> * Easily integrate CSS frameworks such as Bootstrap and Foundation.<br> * Add as much metadata to content as you like, also exposed to templates.<br> * Store your site content in Dropbox, CVS, SVN, Git or whatever you want.</p> <p><a href="http://jbake.org/">Website</a> | <a href="https://github.com/jbake-org/jbake">Source</a></p> https://tedneward.github.io/Research/tools/jdepend/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jdepend/index.html <p><a href="https://github.com/clarkware/jdepend">Source</a></p> https://tedneward.github.io/Research/tools/jlex/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jlex/index.html <p><a href="https://www.cs.princeton.edu/~appel/modern/java/JLex/">Website</a></p> <p>Last updated 2003.</p> https://tedneward.github.io/Research/tools/jxray/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/jxray/index.html <p><a href="https://jxray.com/">Website</a> Commercial</p> https://tedneward.github.io/Research/tools/kitty/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/kitty/index.html <p><a href="https://sw.kovidgoyal.net/kitty/">Website</a> | <a href="https://github.com/kovidgoyal/kitty">Source</a></p> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://www.howtogeek.com/kitty-is-the-open-source-mac-terminal-that-iterm-wishes-it-could-be/">Kitty is the open-source Mac terminal that iTerm wishes it could be</a></li> </ul> https://tedneward.github.io/Research/tools/lazarus/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lazarus/index.html <p><a href="https://www.lazarus-ide.org/">Website</a> | <a href="https://wiki.lazarus.freepascal.org/index.php/Getting_Lazarus">Source</a></p> https://tedneward.github.io/Research/tools/lighttable/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lighttable/index.html <p><a href="http://lighttable.com/">Website</a> | <a href="https://github.com/LightTable/LightTable">Source</a></p> https://tedneward.github.io/Research/tools/liverestore/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/liverestore/index.html <p><a href="https://www.system-rescue.org/">Website</a> | <a href="https://www.system-rescue.org/Download/">Download</a> | <a href="https://www.system-rescue.org/Installing-SystemRescue-on-a-USB-memory-stick/">Bootable USB</a></p> <p>A live bootable Linux environment packed with more than 300 utilities for unbootable Windows machines. It's under 1GB, so it's relatively lightweight on system resources and USB storage, allowing you to store more tools on the USB drive.</p> <p>It shines where Windows tools struggle—like reading locked files or running raw disk diagnostics before attempting repairs. Unlike vendor recovery disks, which often wipe your system, SystemRescue preserves everything, allowing you to copy important files to external drives first.</p> <p>It supports native access to Windows NTFS drives with the NTFS-3g driver and includes powerful utilities such as GParted for partition management, TestDisk for recovering deleted partitions, and ddrescue for cloning failing drives sector-by-sector right out of the box.</p> <p>You can even launch Firefox to check for system and driver updates, or mount your PC to a network for backups. When your PC won't boot, is affected by ransomware, or is crashing randomly, SystemRescue is your go-to tool for data recovery and diagnosis. It’s versatile, fast, and doesn’t rely on the health of your Windows install, making it invaluable for serious recovery jobs via external storage.</p> https://tedneward.github.io/Research/tools/lombok/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/lombok/index.html <p><a href="https://projectlombok.org/">Website</a> | <a href="https://github.com/projectlombok/lombok">Source</a></p> https://tedneward.github.io/Research/tools/gdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gdb/index.html <p><a href="https://www.sourceware.org/gdb/current/onlinedocs/gdb.html">Docs</a></p> https://tedneward.github.io/Research/tools/ghostly/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ghostly/index.html <p><a href="https://ghostlyapp.net/">Website</a> | <a href="https://github.com/patriksvensson/ghostly">Source</a></p> <p>To get the most out of Ghostly, you can use the Ghostly Query Language (shortened GQL) to search among your pull request, issues, releases, vulnerabilities and commits. All queries made with GQL are local and does not require an internet connection.</p> <h3>Examples</h3> <table> <thead> <tr> <th>Query </th> <th> Explanation</th> </tr> </thead> <tbody> <tr> <td><code>(org:microsoft OR org:cake-build) AND is:pullrequest AND is:open</code> </td> <td> Return all open pull requests that originiated in a repository belonging to either the microsoft or cake-build organisation.</td> </tr> <tr> <td><code>is:pullrequest AND is:open AND !muted AND mentions:patriksvensson</code> </td> <td> Return all open pullrequests that you haven't muted that mentions the user patriksvensson in the comment.</td> </tr> </tbody> </table> https://tedneward.github.io/Research/tools/gitmal/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gitmal/index.html <p><a href="https://github.com/antonmedv/gitmal">Source</a></p> <p>Docker: <code>docker run --rm -v $(pwd):/repo antonmedv/gitmal /repo</code></p> <ul> <li><code>gitmal</code> generates pages in <code>./output</code> based on repo in the cwd</li> <li><code>gitmal help</code> to get list of available options</li> </ul> <p>Here are a few examples of repos:</p> <ul> <li><a href="https://git.medv.io/zx/">git.medv.io/zx/</a> — github.com/google/zx</li> <li><a href="https://git.medv.io/zig/">git.medv.io/zig/</a> — codeberg.org/ziglang/zig (light theme)</li> <li><a href="https://git.medv.io/my-badges/">git.medv.io/my-badges/</a> — github.com/my-badges/my-badges</li> </ul> <h2>Reading</h2> <ul> <li><a href="https://github.com/antonmedv/gitmal/blob/master/docs/how-to-self-host-a-git-repository.md">How to Self-Host a Git Repository?</a></li> </ul> https://tedneward.github.io/Research/tools/gluonjs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gluonjs/index.html <p><a href="https://gluonjs.org/">Website</a> | <a href="https://github.com/gluon-framework/gluon">Source</a></p> https://tedneward.github.io/Research/tools/gentle/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gentle/index.html <p><a href="http://gentle.compilertools.net/">Website</a></p> https://tedneward.github.io/Research/tools/git/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/git/index.html <p>Just the tool itself; considered separate from <a href="/clouds/github">Github</a> since it is used by a variety of other source-control providers like Bitbucket.</p> <p><a href="https://www.git-scm.com">Website</a> | <a href="http://git-scm.com/docs">Docs</a></p> <h3>Graphical cheat sheet</h3> <p><img src="https://media.licdn.com/dms/image/D5622AQF8ksdiwWBvyw/feedshare-shrink_800/0/1684905537126?e=1689811200&amp;v=beta&amp;t=P5Pt5DenPxOmgJ_kUJ1-9fq8LykPNB6hrtbJO1_7AW4" alt=""></p> <h3>Cheat sheet</h3> <p><code>git init</code>: Create repository in directory</p> <p><code>git clone {url}</code>: copy git repo at {url} to local machine</p> <p><code>git status</code>: show modified files in repo</p> <p><code>git log</code>: show commit history</p> <p><code>git add -A</code>: add (stage) changed files for commit</p> <p><code>git commit -m 'message'</code>: commit changes using 'message' as the commit message</p> <p><code>git pull origin main</code>: get up to date changes from branch "main"</p> <p><code>git push origin main</code>: push local changes to remote "origin" on branch "main"</p> <p><code>git branch</code>: list all local branches</p> <p><code>git branch {branchname}</code>: create a new branch</p> <p><code>git checkout {branchname}</code>: switch from current branch to named branch</p> <p><code>git branch -m {newbranchname}</code>: rename current branch</p> <p><code>git branch -d {branchname}</code>: delete named branch</p> <p><code>git merge develop</code>: merge branch "develop" into current branch</p> <p><code>git rm {filename}</code>: delete (and stage) file</p> <p><code>git stash</code>: save modified files and staged changes</p> <p><code>git diff {branch1} {branch2}</code>: diff between named branches</p> <p><code>git rebase {branch}</code>: put commits of current branch ahead of named branch</p> <ul> <li><a href="http://res.cloudinary.com/hy4kyit2a/image/upload/SF_git_cheatsheet.pdf">SalesForce Cheat Sheet</a></li> </ul> <h2>Reading</h2> <ul> <li><a href="https://www.atlassian.com/git/">Atlassian Git - Tutorials &amp; Workflows</a></li> <li><a href="https://danielmiessler.com/study/git/">A git Primer</a> - Daniel Miessler</li> <li><a href="http://marklodato.github.io/visual-git-guide/index-en.html">A Visual Git Reference</a> - Mark Lodato</li> <li><a href="http://blog.anvard.org/conversational-git/">Conversational Git</a></li> <li><a href="http://rogerdudler.github.io/git-guide/">git - the simple guide</a></li> <li><a href="http://eagain.net/articles/git-for-computer-scientists/">Git for Computer Scientists</a></li> <li><a href="http://hoth.entp.com/output/git_for_designers.html">Git For Designers</a></li> <li><a href="https://jwiegley.github.io/git-from-the-bottom-up/">Git From The Bottom Up</a> - J. Wiegley</li> <li><a href="http://gitimmersion.com">Git Immersion</a></li> <li><a href="http://cbx33.github.io/gitt/index.html">Git In The Trenches</a></li> <li><a href="https://github.com/pluralsight/git-internals-pdf/raw/master/drafts/peepcode-git.pdf">Git internals</a> - Scott Chacon (PDF)</li> <li><a href="http://www-cs-students.stanford.edu/~blynn/gitmagic/">Git Magic</a></li> <li><a href="http://goalkicker.com/GitBook">Git Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="https://www.oreilly.com/library/view/git-pocket-guide/9781449327507">Git Pocket Guide</a> - Richard E. Silverman</li> <li><a href="https://web.archive.org/web/20170602211147/http://gitref.org/">Git Reference</a></li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/git">Git Succinctly, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://www.tutorialspoint.com/git/">Git Tutorial</a> - Tutorials Point Ltd. (HTML)</li> <li><a href="https://product.hubspot.com/blog/git-and-github-tutorial-for-beginners">Git-Tutorial For-Beginners</a> - HubSpot Product Team</li> <li><a href="https://web.archive.org/web/20210910133251/http://documentup.com/skwp/git-workflows-book">Git Workflows</a> - Yan Pritzker <em>(:card_file_box: archived)</em></li> <li><a href="https://happygitwithr.com">Happy Git and GitHub for the useR</a> - Jenny Bryan</li> <li><a href="https://github.com/eonist/How-to-collaborate-on-github">How to Collaborate on Github</a> - André J</li> <li><a href="https://www.freecodecamp.org/news/how-to-write-better-git-commit-messages/">How to Write Better Git Commit Messages</a></li> <li><a href="https://launchschool.com/books/git">Introduction to Git and Github</a> - Launch School</li> <li><a href="http://cse.unl.edu/~cbourke/gitTutorial.pdf">Introduction to Git and Github - Tutorial</a> - Dr. Chris Bourke (PDF)</li> <li><a href="https://github.com/bobbyiliev/introduction-to-git-and-github-ebook">Introduction to Git and GitHub eBook</a> - Bobby Iliev (Markdown, PDF)</li> <li><a href="http://www.git-tower.com/learn/git/ebook/command-line/introduction">Learn Git - Learn Version Control with Git</a> - Tobias Günther</li> <li><a href="https://ohmygit.org/">Oh My Git!</a> (<a href="https://blinry.itch.io/oh-my-git">Docs</a>, <a href="https://github.com/git-learning-game/oh-my-git">Source</a>): an open source Git learning game</li> <li><a href="http://git-scm.com/book/en/v2">Pro Git</a> - Scott Chacon</li> <li><a href="https://leanpub.com/progitreedited/read">Pro Git Reedited</a> - Jon Forrest</li> <li><a href="https://web.archive.org/web/20161121145226/http://rypress.com:80/tutorials/git/index">Ry's Git Tutorial</a> - Ryan Hodson</li> <li><a href="http://think-like-a-git.net">Think Like (a) Git: A Guide for the Perplexed</a></li> <li><a href="https://blog.stackademic.com/git-rebase-explained-like-youre-new-to-git-263c19fa86ec">https://blog.stackademic.com/git-rebase-explained-like-youre-new-to-git-263c19fa86ec</a></li> <li><a href="https://andrewlock.net/working-with-stacked-branches-in-git-part-1/">Working with Stacked Branches in Git</a></li> </ul> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://www.howtogeek.com/i-turned-git-into-a-private-github-free-sync-system-between-my-own-machines-and-it-completely-changed-how-i-work/">How to sync files between two local machines using Git</a> - Basic git hosting over SSH</li> </ul> <h3>Websites</h3> <ul> <li><a href="http://www.gitguys.com/">GitGuys</a></li> <li><a href="http://learngitbranching.js.org/">Learn Git Branching - the most visual and interactive way to learn Git on the web</a></li> <li><a href="http://try.github.io/levels/1/challenges/1">tryGit - A fun interactive way to learn Git.</a></li> <li><a href="https://gitjin.com/t">GitJin</a></li> </ul> <h3>Videos</h3> <ul> <li><a href="http://git-scm.com/videos">git-scm - Video Tutorials</a></li> <li><a href="https://www.youtube.com/playlist?list=PL6gx4Cwl9DGAKWClAD_iKpNC0bGHxGhcx">The New Boston tutorial to Git covering basic commands and workflow</a></li> </ul> <h3>Libraries</h3> <h3>C/Native</h3> <ul> <li><a href="https://libgit2.org/">libgit2</a>: a portable, pure C implementation of the Git core methods</li> </ul> <h3>JVM</h3> <ul> <li><a href="https://www.gitblit.com/">Gitblit</a>: an <a href="https://github.com/gitblit-org/gitblit">open-source</a>, pure Java stack for managing, viewing, and serving Git repositories</li> <li><a href="https://github.com/eclipse-jgit/jgit">jgit</a>: the Java implementation of git</li> </ul> <h3>Python</h3> <ul> <li><a href="https://github.com/gitpython-developers/GitPython">Git-Python</a>: Python library used to interact with Git repositories</li> </ul> <h3>CLR</h3> <ul> <li><a href="https://github.com/libgit2/libgit2sharp">libgit2sharp</a>: Git + .NET</li> </ul> <h3>ECMAscript</h3> <ul> <li><a href="https://isomorphic-git.org/">isomorphic-git</a> (<a href="https://github.com/isomorphic-git/isomorphic-git">Source</a>): A pure JavaScript implementation of git for node and browsers</li> </ul> <hr> <h1><a href="https://learnxinyminutes.com/docs/git/">Learn X in Y Minutes</a></h1> <p>Git is a distributed version control and source code management system.</p> <p>It does this through a series of snapshots of your project, and it works with those snapshots to provide you with functionality to version and manage your source code.</p> <h2>Git Architecture</h2> <h3>Repository</h3> <p>A set of files, directories, historical records, commits, and heads. Imagine it as a source code data structure, with the attribute that each source code "element" gives you access to its revision history, among other things.</p> <p>A git repository is comprised of the .git directory &amp; working tree.</p> <h4>.git Directory (component of repository)</h4> <p>The .git directory contains all the configurations, logs, branches, HEAD, and more. <a href="http://gitready.com/advanced/2009/03/23/whats-inside-your-git-directory.html">Detailed List.</a></p> <h4>Working Tree (component of repository)</h4> <p>This is basically the directories and files in your repository. It is often referred to as your working directory.</p> <h3>Index (component of .git dir)</h3> <p>The Index is the staging area in git. It's basically a layer that separates your working tree from the Git repository. This gives developers more power over what gets sent to the Git repository.</p> <h3>Commit</h3> <p>A git commit is a snapshot of a set of changes, or manipulations to your Working Tree. For example, if you added 5 files, and removed 2 others, these changes will be contained in a commit (or snapshot). This commit can then be pushed to other repositories, or not!</p> <h3>Branch</h3> <p>A branch is essentially a pointer to the last commit you made. As you go on committing, this pointer will automatically update to point to the latest commit.</p> <h3>Tag</h3> <p>A tag is a mark on specific point in history. Typically people use this functionality to mark release points (v1.0, and so on).</p> <h3>HEAD and head (component of .git dir)</h3> <p>HEAD is a pointer that points to the current branch. A repository only has 1 <em>active</em> HEAD. head is a pointer that points to any commit. A repository can have any number of heads.</p> <h3>Stages of Git</h3> <ul> <li>Modified - Changes have been made to a file but file has not been committed to Git Database yet</li> <li>Staged - Marks a modified file to go into your next commit snapshot</li> <li>Committed - Files have been committed to the Git Database</li> </ul> <h2>Commands</h2> <h3>init</h3> <p>Create an empty Git repository. The Git repository's settings, stored<br> information, and more is stored in a directory (a folder) named ".git".</p> <pre><code class="language-bash">$ git init </code></pre> <h3>config</h3> <p>To configure settings. Whether it be for the repository, the system itself,<br> or global configurations ( global config file is <code>~/.gitconfig</code> ).</p> <pre><code class="language-bash"># Print &amp; Set Some Basic Config Variables (Global) $ git config --global user.email "MyEmail@Zoho.com" $ git config --global user.name "My Name" </code></pre> <p><a href="http://git-scm.com/docs/git-config">Learn More About git config.</a></p> <h3>help</h3> <p>To give you quick access to an extremely detailed guide of each command. Or to<br> just give you a quick reminder of some semantics.</p> <pre><code class="language-bash"># Quickly check available commands $ git help # Check all available commands $ git help -a # Command specific help - user manual # git help &lt;command_here&gt; $ git help add $ git help commit $ git help init # or git &lt;command_here&gt; --help $ git add --help $ git commit --help $ git init --help </code></pre> <h3>ignore files</h3> <p>To intentionally untrack file(s) &amp; folder(s) from git. Typically meant for<br> private &amp; temp files which would otherwise be shared in the repository.</p> <pre><code class="language-bash">$ echo "temp/" &gt;&gt; .gitignore $ echo "private_key" &gt;&gt; .gitignore </code></pre> <h3>status</h3> <p>To show differences between the index file (basically your working copy/repo)<br> and the current HEAD commit.</p> <pre><code class="language-bash"># Will display the branch, untracked files, changes and other differences $ git status # To learn other "tid bits" about git status $ git help status </code></pre> <h3>add</h3> <p>To add files to the staging area/index. If you do not <code>git add</code> new files to<br> the staging area/index, they will not be included in commits!</p> <pre><code class="language-bash"># add a file in your current working directory $ git add HelloWorld.java # add a file in a nested dir $ git add /path/to/file/HelloWorld.c # Regular Expression support! $ git add ./*.java # You can also add everything in your working directory to the staging area. $ git add -A </code></pre> <p>This only adds a file to the staging area/index, it doesn't commit it to the<br> working directory/repo.</p> <h3>branch</h3> <p>Manage your branches. You can view, edit, create, delete branches using this<br> command.</p> <pre><code class="language-bash"># list existing branches &amp; remotes $ git branch -a # create a new branch $ git branch myNewBranch # delete a branch $ git branch -d myBranch # rename a branch # git branch -m &lt;oldname&gt; &lt;newname&gt; $ git branch -m myBranchName myNewBranchName # edit a branch's description $ git branch myBranchName --edit-description </code></pre> <h3>tag</h3> <p>Manage your tags</p> <pre><code class="language-bash"># List tags $ git tag # Create a annotated tag # The -m specifies a tagging message, which is stored with the tag. # If you don’t specify a message for an annotated tag, # Git launches your editor so you can type it in. $ git tag -a v2.0 -m 'my version 2.0' # Show info about tag # That shows the tagger information, the date the commit was tagged, # and the annotation message before showing the commit information. $ git show v2.0 # Push a single tag to remote $ git push origin v2.0 # Push a lot of tags to remote $ git push origin --tags </code></pre> <h3>checkout</h3> <p>Updates all files in the working tree to match the version in the index, or<br> specified tree.</p> <pre><code class="language-bash"># Checkout a repo - defaults to master branch $ git checkout # Checkout a specified branch $ git checkout branchName # Create a new branch &amp; switch to it # equivalent to "git branch &lt;name&gt;; git checkout &lt;name&gt;" $ git checkout -b newBranch </code></pre> <h3>clone</h3> <p>Clones, or copies, an existing repository into a new directory. It also adds<br> remote-tracking branches for each branch in the cloned repo, which allows you<br> to push to a remote branch.</p> <pre><code class="language-bash"># Clone learnxinyminutes-docs $ git clone https://github.com/adambard/learnxinyminutes-docs.git # shallow clone - faster cloning that pulls only latest snapshot $ git clone --depth 1 https://github.com/adambard/learnxinyminutes-docs.git # clone only a specific branch $ git clone -b master-cn https://github.com/adambard/learnxinyminutes-docs.git --single-branch </code></pre> <h3>commit</h3> <p>Stores the current contents of the index in a new "commit." This commit<br> contains the changes made and a message created by the user.</p> <pre><code class="language-bash"># commit with a message $ git commit -m "Added multiplyNumbers() function to HelloWorld.c" # signed commit with a message (user.signingkey must have been set # with your GPG key e.g. git config --global user.signingkey 5173AAD5) $ git commit -S -m "signed commit message" # automatically stage modified or deleted files, except new files, and then commit $ git commit -a -m "Modified foo.php and removed bar.php" # change last commit (this deletes previous commit with a fresh commit) $ git commit --amend -m "Correct message" </code></pre> <h3>diff</h3> <p>Shows differences between a file in the working directory, index and commits.</p> <pre><code class="language-bash"># Show difference between your working dir and the index $ git diff # Show differences between the index and the most recent commit. $ git diff --cached # Show differences between your working dir and the most recent commit $ git diff HEAD </code></pre> <h3>grep</h3> <p>Allows you to quickly search a repository.</p> <p>Optional Configurations:</p> <pre><code class="language-bash"># Thanks to Travis Jeffery for these # Set line numbers to be shown in grep search results $ git config --global grep.lineNumber true # Make search results more readable, including grouping $ git config --global alias.g "grep --break --heading --line-number" </code></pre> <pre><code class="language-bash"># Search for "variableName" in all java files $ git grep 'variableName' -- '*.java' # Search for a line that contains "arrayListName" and, "add" or "remove" $ git grep -e 'arrayListName' --and \( -e add -e remove \) </code></pre> <p>Google is your friend; for more examples<br> <a href="http://travisjeffery.com/b/2012/02/search-a-git-repo-like-a-ninja">Git Grep Ninja</a></p> <h3>log</h3> <p>Display commits to the repository.</p> <pre><code class="language-bash"># Show all commits $ git log # Show only commit message &amp; ref $ git log --oneline # Show merge commits only $ git log --merges # Show all commits represented by an ASCII graph $ git log --graph </code></pre> <h3>merge</h3> <p>"Merge" in changes from external commits into the current branch.</p> <pre><code class="language-bash"># Merge the specified branch into the current. $ git merge branchName # Always generate a merge commit when merging $ git merge --no-ff branchName </code></pre> <h3>mv</h3> <p>Rename or move a file</p> <pre><code class="language-bash"># Renaming a file $ git mv HelloWorld.c HelloNewWorld.c # Moving a file $ git mv HelloWorld.c ./new/path/HelloWorld.c # Force rename or move # "existingFile" already exists in the directory, will be overwritten $ git mv -f myFile existingFile </code></pre> <h3>pull</h3> <p>Pulls from a repository and merges it with another branch.</p> <pre><code class="language-bash"># Update your local repo, by merging in new changes # from the remote "origin" and "master" branch. # git pull &lt;remote&gt; &lt;branch&gt; $ git pull origin master # By default, git pull will update your current branch # by merging in new changes from its remote-tracking branch $ git pull # Merge in changes from remote branch and rebase # branch commits onto your local repo, like: "git fetch &lt;remote&gt; &lt;branch&gt;, git # rebase &lt;remote&gt;/&lt;branch&gt;" $ git pull origin master --rebase </code></pre> <h3>push</h3> <p>Push and merge changes from a branch to a remote &amp; branch.</p> <pre><code class="language-bash"># Push and merge changes from a local repo to a # remote named "origin" and "master" branch. # git push &lt;remote&gt; &lt;branch&gt; $ git push origin master # By default, git push will push and merge changes from # the current branch to its remote-tracking branch $ git push # To link up current local branch with a remote branch, add -u flag: $ git push -u origin master # Now, anytime you want to push from that same local branch, use shortcut: $ git push </code></pre> <h3>stash</h3> <p>Stashing takes the dirty state of your working directory and saves it on a<br> stack of unfinished changes that you can reapply at any time.</p> <p>Let's say you've been doing some work in your git repo, but you want to pull<br> from the remote. Since you have dirty (uncommitted) changes to some files, you<br> are not able to run <code>git pull</code>. Instead, you can run <code>git stash</code> to save your<br> changes onto a stack!</p> <pre><code class="language-bash">$ git stash Saved working directory and index state \ "WIP on master: 049d078 added the index file" HEAD is now at 049d078 added the index file (To restore them type "git stash apply") </code></pre> <p>Now you can pull!</p> <pre><code class="language-bash">git pull </code></pre> <p><code>...changes apply...</code></p> <p>Now check that everything is OK</p> <pre><code class="language-bash">$ git status # On branch master nothing to commit, working directory clean </code></pre> <p>You can see what "hunks" you've stashed so far using <code>git stash list</code>. Since the "hunks" are stored in a Last-In-First-Out stack, our most recent change will be at top.</p> <pre><code class="language-bash">$ git stash list stash@{0}: WIP on master: 049d078 added the index file stash@{1}: WIP on master: c264051 Revert "added file_size" stash@{2}: WIP on master: 21d80a5 added number to log </code></pre> <p>Now let's apply our dirty changes back by popping them off the stack.</p> <pre><code class="language-bash">$ git stash pop # On branch master # Changes not staged for commit: # (use "git add &lt;file&gt;..." to update what will be committed) # # modified: index.html # modified: lib/simplegit.rb # </code></pre> <p><code>git stash apply</code> does the same thing. Now you're ready to get back to work on your stuff! <a href="http://git-scm.com/book/en/v2/Git-Tools-Stashing">Additional Reading.</a></p> <h3>rebase (caution)</h3> <p>Take all changes that were committed on one branch, and replay them onto another branch. <em>Do not rebase commits that you have pushed to a public repo</em>.</p> <pre><code class="language-bash"># Rebase experimentBranch onto master # git rebase &lt;basebranch&gt; &lt;topicbranch&gt; $ git rebase master experimentBranch </code></pre> <p><a href="http://git-scm.com/book/en/Git-Branching-Rebasing">Additional Reading.</a></p> <h3>reset (caution)</h3> <p>Reset the current HEAD to the specified state. This allows you to undo merges, pulls, commits, adds, and more. It's a great command but also dangerous if you don't know what you are doing.</p> <pre><code class="language-bash"># Reset the staging area, to match the latest commit (leaves dir unchanged) $ git reset # Reset the staging area, to match the latest commit, and overwrite working dir $ git reset --hard # Moves the current branch tip to the specified commit (leaves dir unchanged) # all changes still exist in the directory. $ git reset 31f2bb1 # Moves the current branch tip backward to the specified commit # and makes the working dir match (deletes uncommitted changes and all commits # after the specified commit). $ git reset --hard 31f2bb1 </code></pre> <h3>reflog (caution)</h3> <p>Reflog will list most of the git commands you have done for a given time period, default 90 days.</p> <p>This give you the chance to reverse any git commands that have gone wrong (for instance, if a rebase has broken your application).</p> <p>You can do this:</p> <ol> <li><code>git reflog</code> to list all of the git commands for the rebase</li> </ol> <pre><code>38b323f HEAD@{0}: rebase -i (finish): returning to refs/heads/feature/add_git_reflog 38b323f HEAD@{1}: rebase -i (pick): Clarify inc/dec operators 4fff859 HEAD@{2}: rebase -i (pick): Update java.html.markdown 34ed963 HEAD@{3}: rebase -i (pick): [yaml/en] Add more resources (#1666) ed8ddf2 HEAD@{4}: rebase -i (pick): pythonstatcomp spanish translation (#1748) 2e6c386 HEAD@{5}: rebase -i (start): checkout 02fb96d </code></pre> <ol> <li>Select where to reset to, in our case its <code>2e6c386</code>, or <code>HEAD@{5}</code></li> <li>'git reset --hard HEAD@{5}' this will reset your repo to that head</li> <li>You can start the rebase again or leave it alone.</li> </ol> <p><a href="https://git-scm.com/docs/git-reflog">Additional Reading.</a></p> <h3>revert</h3> <p>Revert can be used to undo a commit. It should not be confused with reset which restores the state of a project to a previous point. Revert will add a new commit which is the inverse of the specified commit, thus reverting it.</p> <pre><code class="language-bash"># Revert a specified commit $ git revert &lt;commit&gt; </code></pre> <h3>rm</h3> <p>The opposite of git add, git rm removes files from the current working tree.</p> <pre><code class="language-bash"># remove HelloWorld.c $ git rm HelloWorld.c # Remove a file from a nested dir $ git rm /pather/to/the/file/HelloWorld.c </code></pre> <h3>Useful tidbits</h3> <p><a href="https://twitter.com/cecilphillip/status/1493958954434584580">"Changing your default branch name from master in Git is easy. Do it now"</a>: <code>git config --global init.defaultBranch main</code></p> <h1>Notes from the Git website</h1> <h2>Branching and Merging</h2> <p>"The Git feature that really makes it stand apart from nearly every other SCM out there is its branching model. Git allows and encourages you to have multiple local branches that can be entirely independent of each other. The creation, merging, and deletion of those lines of development takes seconds.</p> <p>This means that you can do things like:</p> <ul> <li> <p>Frictionless Context Switching. Create a branch to try out an idea, commit a few times, switch back to where you branched from, apply a patch, switch back to where you are experimenting, and merge it in.</p> </li> <li> <p>Role-Based Codelines. Have a branch that always contains only what goes to production, another that you merge work into for testing, and several smaller ones for day to day work.</p> </li> <li> <p>Feature Based Workflow. Create new branches for each new feature you're working on so you can seamlessly switch back and forth between them, then delete each branch when that feature gets merged into your main line.</p> </li> <li> <p>Disposable Experimentation. Create a branch to experiment in, realize it's not going to work, and just delete it - abandoning the work—with nobody else ever seeing it (even if you've pushed other branches in the meantime).<br> Branches</p> </li> </ul> <p>Notably, when you push to a remote repository, you do not have to push all of your branches. You can choose to share just one of your branches, a few of them, or all of them. This tends to free people to try new ideas without worrying about having to plan how and when they are going to merge it in or share it with others.</p> <p>There are ways to accomplish some of this with other systems, but the work involved is much more difficult and error-prone. Git makes this process incredibly easy and it changes the way most developers work when they learn it.</p> <h2>Distributed</h2> <p>One of the nicest features of any Distributed SCM, Git included, is that it's distributed. This means that instead of doing a "checkout" of the current tip of the source code, you do a "clone" of the entire repository.</p> <ul> <li> <p><em>Multiple Backups</em>: This means that even if you're using a centralized workflow, every user essentially has a full backup of the main server. Each of these copies could be pushed up to replace the main server in the event of a crash or corruption. In effect, there is no single point of failure with Git unless there is only a single copy of the repository.</p> </li> <li> <p><em>Any Workflow</em>: Because of Git's distributed nature and superb branching system, an almost endless number of workflows can be implemented with relative ease.</p> </li> <li> <p><em>Subversion-Style Workflow</em>: A centralized workflow is very common, especially from people transitioning from a centralized system. Git will not allow you to push if someone has pushed since the last time you fetched, so a centralized model where all developers push to the same server works just fine.</p> </li> <li> <p><em>Integration Manager Workflow</em>: Another common Git workflow involves an integration manager — a single person who commits to the 'blessed' repository. A number of developers then clone from that repository, push to their own independent repositories, and ask the integrator to pull in their changes. This is the type of development model often seen with open source or GitHub repositories.</p> </li> <li> <p><em>Dictator and Lieutenants Workflow</em>: For more massive projects, a development workflow like that of the Linux kernel is often effective. In this model, some people ('lieutenants') are in charge of a specific subsystem of the project and they merge in all changes related to that subsystem. Another integrator (the 'dictator') can pull changes from only his/her lieutenants and then push to the 'blessed' repository that everyone then clones from again.</p> </li> </ul> <h2>Staging Area</h2> <p>Unlike the other systems, Git has something called the "staging area" or "index". This is an intermediate area where commits can be formatted and reviewed before completing the commit.</p> <p>One thing that sets Git apart from other tools is that it's possible to quickly stage some of your files and commit them without committing all of the other modified files in your working directory or having to list them on the command line during the commit.</p> <p>This allows you to stage only portions of a modified file. Gone are the days of making two logically unrelated modifications to a file before you realized that you forgot to commit one of them. Now you can just stage the change you need for the current commit and stage the other change for the next commit. This feature scales up to as many different changes to your file as needed.</p> <p>Of course, Git also makes it easy to ignore this feature if you don't want that kind of control — just add a '-a' to your commit command in order to add all changes to all files to the staging area.</p> <hr> <h3>Using Git to do keyword-expansion within source files</h3> <p>In general, this appears to be best handled with <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">Git Hooks</a>. <a href="https://www.git-scm.com/docs/githooks">Official githook doc page</a>.</p> <p><a href="https://stackoverflow.com/questions/44948352/git-add-author-date-create-modified-to-a-file?noredirect=1&amp;lq=1">StackOverflow: Add author, date, create/modified to a file</a> answer directed to <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes#_keyword_expansion">keyword expansion</a>:</p> <blockquote> <p>It turns out that you can write your own filters for doing substitutions in files on commit/checkout. These are called “clean” and “smudge” filters. In the .gitattributes file, you can set a filter for particular paths and then set up scripts that will process files just before they’re checked out (“smudge”, see <a href="https://git-scm.com/book/en/v2/ch00/filters_a">The “smudge” filter is run on checkout</a>.) and just before they’re staged (“clean”, see <a href="https://git-scm.com/book/en/v2/ch00/filters_b">The “clean” filter is run when files are staged</a>.). These filters can be set to do all sorts of fun things.</p> <p>The original commit message for this feature gives a simple example of running all your C source code through the indent program before committing. You can set it up by setting the filter attribute in your .gitattributes file to filter *.c files with the “indent” filter:</p> <pre><code>*.c filter=indent </code></pre> <p>Then, tell Git what the “indent” filter does on smudge and clean:</p> <pre><code>$ git config --global filter.indent.clean indent $ git config --global filter.indent.smudge cat </code></pre> <p>In this case, when you commit files that match *.c, Git will run them through the indent program before it stages them and then run them through the cat program before it checks them back out onto disk. The cat program does essentially nothing: it spits out the same data that it comes in. This combination effectively filters all C source code files through indent before committing.</p> <p>Another interesting example gets $Date$ keyword expansion, RCS style. To do this properly, you need a small script that takes a filename, figures out the last commit date for this project, and inserts the date into the file. Here is a small Ruby script that does that:</p> <pre><code>#! /usr/bin/env ruby data = STDIN.read last_date = `git log --pretty=format:"%ad" -1` puts data.gsub('$Date$', '$Date: ' + last_date.to_s + '$') </code></pre> <p>All the script does is get the latest commit date from the git log command, stick that into any $Date$ strings it sees in stdin, and print the results – it should be simple to do in whatever language you’re most comfortable in. You can name this file expand_date and put it in your path. Now, you need to set up a filter in Git (call it dater) and tell it to use your expand_date filter to smudge the files on checkout. You’ll use a Perl expression to clean that up on commit:</p> </blockquote> <pre><code>$ git config filter.dater.smudge expand_date $ git config filter.dater.clean 'perl -pe "s/\\\$Date[^\\\$]*\\\$/\\\$Date\\\$/"' </code></pre> <blockquote> <p>This Perl snippet strips out anything it sees in a $Date$ string, to get back to where you started. Now that your filter is ready, you can test it by setting up a Git attribute for that file that engages the new filter and creating a file with your $Date$ keyword:</p> </blockquote> <pre><code>date*.txt filter=dater $ echo '# $Date$' &gt; date_test.txt </code></pre> <blockquote> <p>If you commit those changes and check out the file again, you see the keyword properly substituted:</p> </blockquote> <pre><code>$ git add date_test.txt .gitattributes $ git commit -m "Test date expansion in Git" $ rm date_test.txt $ git checkout date_test.txt $ cat date_test.txt # $Date: Tue Apr 21 07:26:52 2009 -0700$ </code></pre> <blockquote> <p>You can see how powerful this technique can be for customized applications. You have to be careful, though, because the .gitattributes file is committed and passed around with the project, but the driver (in this case, dater) isn’t, so it won’t work everywhere. When you design these filters, they should be able to fail gracefully and have the project still work properly.</p> </blockquote> https://tedneward.github.io/Research/tools/gitnexus/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gitnexus/index.html <p><a href="https://gitnexus.vercel.app/">Website</a> | <a href="https://github.com/abhigyanpatwari/GitNexus">Source</a></p> <p><strong>Building nervous system for agent context.</strong></p> <p>Indexes any codebase into a knowledge graph — every dependency, call chain, cluster, and execution flow — then exposes it through smart tools so AI agents never miss code.</p> <p><a href="https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72">https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72</a></p> <blockquote> <p><em>Like DeepWiki, but deeper.</em> DeepWiki helps you <em>understand</em> code. GitNexus lets you <em>analyze</em> it — because a knowledge graph tracks every relationship, not just descriptions.</p> </blockquote> <p><strong>TL;DR:</strong> The <strong>Web UI</strong> is a quick way to chat with any repo. The <strong>CLI + MCP</strong> is how you make your AI agent actually reliable — it gives Cursor, Claude Code, and friends a deep architectural view of your codebase so they stop missing dependencies, breaking call chains, and shipping blind edits. Even smaller models get full architectural clarity, making it compete with goliath models.</p> <hr> <h2>Star History</h2> <p><a href="https://www.star-history.com/#abhigyanpatwari/GitNexus&amp;type=date&amp;legend=top-left"><img src="https://api.star-history.com/svg?repos=abhigyanpatwari/GitNexus&amp;type=date&amp;legend=top-left" alt="Star History Chart"></a></p> <h2>Two Ways to Use GitNexus</h2> <table> <thead> <tr> <th> </th> <th> <strong>CLI + MCP</strong> </th> <th> <strong>Web UI</strong> </th> </tr> </thead> <tbody> <tr> <td> <strong>What</strong> </td> <td> Index repos locally, connect AI agents via MCP </td> <td> Visual graph explorer + AI chat in browser </td> </tr> <tr> <td> <strong>For</strong> </td> <td> Daily development with Cursor, Claude Code, Windsurf, OpenCode </td> <td> Quick exploration, demos, one-off analysis </td> </tr> <tr> <td> <strong>Scale</strong> </td> <td> Full repos, any size </td> <td> Limited by browser memory (~5k files), or unlimited via backend mode </td> </tr> <tr> <td> <strong>Install</strong> </td> <td> <code>npm install -g gitnexus</code> </td> <td> No install —<a href="https://gitnexus.vercel.app">gitnexus.vercel.app</a> </td> </tr> <tr> <td> <strong>Storage</strong> </td> <td> KuzuDB native (fast, persistent) </td> <td> KuzuDB WASM (in-memory, per session) </td> </tr> <tr> <td> <strong>Parsing</strong> </td> <td> Tree-sitter native bindings </td> <td> Tree-sitter WASM </td> </tr> <tr> <td> <strong>Privacy</strong> </td> <td> Everything local, no network </td> <td> Everything in-browser, no server </td> </tr> </tbody> </table> <blockquote> <p><strong>Bridge mode:</strong> <code>gitnexus serve</code> connects the two — the web UI auto-detects the local server and can browse all your CLI-indexed repos without re-uploading or re-indexing.</p> </blockquote> <hr> <h2>CLI + MCP (recommended)</h2> <p>The CLI indexes your repository and runs an MCP server that gives AI agents deep codebase awareness.</p> <h3>Quick Start</h3> <pre><code class="language-bash"># Index your repo (run from repo root) npx gitnexus analyze </code></pre> <p>That's it. This indexes the codebase, installs agent skills, registers Claude Code hooks, and creates <code>AGENTS.md</code> / <code>CLAUDE.md</code> context files — all in one command.</p> <p>To configure MCP for your editor, run <code>npx gitnexus setup</code> once — or set it up manually below.</p> <h3>MCP Setup</h3> <p><code>gitnexus setup</code> auto-detects your editors and writes the correct global MCP config. You only need to run it once.</p> <h3>Editor Support</h3> <table> <thead> <tr> <th> Editor </th> <th> MCP </th> <th> Skills </th> <th> Hooks (auto-augment) </th> <th> Support </th> </tr> </thead> <tbody> <tr> <td> <strong>Claude Code</strong> </td> <td> Yes </td> <td> Yes </td> <td> Yes (PreToolUse + PostToolUse) </td> <td> <strong>Full</strong> </td> </tr> <tr> <td> <strong>Cursor</strong> </td> <td> Yes </td> <td> Yes </td> <td> — </td> <td> MCP + Skills </td> </tr> <tr> <td> <strong>Windsurf</strong> </td> <td> Yes </td> <td> — </td> <td> — </td> <td> MCP </td> </tr> <tr> <td> <strong>OpenCode</strong> </td> <td> Yes </td> <td> Yes </td> <td> — </td> <td> MCP + Skills </td> </tr> </tbody> </table> <blockquote> <p><strong>Claude Code</strong> gets the deepest integration: MCP tools + agent skills + PreToolUse hooks that enrich searches with graph context + PostToolUse hooks that auto-reindex after commits.</p> </blockquote> <h3>Community Integrations</h3> <table> <thead> <tr> <th> Agent </th> <th> Install </th> <th> Source </th> </tr> </thead> <tbody> <tr> <td> <a href="https://pi.dev">pi</a> </td> <td> <code>pi install npm:pi-gitnexus</code> </td> <td> <a href="https://github.com/tintinweb/pi-gitnexus">pi-gitnexus</a> </td> </tr> </tbody> </table> <p>If you prefer manual configuration:</p> <p><strong>Claude Code</strong> (full support — MCP + skills + hooks):</p> <pre><code class="language-bash">claude mcp add gitnexus -- npx -y gitnexus@latest mcp </code></pre> <p><strong>Cursor</strong> (<code>~/.cursor/mcp.json</code> — global, works for all projects):</p> <pre><code class="language-json">{ "mcpServers": { "gitnexus": { "command": "npx", "args": ["-y", "gitnexus@latest", "mcp"] } } } </code></pre> <p><strong>OpenCode</strong> (<code>~/.config/opencode/config.json</code>):</p> <pre><code class="language-json">{ "mcp": { "gitnexus": { "command": "npx", "args": ["-y", "gitnexus@latest", "mcp"] } } } </code></pre> <h3>CLI Commands</h3> <pre><code class="language-bash">gitnexus setup # Configure MCP for your editors (one-time) gitnexus analyze [path] # Index a repository (or update stale index) gitnexus analyze --force # Force full re-index gitnexus analyze --skip-embeddings # Skip embedding generation (faster) gitnexus mcp # Start MCP server (stdio) — serves all indexed repos gitnexus serve # Start local HTTP server (multi-repo) for web UI connection gitnexus list # List all indexed repositories gitnexus status # Show index status for current repo gitnexus clean # Delete index for current repo gitnexus clean --all --force # Delete all indexes gitnexus wiki [path] # Generate repository wiki from knowledge graph gitnexus wiki --model &lt;model&gt; # Wiki with custom LLM model (default: gpt-4o-mini) gitnexus wiki --base-url &lt;url&gt; # Wiki with custom LLM API base URL </code></pre> <h3>What Your AI Agent Gets</h3> <p><strong>7 tools</strong> exposed via MCP:</p> <table> <thead> <tr> <th> Tool </th> <th> What It Does </th> <th> <code>repo</code> Param </th> </tr> </thead> <tbody> <tr> <td> <code>list_repos</code> </td> <td> Discover all indexed repositories </td> <td> — </td> </tr> <tr> <td> <code>query</code> </td> <td> Process-grouped hybrid search (BM25 + semantic + RRF) </td> <td> Optional </td> </tr> <tr> <td> <code>context</code> </td> <td> 360-degree symbol view — categorized refs, process participation </td> <td> Optional </td> </tr> <tr> <td> <code>impact</code> </td> <td> Blast radius analysis with depth grouping and confidence </td> <td> Optional </td> </tr> <tr> <td> <code>detect_changes</code> </td> <td> Git-diff impact — maps changed lines to affected processes </td> <td> Optional </td> </tr> <tr> <td> <code>rename</code> </td> <td> Multi-file coordinated rename with graph + text search </td> <td> Optional </td> </tr> <tr> <td> <code>cypher</code> </td> <td> Raw Cypher graph queries </td> <td> Optional </td> </tr> </tbody> </table> <blockquote> <p>When only one repo is indexed, the <code>repo</code> parameter is optional. With multiple repos, specify which one: <code>query({query: "auth", repo: "my-app"})</code>.</p> </blockquote> <p><strong>Resources</strong> for instant context:</p> <table> <thead> <tr> <th> Resource </th> <th> Purpose </th> </tr> </thead> <tbody> <tr> <td> <code>gitnexus://repos</code> </td> <td> List all indexed repositories (read this first) </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/context</code> </td> <td> Codebase stats, staleness check, and available tools </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/clusters</code> </td> <td> All functional clusters with cohesion scores </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/cluster/{name}</code> </td> <td> Cluster members and details </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/processes</code> </td> <td> All execution flows </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/process/{name}</code> </td> <td> Full process trace with steps </td> </tr> <tr> <td> <code>gitnexus://repo/{name}/schema</code> </td> <td> Graph schema for Cypher queries </td> </tr> </tbody> </table> <p><strong>2 MCP prompts</strong> for guided workflows:</p> <table> <thead> <tr> <th> Prompt </th> <th> What It Does </th> </tr> </thead> <tbody> <tr> <td> <code>detect_impact</code> </td> <td> Pre-commit change analysis — scope, affected processes, risk level </td> </tr> <tr> <td> <code>generate_map</code> </td> <td> Architecture documentation from the knowledge graph with mermaid diagrams </td> </tr> </tbody> </table> <p><strong>4 agent skills</strong> installed to <code>.claude/skills/</code> automatically:</p> <ul> <li><strong>Exploring</strong> — Navigate unfamiliar code using the knowledge graph</li> <li><strong>Debugging</strong> — Trace bugs through call chains</li> <li><strong>Impact Analysis</strong> — Analyze blast radius before changes</li> <li><strong>Refactoring</strong> — Plan safe refactors using dependency mapping</li> </ul> <hr> <h2>Multi-Repo MCP Architecture</h2> <p>GitNexus uses a <strong>global registry</strong> so one MCP server can serve multiple indexed repos. No per-project MCP config needed — set it up once and it works everywhere.</p> <pre><code class="language-mermaid">flowchart TD subgraph CLI [CLI Commands] Setup["gitnexus setup"] Analyze["gitnexus analyze"] Clean["gitnexus clean"] List["gitnexus list"] end subgraph Registry ["~/.gitnexus/"] RegFile["registry.json"] end subgraph Repos [Project Repos] RepoA[".gitnexus/ in repo A"] RepoB[".gitnexus/ in repo B"] end subgraph MCP [MCP Server] Server["server.ts"] Backend["LocalBackend"] Pool["Connection Pool"] ConnA["KuzuDB conn A"] ConnB["KuzuDB conn B"] end Setup --&gt;|"writes global MCP config"| CursorConfig["~/.cursor/mcp.json"] Analyze --&gt;|"registers repo"| RegFile Analyze --&gt;|"stores index"| RepoA Clean --&gt;|"unregisters repo"| RegFile List --&gt;|"reads"| RegFile Server --&gt;|"reads registry"| RegFile Server --&gt; Backend Backend --&gt; Pool Pool --&gt;|"lazy open"| ConnA Pool --&gt;|"lazy open"| ConnB ConnA --&gt;|"queries"| RepoA ConnB --&gt;|"queries"| RepoB </code></pre> <p><strong>How it works:</strong> Each <code>gitnexus analyze</code> stores the index in <code>.gitnexus/</code> inside the repo (portable, gitignored) and registers a pointer in <code>~/.gitnexus/registry.json</code>. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. KuzuDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). If only one repo is indexed, the <code>repo</code> parameter is optional on all tools — agents don't need to change anything.</p> <hr> <h2>Web UI (browser-based)</h2> <p>A fully client-side graph explorer and AI chat. No server, no install — your code never leaves the browser.</p> <p><strong>Try it now:</strong> <a href="https://gitnexus.vercel.app">gitnexus.vercel.app</a> — drag &amp; drop a ZIP and start exploring.</p> <p><img width="2550" height="1343" alt="gitnexus_img" src="https://github.com/user-attachments/assets/cc5d637d-e0e5-48e6-93ff-5bcfdb929285"></p> <p>Or run locally:</p> <pre><code class="language-bash">git clone https://github.com/abhigyanpatwari/gitnexus.git cd gitnexus/gitnexus-web npm install npm run dev </code></pre> <p>The web UI uses the same indexing pipeline as the CLI but runs entirely in WebAssembly (Tree-sitter WASM, KuzuDB WASM, in-browser embeddings). It's great for quick exploration but limited by browser memory for larger repos.</p> <p><strong>Local Backend Mode:</strong> Run <code>gitnexus serve</code> and open the web UI locally — it auto-detects the server and shows all your indexed repos, with full AI chat support. No need to re-upload or re-index. The agent's tools (Cypher queries, search, code navigation) route through the backend HTTP API automatically.</p> <hr> <h2>The Problem GitNexus Solves</h2> <p>Tools like <strong>Cursor</strong>, <strong>Claude Code</strong>, <strong>Cline</strong>, <strong>Roo Code</strong>, and <strong>Windsurf</strong> are powerful — but they don't truly know your codebase structure.</p> <p><strong>What happens:</strong></p> <ol> <li>AI edits <code>UserService.validate()</code></li> <li>Doesn't know 47 functions depend on its return type</li> <li><strong>Breaking changes ship</strong></li> </ol> <h3>Traditional Graph RAG vs GitNexus</h3> <p>Traditional approaches give the LLM raw graph edges and hope it explores enough. GitNexus <strong>precomputes structure at index time</strong> — clustering, tracing, scoring — so tools return complete context in one call:</p> <pre><code class="language-mermaid">flowchart TB subgraph Traditional["Traditional Graph RAG"] direction TB U1["User: What depends on UserService?"] U1 --&gt; LLM1["LLM receives raw graph"] LLM1 --&gt; Q1["Query 1: Find callers"] Q1 --&gt; Q2["Query 2: What files?"] Q2 --&gt; Q3["Query 3: Filter tests?"] Q3 --&gt; Q4["Query 4: High-risk?"] Q4 --&gt; OUT1["Answer after 4+ queries"] end subgraph GN["GitNexus Smart Tools"] direction TB U2["User: What depends on UserService?"] U2 --&gt; TOOL["impact UserService upstream"] TOOL --&gt; PRECOMP["Pre-structured response: 8 callers, 3 clusters, all 90%+ confidence"] PRECOMP --&gt; OUT2["Complete answer, 1 query"] end </code></pre> <p><strong>Core innovation: Precomputed Relational Intelligence</strong></p> <ul> <li><strong>Reliability</strong> — LLM can't miss context, it's already in the tool response</li> <li><strong>Token efficiency</strong> — No 10-query chains to understand one function</li> <li><strong>Model democratization</strong> — Smaller LLMs work because tools do the heavy lifting</li> </ul> <hr> <h2>How It Works</h2> <p>GitNexus builds a complete knowledge graph of your codebase through a multi-phase indexing pipeline:</p> <ol> <li><strong>Structure</strong> — Walks the file tree and maps folder/file relationships</li> <li><strong>Parsing</strong> — Extracts functions, classes, methods, and interfaces using Tree-sitter ASTs</li> <li><strong>Resolution</strong> — Resolves imports and function calls across files with language-aware logic</li> <li><strong>Clustering</strong> — Groups related symbols into functional communities</li> <li><strong>Processes</strong> — Traces execution flows from entry points through call chains</li> <li><strong>Search</strong> — Builds hybrid search indexes for fast retrieval</li> </ol> <h3>Supported Languages</h3> <p>TypeScript, JavaScript, Python, Java, Kotlin, C, C++, C#, Go, Rust, PHP, Swift</p> <hr> <h2>Tool Examples</h2> <h3>Impact Analysis</h3> <pre><code>impact({target: "UserService", direction: "upstream", minConfidence: 0.8}) TARGET: Class UserService (src/services/user.ts) UPSTREAM (what depends on this): Depth 1 (WILL BREAK): handleLogin [CALLS 90%] -&gt; src/api/auth.ts:45 handleRegister [CALLS 90%] -&gt; src/api/auth.ts:78 UserController [CALLS 85%] -&gt; src/controllers/user.ts:12 Depth 2 (LIKELY AFFECTED): authRouter [IMPORTS] -&gt; src/routes/auth.ts </code></pre> <p>Options: <code>maxDepth</code>, <code>minConfidence</code>, <code>relationTypes</code> (<code>CALLS</code>, <code>IMPORTS</code>, <code>EXTENDS</code>, <code>IMPLEMENTS</code>), <code>includeTests</code></p> <h3>Process-Grouped Search</h3> <pre><code>query({query: "authentication middleware"}) processes: - summary: "LoginFlow" priority: 0.042 symbol_count: 4 process_type: cross_community step_count: 7 process_symbols: - name: validateUser type: Function filePath: src/auth/validate.ts process_id: proc_login step_index: 2 definitions: - name: AuthConfig type: Interface filePath: src/types/auth.ts </code></pre> <h3>Context (360-degree Symbol View)</h3> <pre><code>context({name: "validateUser"}) symbol: uid: "Function:validateUser" kind: Function filePath: src/auth/validate.ts startLine: 15 incoming: calls: [handleLogin, handleRegister, UserController] imports: [authRouter] outgoing: calls: [checkPassword, createSession] processes: - name: LoginFlow (step 2/7) - name: RegistrationFlow (step 3/5) </code></pre> <h3>Detect Changes (Pre-Commit)</h3> <pre><code>detect_changes({scope: "all"}) summary: changed_count: 12 affected_count: 3 changed_files: 4 risk_level: medium changed_symbols: [validateUser, AuthService, ...] affected_processes: [LoginFlow, RegistrationFlow, ...] </code></pre> <h3>Rename (Multi-File)</h3> <pre><code>rename({symbol_name: "validateUser", new_name: "verifyUser", dry_run: true}) status: success files_affected: 5 total_edits: 8 graph_edits: 6 (high confidence) text_search_edits: 2 (review carefully) changes: [...] </code></pre> <h3>Cypher Queries</h3> <pre><code class="language-cypher">-- Find what calls auth functions with high confidence MATCH (c:Community {heuristicLabel: 'Authentication'})&lt;-[:CodeRelation {type: 'MEMBER_OF'}]-(fn) MATCH (caller)-[r:CodeRelation {type: 'CALLS'}]-&gt;(fn) WHERE r.confidence &gt; 0.8 RETURN caller.name, fn.name, r.confidence ORDER BY r.confidence DESC </code></pre> <hr> <h2>Wiki Generation</h2> <p>Generate LLM-powered documentation from your knowledge graph:</p> <pre><code class="language-bash"># Requires an LLM API key (OPENAI_API_KEY, etc.) gitnexus wiki # Use a custom model or provider gitnexus wiki --model gpt-4o gitnexus wiki --base-url https://api.anthropic.com/v1 # Force full regeneration gitnexus wiki --force </code></pre> <p>The wiki generator reads the indexed graph structure, groups files into modules via LLM, generates per-module documentation pages, and creates an overview page — all with cross-references to the knowledge graph.</p> <hr> <h2>Tech Stack</h2> <table> <thead> <tr> <th> Layer </th> <th> CLI </th> <th> Web </th> </tr> </thead> <tbody> <tr> <td> <strong>Runtime</strong> </td> <td> Node.js (native) </td> <td> Browser (WASM) </td> </tr> <tr> <td> <strong>Parsing</strong> </td> <td> Tree-sitter native bindings </td> <td> Tree-sitter WASM </td> </tr> <tr> <td> <strong>Database</strong> </td> <td> KuzuDB native </td> <td> KuzuDB WASM </td> </tr> <tr> <td> <strong>Embeddings</strong> </td> <td> HuggingFace transformers.js (GPU/CPU) </td> <td> transformers.js (WebGPU/WASM) </td> </tr> <tr> <td> <strong>Search</strong> </td> <td> BM25 + semantic + RRF </td> <td> BM25 + semantic + RRF </td> </tr> <tr> <td> <strong>Agent Interface</strong> </td> <td> MCP (stdio) </td> <td> LangChain ReAct agent </td> </tr> <tr> <td> <strong>Visualization</strong> </td> <td> — </td> <td> Sigma.js + Graphology (WebGL) </td> </tr> <tr> <td> <strong>Frontend</strong> </td> <td> — </td> <td> React 18, TypeScript, Vite, Tailwind v4 </td> </tr> <tr> <td> <strong>Clustering</strong> </td> <td> Graphology </td> <td> Graphology </td> </tr> <tr> <td> <strong>Concurrency</strong> </td> <td> Worker threads + async </td> <td> Web Workers + Comlink </td> </tr> </tbody> </table> <hr> <h2>Security &amp; Privacy</h2> <ul> <li><strong>CLI</strong>: Everything runs locally on your machine. No network calls. Index stored in <code>.gitnexus/</code> (gitignored). Global registry at <code>~/.gitnexus/</code> stores only paths and metadata.</li> <li><strong>Web</strong>: Everything runs in your browser. No code uploaded to any server. API keys stored in localStorage only.</li> <li>Open source — audit the code yourself.</li> </ul> https://tedneward.github.io/Research/tools/glush/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/glush/index.html <p><a href="https://github.com/judofyr/glush">Source</a></p> <h2>Reading</h2> <ul> <li><a href="https://www.sanity.io/blog/why-we-wrote-yet-another-parser-compiler">"Introducing Glush: a robust, human readable, top-down parser compiler"</a></li> </ul> https://tedneward.github.io/Research/tools/get-shit-done/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/get-shit-done/index.html <p><a href="https://github.com/gsd-build/get-shit-done">Source</a></p> <p>A light-weight and powerful meta-prompting, context engineering and spec-driven development system for Claude Code, OpenCode, Gemini CLI, Kilo, Codex, Copilot, Cursor, Windsurf, Antigravity, Augment, Trae, Qwen Code, Cline, and CodeBuddy.</p> <p>Solves context rot — the quality degradation that happens as Claude fills its context window.</p> <p><code>npx get-shit-done-cc@latest</code></p> https://tedneward.github.io/Research/tools/gitbook/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gitbook/index.html <p><a href="https://www.gitbook.com/">Website</a> | Commercial | <a href="https://github.com/GitBookIO">Source?</a> | <a href="https://gitbook.com/docs">Docs</a></p> https://tedneward.github.io/Research/tools/glim/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/glim/index.html <p><a href="https://github.com/thias/glim">Source</a> | <a href="https://glee.thias.es/GLIM">Docs</a></p> <p>Advantages over extracting files or using special Live USB creation tools :</p> <ul> <li>A single USB memory can hold all Live environments (the limit is its size)</li> <li>ISO images stay available to burn real CDs or DVDs</li> <li>ISO images are quick to manipulate (vs. hundreds+ files)</li> </ul> <p>Disadvantages :</p> <ul> <li>There is no persistence overlay for distributions which normally support it</li> <li>Setting up isn't as easy as a simple cat from the ISO image to a block device</li> </ul> <p>My experience has been that the safest filesystem to use is FAT32 (surprisingly!), though it will mean that ISO images greater than 4GB won't be supported. Other filesystems supported by GRUB2 also work, such as ext3/ext4, NTFS and exFAT, but the boot of the distributions must also support it, which isn't the case for many with NTFS (Ubuntu does, Fedora doesn't) and exFAT (Ubuntu doesn't, Fedora does). So FAT32 stays the safe bet; make sure your device is partitioned with MBR (not GPT) for legacy BIOS and EFI hybrid support for peak compatibility.</p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/glim-ventoy-alternative-linux-enthusiasts-should-use-instead/">Glim is the Ventoy alternative Linux enthusiasts should use instead</a></li> </ul> https://tedneward.github.io/Research/tools/gnu/awk/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gnu/awk/index.html <p><a href="https://www.gnu.org/software/gawk/manual/gawk.html">Website: GNU awk</a> | <a href="">Source</a> | <a href="https://www-zeuthen.desy.de/dv/documentation/unixguide/infohtml/gawk/gawk.html">GNU Awk User's Guide</a></p> <ul> <li><a href="https://gist.github.com/Rafe/3102414">AWK cheatsheet</a></li> <li><a href="https://en.wikibooks.org/wiki/An_Awk_Primer">An Awk Primer</a> - Wikibooks</li> <li><a href="https://www.grymoire.com/Unix/Awk.html">Awk</a> - Bruce Barnett</li> <li><a href="https://learnbyexample.github.io/learn_gnuawk/">GNU awk</a> - Sundeep Agarwal</li> <li><a href="https://awka.sourceforge.net/index.html">AWKA</a> - Awka is an open-source implementation. Awka is not an interpreter like Gawk, Mawk or Nawk, but instead it converts the program to ANSI-C, then compiles this using gcc or a native C compiler to create a binary executable. This means you must have an ANSI C compiler present on your system for Awka to work.</li> </ul> <h3><a href="https://github.com/adambard/learnxinyminutes-docs/blob/master/awk.html.markdown">Learn X in Y Minutes</a> short intro</h3> <pre><code class="language-awk">#!/usr/bin/awk -f # Comments are like this # AWK programs consist of a collection of patterns and actions. pattern1 { action; } # just like lex pattern2 { action; } # There is an implied loop and AWK automatically reads and parses each # record of each file supplied. Each record is split by the FS delimiter, # which defaults to white-space (multiple spaces,tabs count as one) # You can assign FS either on the command line (-F C) or in your BEGIN # pattern # One of the special patterns is BEGIN. The BEGIN pattern is true # BEFORE any of the files are read. The END pattern is true after # an End-of-file from the last file (or standard-in if no files specified) # There is also an output field separator (OFS) that you can assign, which # defaults to a single space BEGIN { # BEGIN will run at the beginning of the program. It's where you put all # the preliminary set-up code, before you process any text files. If you # have no text files, then think of BEGIN as the main entry point. # Variables are global. Just set them or use them, no need to declare.. count = 0; # Operators just like in C and friends a = count + 1; b = count - 1; c = count * 1; d = count / 1; # integer division e = count % 1; # modulus f = count ^ 1; # exponentiation a += 1; b -= 1; c *= 1; d /= 1; e %= 1; f ^= 1; # Incrementing and decrementing by one a++; b--; # As a prefix operator, it returns the incremented value ++a; --b; # Notice, also, no punctuation such as semicolons to terminate statements # Control statements if (count == 0) print "Starting with count of 0"; else print "Huh?"; # Or you could use the ternary operator print (count == 0) ? "Starting with count of 0" : "Huh?"; # Blocks consisting of multiple lines use braces while (a &lt; 10) { print "String concatenation is done" " with a series" " of" " space-separated strings"; print a; a++; } for (i = 0; i &lt; 10; i++) print "Good ol' for loop"; # As for comparisons, they're the standards: # a &lt; b # Less than # a &lt;= b # Less than or equal # a != b # Not equal # a == b # Equal # a &gt; b # Greater than # a &gt;= b # Greater than or equal # Logical operators as well # a &amp;&amp; b # AND # a || b # OR # In addition, there's the super useful regular expression match if ("foo" ~ "^fo+$") print "Fooey!"; if ("boo" !~ "^fo+$") print "Boo!"; # Arrays arr[0] = "foo"; arr[1] = "bar"; # You can also initialize an array with the built-in function split() n = split("foo:bar:baz", arr, ":"); # You also have associative arrays (actually, they're all associative arrays) assoc["foo"] = "bar"; assoc["bar"] = "baz"; # And multi-dimensional arrays, with some limitations I won't mention here multidim[0,0] = "foo"; multidim[0,1] = "bar"; multidim[1,0] = "baz"; multidim[1,1] = "boo"; # You can test for array membership if ("foo" in assoc) print "Fooey!"; # You can also use the 'in' operator to traverse the keys of an array for (key in assoc) print assoc[key]; # The command line is in a special array called ARGV for (argnum in ARGV) print ARGV[argnum]; # You can remove elements of an array # This is particularly useful to prevent AWK from assuming the arguments # are files for it to process delete ARGV[1]; # The number of command line arguments is in a variable called ARGC print ARGC; # AWK has several built-in functions. They fall into three categories. I'll # demonstrate each of them in their own functions, defined later. return_value = arithmetic_functions(a, b, c); string_functions(); io_functions(); } # Here's how you define a function function arithmetic_functions(a, b, c, d) { # Probably the most annoying part of AWK is that there are no local # variables. Everything is global. For short scripts, this is fine, even # useful, but for longer scripts, this can be a problem. # There is a work-around (ahem, hack). Function arguments are local to the # function, and AWK allows you to define more function arguments than it # needs. So just stick local variable in the function declaration, like I # did above. As a convention, stick in some extra whitespace to distinguish # between actual function parameters and local variables. In this example, # a, b, and c are actual parameters, while d is merely a local variable. # Now, to demonstrate the arithmetic functions # Most AWK implementations have some standard trig functions localvar = sin(a); localvar = cos(a); localvar = atan2(b, a); # arc tangent of b / a # And logarithmic stuff localvar = exp(a); localvar = log(a); # Square root localvar = sqrt(a); # Truncate floating point to integer localvar = int(5.34); # localvar =&gt; 5 # Random numbers srand(); # Supply a seed as an argument. By default, it uses the time of day localvar = rand(); # Random number between 0 and 1. # Here's how to return a value return localvar; } function string_functions( localvar, arr) { # AWK, being a string-processing language, has several string-related # functions, many of which rely heavily on regular expressions. # Search and replace, first instance (sub) or all instances (gsub) # Both return number of matches replaced localvar = "fooooobar"; sub("fo+", "Meet me at the ", localvar); # localvar =&gt; "Meet me at the bar" gsub("e+", ".", localvar); # localvar =&gt; "m..t m. at th. bar" # Search for a string that matches a regular expression # index() does the same thing, but doesn't allow a regular expression match(localvar, "t"); # =&gt; 4, since the 't' is the fourth character # Split on a delimiter n = split("foo-bar-baz", arr, "-"); # a[1] = "foo"; a[2] = "bar"; a[3] = "baz"; n = 3 # Other useful stuff sprintf("%s %d %d %d", "Testing", 1, 2, 3); # =&gt; "Testing 1 2 3" substr("foobar", 2, 3); # =&gt; "oob" substr("foobar", 4); # =&gt; "bar" length("foo"); # =&gt; 3 tolower("FOO"); # =&gt; "foo" toupper("foo"); # =&gt; "FOO" } function io_functions( localvar) { # You've already seen print print "Hello world"; # There's also printf printf("%s %d %d %d\n", "Testing", 1, 2, 3); # AWK doesn't have file handles, per se. It will automatically open a file # handle for you when you use something that needs one. The string you used # for this can be treated as a file handle, for purposes of I/O. This makes # it feel sort of like shell scripting, but to get the same output, the string # must match exactly, so use a variable: outfile = "/tmp/foobar.txt"; print "foobar" &gt; outfile; # Now the string outfile is a file handle. You can close it: close(outfile); # Here's how you run something in the shell system("echo foobar"); # =&gt; prints foobar # Reads a line from standard input and stores in localvar getline localvar; # Reads a line from a pipe (again, use a string so you close it properly) cmd = "echo foobar"; cmd | getline localvar; # localvar =&gt; "foobar" close(cmd); # Reads a line from a file and stores in localvar infile = "/tmp/foobar.txt"; getline localvar &lt; infile; close(infile); } # As I said at the beginning, AWK programs consist of a collection of patterns # and actions. You've already seen the BEGIN pattern. Other # patterns are used only if you're processing lines from files or standard # input. # # When you pass arguments to AWK, they are treated as file names to process. # It will process them all, in order. Think of it like an implicit for loop, # iterating over the lines in these files. these patterns and actions are like # switch statements inside the loop. /^fo+bar$/ { # This action will execute for every line that matches the regular # expression, /^fo+bar$/, and will be skipped for any line that fails to # match it. Let's just print the line: print; # Whoa, no argument! That's because print has a default argument: $0. # $0 is the name of the current line being processed. It is created # automatically for you. # You can probably guess there are other $ variables. Every line is # implicitly split before every action is called, much like the shell # does. And, like the shell, each field can be access with a dollar sign # This will print the second and fourth fields in the line print $2, $4; # AWK automatically defines many other variables to help you inspect and # process each line. The most important one is NF # Prints the number of fields on this line print NF; # Print the last field on this line print $NF; } # Every pattern is actually a true/false test. The regular expression in the # last pattern is also a true/false test, but part of it was hidden. If you # don't give it a string to test, it will assume $0, the line that it's # currently processing. Thus, the complete version of it is this: $0 ~ /^fo+bar$/ { print "Equivalent to the last pattern"; } a &gt; 0 { # This will execute once for each line, as long as a is positive } # You get the idea. Processing text files, reading in a line at a time, and # doing something with it, particularly splitting on a delimiter, is so common # in UNIX that AWK is a scripting language that does all of it for you, without # you needing to ask. All you have to do is write the patterns and actions # based on what you expect of the input, and what you want to do with it. # Here's a quick example of a simple script, the sort of thing AWK is perfect # for. It will read a name from standard input and then will print the average # age of everyone with that first name. Let's say you supply as an argument the # name of a this data file: # # Bob Jones 32 # Jane Doe 22 # Steve Stevens 83 # Bob Smith 29 # Bob Barker 72 # # Here's the script: BEGIN { # First, ask the user for the name print "What name would you like the average age for?"; # Get a line from standard input, not from files on the command line getline name &lt; "/dev/stdin"; } # Now, match every line whose first field is the given name $1 == name { # Inside here, we have access to a number of useful variables, already # pre-loaded for us: # $0 is the entire line # $3 is the third field, the age, which is what we're interested in here # NF is the number of fields, which should be 3 # NR is the number of records (lines) seen so far # FILENAME is the name of the file being processed # FS is the field separator being used, which is " " here # ...etc. There are plenty more, documented in the man page. # Keep track of a running total and how many lines matched sum += $3; nlines++; } # Another special pattern is called END. It will run after processing all the # text files. Unlike BEGIN, it will only run if you've given it input to # process. It will run after all the files have been read and processed # according to the rules and actions you've provided. The purpose of it is # usually to output some kind of final report, or do something with the # aggregate of the data you've accumulated over the course of the script. END { if (nlines) print "The average age for " name " is " sum / nlines; } </code></pre> <h3><a href="https://github.com/Randy8080/reference/blob/main/awk.md">Cheat Sheet</a></h3> <p><strong>Have a try</strong></p> <pre><code class="language-shell">$ awk -F: '{print $1, $NF}' /etc/passwd </code></pre> <table> <thead> <tr> <th>- </th> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> </td> <td> <code>-F:</code> </td> <td> Colon as a separator </td> </tr> <tr> <td> </td> <td> <code>{...}</code> </td> <td> Awk program </td> </tr> <tr> <td> </td> <td> <code>print</code> </td> <td> Prints the current record </td> </tr> <tr> <td> </td> <td> <code>$1</code> </td> <td> First field </td> </tr> <tr> <td> </td> <td> <code>$NF</code> </td> <td> Last field </td> </tr> <tr> <td> </td> <td> <code>/etc/passwd</code> </td> <td> Input data file </td> </tr> </tbody> </table> <h4>Awk program</h4> <pre><code>BEGIN {&lt;initializations&gt;} &lt;pattern 1&gt; {&lt;program actions&gt;} &lt;pattern 2&gt; {&lt;program actions&gt;} ... END {&lt; final actions &gt;} </code></pre> <p><strong>Example</strong></p> <pre><code>awk ' BEGIN { print "\n&gt;&gt;&gt;Start" } !/(login|shutdown)/ { print NR, $0 } END { print "&lt;&lt;&lt;END\n" } ' /etc/passwd </code></pre> <h4>Variables</h4> <pre><code class="language-bash"> $1 $2/$(NF-1) $3/$NF ▼ ▼ ▼ ┌──────┬──────────────┬───────┐ $0/NR ▶ │ ID │ WEBSITE │ URI │ ├──────┼──────────────┼───────┤ $0/NR ▶ │ 1 │ quickref.me │ awk │ ├──────┼──────────────┼───────┤ $0/NR ▶ │ 2 │ google.com │ 25 │ └──────┴──────────────┴───────┘ </code></pre> <pre><code># First and last field awk -F: '{print $1,$NF}' /etc/passwd # With line number awk -F: '{print NR, $0}' /etc/passwd # Second last field awk -F: '{print $(NF-1)}' /etc/passwd # Custom string awk -F: '{print $1 "=" $6}' /etc/passwd </code></pre> <h4>Awk program examples</h4> <pre><code>awk 'BEGIN {print "hello world"}' # Prints "hello world" awk -F: '{print $1}' /etc/passwd # -F: Specify field separator # /pattern/ Execute actions only for matched pattern awk -F: '/root/ {print $1}' /etc/passwd # BEGIN block is executed once at the start awk -F: 'BEGIN { print "uid"} { print $1 }' /etc/passwd # END block is executed once at the end awk -F: '{print $1} END { print "-done-"}' /etc/passwd </code></pre> <p><strong>Conditions</strong></p> <pre><code>awk '{if ($3&gt;30) print $1}' /etc/passwd </code></pre> <p><strong>Generate 1000 spaces</strong></p> <pre><code>awk 'BEGIN{ while (a++ &lt; 1000) s=s " "; print s }' </code></pre> <p><strong>Arrays</strong></p> <pre><code>awk 'BEGIN { fruits["mango"] = "yellow"; fruits["orange"] = "orange" print fruits["orange"] print fruits["mango"] }' </code></pre> <p><strong>Functions</strong></p> <pre><code># =&gt; 5 awk 'BEGIN{print length("hello")}' # =&gt; HELLO awk 'BEGIN{print toupper("hello")}' # =&gt; hel awk 'BEGIN{print substr("hello", 1, 3)}' </code></pre> <h4>Variables</h4> <p><strong>Build-in variables</strong></p> <table> <thead></thead> <tbody> <tr> <td> <code>$0</code> </td> <td> Whole line </td> </tr> <tr> <td> <code>$1, $2...$NF</code> </td> <td> First, second… last field </td> </tr> <tr> <td> <code>NR</code> </td> <td> Total <code>N</code>umber of <code>R</code>ecords </td> </tr> <tr> <td> <code>NF</code> </td> <td> <code>N</code>number of <code>F</code>ields </td> </tr> <tr> <td> <code>OFS</code> </td> <td> <code>O</code>utput <code>F</code>ield <code>S</code>eparator <br> <em>(default " ")</em> </td> </tr> <tr> <td> <code>FS</code> </td> <td> input <code>F</code>ield <code>S</code>eparator <br> <em>(default " ")</em> </td> </tr> <tr> <td> <code>ORS</code> </td> <td> <code>O</code>utput <code>R</code>ecord <code>S</code>eparator <br> <em>(default "\n")</em> </td> </tr> <tr> <td> <code>RS</code> </td> <td> input <code>R</code>ecord <code>S</code>eparator <br> <em>(default "\n")</em> </td> </tr> <tr> <td> <code>FILENAME</code> </td> <td> Name of the file </td> </tr> </tbody> </table> <p><strong>Expressions</strong></p> <table> <thead></thead> <tbody> <tr> <td> <code>$1 == "root"</code> </td> <td> First field equals root </td> </tr> <tr> <td> <code>{print $(NF-1)}</code> </td> <td> Second last field </td> </tr> <tr> <td> <code>NR!=1{print $0}</code> </td> <td> From 2th record </td> </tr> <tr> <td> <code>NR &gt; 3</code> </td> <td> From 4th record </td> </tr> <tr> <td> <code>NR == 1</code> </td> <td> First record </td> </tr> <tr> <td> <code>END{print NR}</code> </td> <td> Total records </td> </tr> <tr> <td> <code>BEGIN{print OFMT}</code> </td> <td> Output format </td> </tr> <tr> <td> <code>{print NR, $0}</code> </td> <td> Line number </td> </tr> <tr> <td> <code>{print NR " " $0}</code> </td> <td> Line number (tab) </td> </tr> <tr> <td> <code>{$1 = NR; print}</code> </td> <td> Replace 1th field with line number </td> </tr> <tr> <td> <code>$NF &gt; 4</code> </td> <td> Last field &gt; 4 </td> </tr> <tr> <td> <code>NR % 2 == 0</code> </td> <td> Even records </td> </tr> <tr> <td> <code>NR==10, NR==20</code> </td> <td> Records 10 to 20 </td> </tr> <tr> <td> <code>BEGIN{print ARGC}</code> </td> <td> Total arguments </td> </tr> <tr> <td> <code>ORS=NR%5?",":"\n"</code> </td> <td> Concatenate records </td> </tr> </tbody> </table> <p><strong>Examples</strong><br> Print sum and average</p> <pre><code>awk -F: '{sum += $3} END { print sum, sum/NR } ' /etc/passwd </code></pre> <p>Printing parameters</p> <pre><code>awk 'BEGIN { for (i = 1; i &lt; ARGC; i++) print ARGV[i] }' a b c </code></pre> <p>Output field separator as a comma</p> <pre><code>awk 'BEGIN { FS=":";OFS=","} {print $1,$2,$3,$4}' /etc/passwd </code></pre> <p>Position of match</p> <pre><code>awk 'BEGIN { if (match("One Two Three", "Tw")) print RSTART }' </code></pre> <p>Length of match</p> <pre><code>awk 'BEGIN { if (match("One Two Three", "re")) print RLENGTH }' </code></pre> <p><strong>Environment Variables</strong></p> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>ARGC</code> </td> <td> Number or arguments </td> </tr> <tr> <td> <code>ARGV</code> </td> <td> Array of arguments </td> </tr> <tr> <td> <code>FNR</code> </td> <td> <code>F</code>ile <code>N</code>umber of <code>R</code>ecords </td> </tr> <tr> <td> <code>OFMT</code> </td> <td> Format for numbers <br> <em>(default "%.6g")</em> </td> </tr> <tr> <td> <code>RSTART</code> </td> <td> Location in the string </td> </tr> <tr> <td> <code>RLENGTH</code> </td> <td> Length of match </td> </tr> <tr> <td> <code>SUBSEP</code> </td> <td> Multi-dimensional array separator <br> <em>(default "\034")</em> </td> </tr> <tr> <td> <code>ARGIND</code> </td> <td> Argument Index </td> </tr> </tbody> </table> <h4>GNU awk only</h4> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>ENVIRON</code> </td> <td> Environment variables </td> </tr> <tr> <td> <code>IGNORECASE</code> </td> <td> Ignore case </td> </tr> <tr> <td> <code>CONVFMT</code> </td> <td> Conversion format </td> </tr> <tr> <td> <code>ERRNO</code> </td> <td> System errors </td> </tr> <tr> <td> <code>FIELDWIDTHS</code> </td> <td> Fixed width fields </td> </tr> </tbody> </table> <p><strong>Defining variable</strong></p> <pre><code>awk -v var1="Hello" -v var2="Wold" ' END {print var1, var2} ' &lt;/dev/null </code></pre> <p><strong>Use shell variables</strong></p> <pre><code>awk -v varName="$PWD" ' END {print varName}' &lt;/dev/null </code></pre> <h3>Operators</h3> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>{print $1}</code> </td> <td> First field </td> </tr> <tr> <td> <code>$2 == "foo"</code> </td> <td> Equals </td> </tr> <tr> <td> <code>$2 != "foo"</code> </td> <td> Not equals </td> </tr> <tr> <td> <code>"foo" in array</code> </td> <td> In array </td> </tr> </tbody> </table> <p><strong>Regular expression</strong></p> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>/regex/</code> </td> <td> Line matches </td> </tr> <tr> <td> <code>!/regex/</code> </td> <td> Line not matches </td> </tr> <tr> <td> <code>$1 ~ /regex/</code> </td> <td> Field matches </td> </tr> <tr> <td> <code>$1 !~ /regex/</code> </td> <td> Field not matches </td> </tr> </tbody> </table> <p><strong>More conditions</strong></p> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>($2 &lt;= 4 || $3 &lt; 20)</code> </td> <td> Or </td> </tr> <tr> <td> <code>($1 == 4 &amp;&amp; $3 &lt; 20)</code> </td> <td> And </td> </tr> </tbody> </table> <p><strong>Arithmetic operations</strong><br> - <code>+</code><br> - <code>-</code><br> - <code>*</code><br> - <code>/</code><br> - <code>%</code><br> - <code>++</code><br> - <code>--</code></p> <p><strong>Shorthand assignments</strong><br> - <code>+=</code><br> - <code>-=</code><br> - <code>*=</code><br> - <code>/=</code><br> - <code>%=</code></p> <p><strong>Comparison operators</strong><br> - <code>==</code><br> - <code>!=</code><br> - <code>&lt;</code><br> - <code>&gt;</code><br> - <code>&lt;=</code><br> - <code>&gt;=</code></p> <p><strong>Examples</strong></p> <pre><code>awk 'BEGIN { if ("foo" ~ "^fo+$") print "Fooey!"; }' </code></pre> <p><strong>Not match</strong></p> <pre><code>awk 'BEGIN { if ("boo" !~ "^fo+$") print "Boo!"; }' </code></pre> <p><strong>if in array</strong></p> <pre><code>awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("foo" in assoc) print "Fooey!"; }' </code></pre> <h4>Functions</h4> <p><strong>Common functions</strong></p> <table> <thead> <tr> <th> Function </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>index(s,t)</code> </td> <td> Position in string s where string t occurs, 0 if not found </td> </tr> <tr> <td> <code>length(s)</code> </td> <td> Length of string s (or $0 if no arg) </td> </tr> <tr> <td> <code>rand</code> </td> <td> Random number between 0 and 1 </td> </tr> <tr> <td> <code>substr(s,index,len)</code> </td> <td> Return len-char substring of s that begins at index (counted from 1) </td> </tr> <tr> <td> <code>srand</code> </td> <td> Set seed for rand and return previous seed </td> </tr> <tr> <td> <code>int(x)</code> </td> <td> Truncate x to integer value </td> </tr> <tr> <td> <code>split(s,a,fs)</code> </td> <td> Split string s into array a split by fs, returning length of a </td> </tr> <tr> <td> <code>match(s,r)</code> </td> <td> Position in string s where regex r occurs, or 0 if not found </td> </tr> <tr> <td> <code>sub(r,t,s)</code> </td> <td> Substitute t for first occurrence of regex r in string s (or $0 if s not given) </td> </tr> <tr> <td> <code>gsub(r,t,s)</code> </td> <td> Substitute t for all occurrences of regex r in string s </td> </tr> <tr> <td> <code>system(cmd)</code> </td> <td> Execute cmd and return exit status </td> </tr> <tr> <td> <code>tolower(s)</code> </td> <td> String s to lowercase </td> </tr> <tr> <td> <code>toupper(s)</code> </td> <td> String s to uppercase </td> </tr> <tr> <td> <code>getline</code> </td> <td> Set $0 to next input record from current input file. </td> </tr> </tbody> </table> <p><strong>User defined function</strong></p> <pre><code>awk ' # Returns minimum number function find_min(num1, num2){ if (num1 &lt; num2) return num1 return num2 } # Returns maximum number function find_max(num1, num2){ if (num1 &gt; num2) return num1 return num2 } # Main function function main(num1, num2){ result = find_min(num1, num2) print "Minimum =", result result = find_max(num1, num2) print "Maximum =", result } # Script execution starts here BEGIN { main(10, 60) } ' </code></pre> <h4>Arrays</h4> <p><strong>Array with index</strong></p> <pre><code>awk 'BEGIN { arr[0] = "foo"; arr[1] = "bar"; print(arr[0]); # =&gt; foo delete arr[0]; print(arr[0]); # =&gt; "" }' </code></pre> <p><strong>Array with key</strong></p> <pre><code>awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; print("baz" in assoc); # =&gt; 0 print("foo" in assoc); # =&gt; 1 }' </code></pre> <p><strong>Array with split</strong></p> <pre><code>awk 'BEGIN { split("foo:bar:baz", arr, ":"); for (key in arr) print arr[key]; }' </code></pre> <p><strong>Array with asort</strong></p> <pre><code>awk 'BEGIN { arr[0] = 3 arr[1] = 2 arr[2] = 4 n = asort(arr) for (i = 1; i &lt;= n ; i++) print(arr[i]) }' </code></pre> <p><strong>Multi-dimensional</strong></p> <pre><code>awk 'BEGIN { multidim[0,0] = "foo"; multidim[0,1] = "bar"; multidim[1,0] = "baz"; multidim[1,1] = "boo"; }' </code></pre> <p><strong>Multi-dimensional iteration</strong></p> <pre><code>awk 'BEGIN { array[1,2]=3; array[2,3]=5; for (comb in array) { split(comb,sep,SUBSEP); print sep[1], sep[2], array[sep[1],sep[2]] } }' </code></pre> <h4>Conditions</h4> <p><strong>if-else statement</strong></p> <pre><code>awk -v count=2 'BEGIN { if (count == 1) print "Yes"; else print "Huh?"; }' </code></pre> <p><strong>Ternary operator</strong></p> <pre><code>awk -v count=2 'BEGIN { print (count==1) ? "Yes" : "Huh?"; }' </code></pre> <p><strong>Exists</strong></p> <pre><code>awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("foo" in assoc) print "Fooey!"; }' </code></pre> <p><strong>Not exists</strong></p> <pre><code>awk 'BEGIN { assoc["foo"] = "bar"; assoc["bar"] = "baz"; if ("Huh" in assoc == 0 ) print "Huh!"; }' </code></pre> <p><strong>switch</strong></p> <pre><code>awk -F: '{ switch (NR * 2 + 1) { case 3: case "11": print NR - 1 break case /2[[:digit:]]+/: print NR default: print NR + 1 case -1: print NR * -1 } }' /etc/passwd </code></pre> <h4>Loops</h4> <p><strong><code>for...i</code></strong></p> <pre><code>awk 'BEGIN { for (i = 0; i &lt; 10; i++) print "i=" i; }' </code></pre> <p>**Powers of two between 1 and 100 **</p> <pre><code>awk 'BEGIN { for (i = 1; i &lt;= 100; i *= 2) print i }' </code></pre> <p><strong><code>for...in</code></strong></p> <pre><code>awk 'BEGIN { assoc["key1"] = "val1" assoc["key2"] = "val2" for (key in assoc) print assoc[key]; }' </code></pre> <p><strong>Arguments</strong></p> <pre><code>awk 'BEGIN { for (argnum in ARGV) print ARGV[argnum]; }' a b c </code></pre> <h4>Examples</h4> <p><strong>Reverse records</strong></p> <pre><code>awk -F: '{ x[NR] = $0 } END { for (i = NR; i &gt; 0; i--) print x[i] } ' /etc/passwd </code></pre> <p><strong>Reverse fields</strong></p> <pre><code>awk -F: '{ for (i = NF; i &gt; 0; i--) printf("%s ",$i); print "" }' /etc/passwd </code></pre> <p><strong>Sum by record</strong></p> <pre><code>awk -F: '{ s=0; for (i = 1; i &lt;= NF; i++) s += $i; print s }' /etc/passwd </code></pre> <p><strong>Sum whole file</strong></p> <pre><code>awk -F: ' {for (i = 1; i &lt;= NF; i++) s += $i; }; END{print s} ' /etc/passwd </code></pre> <p><strong><code>while</code></strong></p> <pre><code>awk 'BEGIN { while (a &lt; 10) { print "- " " concatenation: " a a++; } }' </code></pre> <p><strong><code>do...while</code></strong></p> <pre><code>awk '{ i = 1 do { print $0 i++ } while (i &lt;= 5) }' /etc/passwd </code></pre> <p><strong>Break</strong></p> <pre><code>awk 'BEGIN { break_num = 5 for (i = 0; i &lt; 10; i++) { print i if (i == break_num) break } }' </code></pre> <p><strong>Continue</strong></p> <pre><code>awk 'BEGIN { for (x = 0; x &lt;= 10; x++) { if (x == 5 || x == 6) continue printf "%d ", x } print "" }' </code></pre> <h4>Formatted Printing</h4> <p><strong>Right align</strong></p> <pre><code>awk 'BEGIN{printf "|%10s|\n", "hello"}' | hello| </code></pre> <p><strong>Left align</strong></p> <pre><code>awk 'BEGIN{printf "|%-10s|\n", "hello"}' |hello | </code></pre> <p>**Common specifiers **</p> <table> <thead> <tr> <th> Character </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>c</code> </td> <td> ASCII character </td> </tr> <tr> <td> <code>d</code> </td> <td> Decimal integer </td> </tr> <tr> <td> <code>e</code>, <code>E</code>, <code>f</code> </td> <td> Floating-point format </td> </tr> <tr> <td> <code>o</code> </td> <td> Unsigned octal value </td> </tr> <tr> <td> <code>s</code> </td> <td> String </td> </tr> <tr> <td> <code>%</code> </td> <td> Literal % </td> </tr> </tbody> </table> <p><strong>Space</strong></p> <pre><code>awk -F: '{ printf "%-10s %s\n", $1, $(NF-1) }' /etc/passwd | head -n 3 </code></pre> <p>Outputs</p> <pre><code class="language-shell">root /root bin /bin daemon /sbin </code></pre> <p><strong>Header</strong></p> <pre><code>awk -F: 'BEGIN { printf "%-10s %s\n", "User", "Home" printf "%-10s %s\n", "----","----"} { printf "%-10s %s\n", $1, $(NF-1) } ' /etc/passwd | head -n 5 </code></pre> <p>Outputs</p> <pre><code>User Home ---- ---- root /root bin /bin daemon /sbin </code></pre> <h4>Miscellaneous</h4> <p><strong>Regex Metacharacters</strong><br> - <code>\</code><br> - <code>^</code><br> - <code>$</code><br> - <code>.</code><br> - <code>[</code><br> - <code>]</code><br> - <code>|</code><br> - <code>(</code><br> - <code>)</code><br> - <code>*</code><br> - <code>+</code><br> - <code>?</code></p> <p><strong>Escape Sequences</strong></p> <table> <thead> <tr> <th> - </th> <th> - </th> </tr> </thead> <tbody> <tr> <td> <code>\b</code> </td> <td> Backspace </td> </tr> <tr> <td> <code>\f</code> </td> <td> Form feed </td> </tr> <tr> <td> <code>\n</code> </td> <td> Newline (line feed) </td> </tr> <tr> <td> <code>\r</code> </td> <td> Carriage return </td> </tr> <tr> <td> <code>\t</code> </td> <td> Horizontal tab </td> </tr> <tr> <td> <code>\v</code> </td> <td> Vertical tab </td> </tr> </tbody> </table> <p><strong>Run script</strong></p> <pre><code class="language-shell">$ cat demo.awk #!/usr/bin/awk -f BEGIN { x = 23 } { x += 2 } END { print x } $ awk -f demo.awk /etc/passwd 69 </code></pre> https://tedneward.github.io/Research/tools/gcc/static-analyzer/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gcc/static-analyzer/index.html <ul> <li><a href="https://gcc.gnu.org/onlinedocs/gcc/Static-Analyzer-Options.html">https://gcc.gnu.org/onlinedocs/gcc/Static-Analyzer-Options.html</a></li> <li><a href="https://gcc.gnu.org/wiki/DavidMalcolm/StaticAnalyzer">https://gcc.gnu.org/wiki/DavidMalcolm/StaticAnalyzer</a></li> <li>Static analysis in GCC 10 <ul> <li><a href="https://developers.redhat.com/blog/2020/03/26/static-analysis-in-gcc-10/">https://developers.redhat.com/blog/2020/03/26/static-analysis-in-gcc-10/</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/ghidra/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ghidra/index.html <p><a href="https://ghidra-sre.org/">Website</a> | <a href="https://github.com/NationalSecurityAgency/ghidra">Source</a></p> <p>JDK 21-based, using native plugins</p> https://tedneward.github.io/Research/tools/gitea/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gitea/index.html <p><a href="https://gitea.com/">Website</a> | <a href="https://github.com/go-gitea/gitea">Source</a></p> https://tedneward.github.io/Research/tools/glueysh/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/glueysh/index.html <p><a href="https://gluey.sh/">Website</a> | <a href="https://github.com/rebels-software/gluey">Source</a></p> https://tedneward.github.io/Research/tools/eleventy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/eleventy/index.html <p><a href="https://www.11ty.dev/">Website</a> | <a href="https://github.com/11ty/eleventy/">Source</a></p> https://tedneward.github.io/Research/tools/excalidraw/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/excalidraw/index.html <p><a href="https://excalidraw.com/">Website</a> | <a href="https://docs.excalidraw.com/">Docs</a> | <a href="https://github.com/excalidraw/excalidraw">Source</a> | <a href="https://hub.docker.com/r/excalidraw/excalidraw">DockerHub</a></p> https://tedneward.github.io/Research/tools/findbugs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/findbugs/index.html <p><a href="https://www.methodsandtools.com/tools/findbugs.php">Website</a> | <a href="http://findbugs.sourceforge.net/">Source</a></p> https://tedneward.github.io/Research/tools/fluidapp/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fluidapp/index.html <p><a href="https://fluidapp.com/">Website</a></p> <p>Free but no source.</p> https://tedneward.github.io/Research/tools/fray/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fray/index.html <p><a href="https://github.com/cmu-pasta/fray">Source</a> | <a href="https://dl.acm.org/doi/10.1145/3764119">Paper</a> | <a href="https://arxiv.org/abs/2501.12618">Technical Report</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.infoq.com/news/2025/12/fray-detects-concurrency-issues/">https://www.infoq.com/news/2025/12/fray-detects-concurrency-issues/</a></li> </ul> https://tedneward.github.io/Research/tools/functionsimsearch/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/functionsimsearch/index.html <p><a href="https://github.com/googleprojectzero/functionsimsearch">Github</a></p> <p><a href="https://googleprojectzero.blogspot.com/2018/12/searching-statically-linked-vulnerable.html">Searching statically-linked vulnerable library functions in executable code</a></p> https://tedneward.github.io/Research/tools/gaphor/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gaphor/index.html <p><a href="https://gaphor.org/">Website</a> | <a href="https://github.com/gaphor/gaphor">Source</a></p> https://tedneward.github.io/Research/tools/elkhound/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/elkhound/index.html <p><a href="http://scottmcpeak.com/elkhound/sources/elkhound/index.html">Website</a></p> https://tedneward.github.io/Research/tools/fastbuild/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fastbuild/index.html <p>A high performance, open-source build system for Windows, Linux, and OS X. Supports highly scalable compilation, caching and network distribution.</p> <p><a href="https://www.fastbuild.org/">Website</a> | <a href="https://github.com/fastbuild/fastbuild">Github</a></p> https://tedneward.github.io/Research/tools/findsecuritybugs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/findsecuritybugs/index.html <p><a href="https://find-sec-bugs.github.io/">Website</a> | <a href="https://github.com/find-sec-bugs/find-sec-bugs">Source</a></p> https://tedneward.github.io/Research/tools/fonts/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fonts/index.html <p><a href="https://github.com/tonsky/FiraCode">FiraCode</a>: Free monospaced font with programming ligatures</p> https://tedneward.github.io/Research/tools/frida/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/frida/index.html <p>Inject JavaScript to explore native apps on Windows, Mac, Linux, iOS, Android, and QNX.</p> <p><a href="https://www.frida.re/">Website</a> | <a href="https://github.com/frida/frida">Source</a></p> <p><a href="https://github.com/dweinstein/awesome-frida">Awesome Frida</a>: A curated list of Frida resources</p> <p><a href="https://github.com/agustingianni/memrepl">MemREPL</a>: Memory inspection REPL interface</p> https://tedneward.github.io/Research/tools/fx/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fx/index.html <p><a href="https://fx.wtf/">Website</a> (Currently redirects to) <a href="https://github.com/antonmedv/fx">Source</a> | <a href="https://github.com/antonmedv/fx/blob/master/DOCS.md">Docs</a>.</p> <p>Install: <code>brew install fx</code> or <code>npm install -g fx</code></p> <h2>Usage</h2> <p>Start <a href="https://github.com/antonmedv/fx/blob/master/DOCS.md#interactive-mode">interactive mode</a> without passing any arguments.</p> <pre><code class="language-bash">$ curl ... | fx </code></pre> <p>Or by passing filename as first argument.</p> <pre><code class="language-bash">$ fx data.json </code></pre> <p>Pass a few JSON files.</p> <pre><code class="language-bash">cat foo.json bar.json baz.json | fx .message </code></pre> <p>Use full power of JavaScript.</p> <pre><code class="language-bash">$ curl ... | fx '.filter(x =&gt; x.startsWith("a"))' </code></pre> <p>Access all lodash (or ramda, etc) methods by using <a href="https://github.com/antonmedv/fx/blob/master/DOCS.md#using-fxrc">.fxrc</a> file.</p> <pre><code class="language-bash">$ curl ... | fx '_.groupBy("commit.committer.name")' '_.mapValues(_.size)' </code></pre> <p>Update JSON using spread operator.</p> <pre><code class="language-bash">$ echo '{"count": 0}' | fx '{...this, count: 1}' { "count": 1 } </code></pre> <p>Extract values from maps.</p> <pre><code class="language-bash">$ fx commits.json | fx .[].author.name </code></pre> <p>Print formatted JSON to stdout.</p> <pre><code class="language-bash">$ curl ... | fx . </code></pre> <p>Pipe JSON logs stream into fx.</p> <pre><code class="language-bash">$ kubectl logs ... -f | fx .message </code></pre> <p>And try this:</p> <pre><code class="language-bash">$ fx --life </code></pre> <h2>Links</h2> <ul> <li><a href="http://bit.ly/discover-how-to-use-fx-effectively">Discover how to use fx effectively</a></li> <li><a href="http://bit.ly/youtube-fx-tutorial">Video tutorial</a></li> </ul> <h2>Related</h2> <ul> <li><a href="https://github.com/antonmedv/gofx">gofx</a> – fx-like JSON tool (<em>go</em>)</li> <li><a href="https://github.com/antonmedv/eat">eat</a> – converts anything into JSON</li> <li><a href="https://github.com/matthewadams/ymlx">ymlx</a> – fx-like YAML cli processor</li> <li><a href="https://github.com/antonmedv/fx-completion">fx-completion</a> – bash completion for fx</li> <li><a href="https://github.com/antonmedv/fx-theme-monokai">fx-theme-monokai</a> – monokai theme</li> <li><a href="https://github.com/antonmedv/fx-theme-night">fx-theme-night</a> – night theme</li> </ul> https://tedneward.github.io/Research/tools/gatsbyjs/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gatsbyjs/index.html <p>title=Gatsby (GatsbyJS)<br> tags=tool, static site generator, nodejs<br> summary=React-based static site generator.</p> <pre><code></code></pre> https://tedneward.github.io/Research/tools/entire/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/entire/index.html <p><a href="https://entire.io/">Website</a></p> https://tedneward.github.io/Research/tools/filebytes/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/filebytes/index.html <p><a href="https://scoding.de/filebytes-introduction">Website</a> | <a href="https://github.com/sashs/filebytes/">Github</a></p> https://tedneward.github.io/Research/tools/fizzy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fizzy/index.html <p><a href="https://github.com/basecamp/fizzy">Source</a></p> https://tedneward.github.io/Research/tools/forgejo/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/forgejo/index.html <p><a href="https://forgejo.org/">Website</a> | <a href="https://codeberg.org/forgejo/forgejo">Source</a> | <a href="https://forgejo.org/docs/latest/">Docs</a></p> <p>Forgejo is a self-hosted lightweight software forge. Easy to install and low maintenance, it just does the job.</p> <p>Built in Go, looks like.</p> <h1><a href="https://codeberg.org/teaserbot-labs/delightful">Delightful Forgejo</a></h1> <h2>Official resources</h2> <p>Resources maintained as part of the primary Forgejo project.<br> - <a href="https://forgejo.org/">![][forgejo] Forgejo website</a><br> - <a href="https://forgejo.org/docs/latest/">![][forgejo] Forgejo documentation</a><br> - <a href="https://codeberg.org/forgejo/forgejo">![][forgejo] Forgejo issue tracker</a><br> - <a href="https://forgejo.org/releases">![][forgejo] Forgejo releases</a> (<a href="https://forgejo.org/releases/rss.xml">RSS feed</a>; <a href="https://codeberg.org/forgejo/forgejo/src/branch/forgejo/RELEASE-NOTES.md">Release notes</a>)<br> - <a href="https://code.forgejo.org/forgejo/runner">![][forgejo] Forgejo runner</a><br> - <a href="https://codeberg.org/forgejo/-/packages/container/forgejo/versions">![][forgejo] Forgejo container images</a><br> - <a href="https://floss.social/@forgejo">![][forgejo] Forgejo Mastodon account</a><br> - <a href="https://keyoxide.org/contact@forgejo.org">![][forgejo] Forgejo identity proofs (Keyoxide)</a></p> <h2>Documentation</h2> <p>In addition to <a href="https://forgejo.org/docs/latest/">![][forgejo] Forgejo's official documentation</a>, the following resources may be useful.</p> <ul> <li><a href="https://docs.codeberg.org/">Codeberg documentation</a> - contains some Codeberg-specific parts but is mostly applicable to every Forgejo instance.</li> <li><a href="https://docs.gitea.io/">Gitea documentation</a> - mostly applicable to Forgejo as well as Gitea.</li> </ul> <h3>Tutorials</h3> <ul> <li><a href="https://devforum.roblox.com/t/forgejo-in-a-nutshell/2505867">Forgejo in a nutshell</a> - guide to setup Forgejo.</li> <li><a href="https://blog.gibson.sh/2023/05/28/server-with-wireguard-and-forgejo/">Forgejo with LFS behind a VPN</a> - guide to set up a Linux web server running Forgejo, Git-LFS, and a Wireguard VPN.</li> <li><a href="https://lab.uberspace.de/guide_forgejo/">UberLab installation guide</a> - Forgejo installation guide for <a href="https://uberspace.de/">Uberspace</a>.</li> <li><a href="https://linus.dev/posts/setting-up-a-self-hosted-forgejo-actions-runner-with-docker-compose/">Setting up a Self-Hosted Forgejo Actions Runner with Docker Compose</a>.</li> <li><a href="https://jan.wildeboer.net/2024/08/Running-a-runner-codeberg/">Running a runner for codeberg/forgejo on RHEL9 (Red Hat Enterprise Linux) with podman as user systemd service</a></li> </ul> <h2>Public instances</h2> <p>Forgejo instances with open registration.<br> - <a href="https://codeberg.org">Codeberg</a> - free and open <em>for FOSS projects only</em>, run by a German non-profit organization. Also provides hosted Woodpecker (CI) and Weblate (localisation).<br> - <a href="https://codefloe.com">CodeFloe</a> - free and open. Legally operated by <a href="https://devxy.io">devXY</a>, a Swiss DevOps company. Open to community engagement and management.<br> Provides Crow CI and Forgejo Actions for CI/CD and a <a href="https://forum.codefloe.com">Forum</a> for public user discussions.<br> - <a href="https://disroot.org/en/services/git">Disroot</a> - a platform providing online services based on principles of freedom, privacy, federation and decentralization. Also provides many other free hosted services.<br> - <a href="https://git.pub.solar">pub.solar</a> - Register/Login via <a href="https://auth.pub.solar">pub.solar ID</a>, run by a German tech collective with the goal to enable more people to use free software, have secure digital communications, and to take control over their private data.<br> - <a href="https://git.kaki87.net">KaKi's git</a> - free and open, run by a French web developer.<br> - <a href="https://sij.ai">sij.ai</a> - Free and open platform run by a public interest environmental lawyer that is open for anyone but especially intended for AI-ML hobbyists &amp; enthusiasts.<br> - <a href="https://git.gay/">git.gay</a> - instance run by <a href="https://besties.house/">Besties</a>, a queer collective. Uses modified version of Forgejo.<br> - <a href="https://bolha.dev/">bolha.dev</a> - instance run by <a href="https://bolha.us/">bolha.us</a>, and <a href="https://bolha.io/">bolha.io</a> a Brazilian IT collective. Running the vanilla version.</p> <h2>Packaging</h2> <p>Platform-specific packages to easily install and update Forgejo on your system. These are generally maintained either by distro packagers or by community volunteers.</p> <p><a href="https://repology.org/project/forgejo/versions"><br> <img src="https://repology.org/badge/vertical-allrepos/forgejo.svg" alt="Packaging status" align="right"><br> </a></p> <p>Also see <a href="https://repology.org/project/forgejo/versions">forgejo package versions on Repology</a>.<br> - <strong>Alpine</strong><br> - <a href="https://pkgs.alpinelinux.org/packages?name=forgejo">Alpine community <code>forgejo</code> package</a><br> - <a href="https://pkgs.alpinelinux.org/packages?name=forgejo-lts">Alpine community <code>forgejo-lts</code> package</a><br> - <strong>Arch Linux</strong><br> - <a href="https://archlinux.org/packages/extra/x86_64/forgejo">Arch extra <code>forgejo</code> package</a></p> <!-- - **AOSC OS** - 👻 [AOSC OS stable `forgejo` package](https://packages.aosc.io/packages/forgejo)--> <ul> <li><strong>Debian/Ubuntu</strong> <ul> <li><a href="https://codeberg.org/forgejo-contrib/forgejo-deb">Unofficial <code>forgejo-deb</code> packages + repo</a> <img src="https://codeberg.org/forgejo-contrib/forgejo-deb/badges/release.svg" alt="package version"></li> </ul> </li> <li><strong>Fedora</strong> <ul> <li><a href="https://packages.fedoraproject.org/pkgs/forgejo/">Fedora <code>forgejo</code> package</a></li> <li><a href="https://copr.fedorainfracloud.org/coprs/mdwalters/forgejo/">RPM copr repo</a> <img src="https://img.shields.io/badge/dynamic/json?url=https%3A%2F%2Fcopr.fedorainfracloud.org%2Fapi_3%2Fpackage%2F%3Fownername%3Dmdwalters%26projectname%3Dforgejo%26packagename%3Dforgejo%26with_latest_build%3Dfalse%26with_latest_succeeded_build%3Dfalse&amp;query=%24.builds.latest_succeeded.source_package.version&amp;label=version" alt="package version"></li> </ul> </li> <li><strong>FreeBSD</strong> <ul> <li><a href="https://www.freshports.org/www/forgejo">Port <code>www/forgejo</code> / package <code>forgejo</code></a> - binary packages available in latest and quarterly 2024Q2 or newer.</li> </ul> </li> <li><strong>Gentoo</strong> <ul> <li><a href="https://gitweb.gentoo.org/repo/proj/guru.git/tree/www-apps/forgejo">Gentoo GURU <code>www-apps/forgejo</code> package</a></li> </ul> </li> <li><strong>Kubernetes</strong> <ul> <li><a href="https://codeberg.org/forgejo-contrib/forgejo-helm">Helm chart</a> - Helm chart for Forgejo, forked from the official Gitea helm chart</li> <li><a href="https://repo.prod.meissa.de/meissa/c4k-forgejo">c4k-forgejo</a> - Convention 4 Kubernetes, generates a kubernetes manifest for Forgejo including backup &amp; monitoring.</li> </ul> </li> <li><strong>NixOS</strong> <ul> <li><a href="https://nixos.org/manual/nixos/stable/options#opt-services.forgejo.enable">NixOS <code>forgejo</code> service</a></li> </ul> </li> <li><strong>Snapcraft</strong> <ul> <li><a href="https://codeberg.org/forgejo-contrib/snap">Unofficial <code>forgejo</code> snap package</a> - building in <a href="https://launchpad.net/~popey/forgejo-snap/+snap/forgejo-snap">launchpad</a> <img src="https://snapcraft.io/forgejo/badge.svg" alt="package version"></li> </ul> </li> <li><strong>Termux (Android)</strong> <ul> <li><a href="https://github.com/termux/termux-packages/tree/master/packages/forgejo">Termux <code>forgejo</code> package</a></li> </ul> </li> <li><strong>YunoHost</strong> <ul> <li><a href="https://github.com/YunoHost-Apps/forgejo_ynh">YunoHost package</a> - install Forgejo quickly and simply on a YunoHost server</li> </ul> </li> </ul> <!-- - **Cloudron** - 👻 [Cloudron package](https://codeberg.org/bart/forgejo-app) - Install Forgejo on a Cloudron server--> <h2>CI/CD</h2> <p>CI/CD tools that integrate via Forgejo officially or inofficially.</p> <ul> <li><a href="#forgejo-actions">Forgejo Actions</a> - the integrated CI/CD solution with external runners. See below.</li> <li><a href="https://woodpecker-ci.org/">Woodpecker CI</a> - a community-maintained powerful CI/CD based on Docker containers, historically forked from Drone CI. <ul> <li><a href="https://crowci.dev/">Crow CI</a> - a soft-fork of Woodpecker CI.</li> </ul> </li> <li><a href="https://www.jenkins.io/">Jenkins</a> - an extensible CI/CD with a large plugin ecosystem. <ul> <li><a href="https://plugins.jenkins.io/gitea/">Gitea Plugin</a> - use Jenkins with Forgejo (API-compatible to Gitea).</li> <li><a href="https://plugins.jenkins.io/gitea-checks/">Gitea Checks Plugin</a> - use Forgejo status checks with Jenkins.</li> </ul> </li> <li><a href="https://codeberg.org/emersion/yojo">yoyo Sourcehut CI bridge</a> - develop on Forgejo and run builds within the Sourcehut CI.</li> <li><a href="https://agola.io/">agola</a> - containerized and flexible CI/CD platform.</li> <li><a href="https://www.buildbot.net/">Buildbot</a> - a flexible CI/CD framework. Can be used <a href="https://github.com/lab132/buildbot-gitea">with a plugin</a>.</li> <li><a href="https://codeberg.org/snaums/mvoCI">mvoCI</a> - a simple and personal CI/CD solution.</li> </ul> <h2>Forgejo Actions</h2> <p>Learn about Forgejo actions in the <a href="https://forgejo.org/docs/latest/admin/actions/">![][forgejo] Actions admin guide</a> and <a href="https://forgejo.org/docs/latest/user/actions/">![][forgejo] Actions user guide</a>.</p> <h3>Actions runners</h3> <ul> <li><a href="https://code.forgejo.org/forgejo/runner">![][forgejo] Forgejo runner</a> - supports LXC &amp; Docker &amp; shell with binaries for GNU/Linux (amd64, arm64)</li> <li><a href="https://gitea.com/gitea/act_runner">act runner</a> - supports Docker &amp; shell with binaries for GNU/Linux (amd64, arm64, arm[567]), FreeBSD, Windows, MacOS</li> </ul> <h3>Actions runner deployment tools</h3> <ul> <li><a href="https://codeberg.org/pierreprinetti/forgejo-hetzner-runner">forgejo-hetzner-runner</a> - Deploy Forgejo Actions runners on Hetzner infrastructure</li> <li><a href="https://codeberg.org/wrenix/helm-charts/src/branch/main/forgejo-runner">helm-chart</a> - Deploy a Forgejo Actions runner on the specified Forgejo instance</li> </ul> <h3>Actions</h3> <ul> <li><a href="https://code.forgejo.org/Seltsamsel/trust-self-signed-cert">trust-self-signed-cert</a> - Trust a self-signed SSL certificate for future operations (e.g. checkout action).</li> <li><a href="https://codeberg.org/kemitix/forgejo-todo-checker">forgejo-todo-checker</a> - Checks your source files for TODO and FIXME comments, where they don't have an open issue number.</li> <li><a href="https://codeberg.org/kemitix/forgejo-commit-path-rules">forgejo-commit-path-rules</a> - Enforce consistency between commit messages and file changes in your repository.</li> <li><a href="https://codeberg.org/Native-CI/">Native-CI</a> - A collection of actions for helping maintain native projects (C/C++/...) that may require cross-platform builds.</li> </ul> <h2>Clients</h2> <h3>CMS (Content Management System)</h3> <ul> <li><a href="https://github.com/sveltia/sveltia-cms">Sveltia CMS</a> - Configurable admin panel for static site generator authoring.</li> <li><a href="https://decapcms.org/docs/gitea-backend/">DecapCMS</a> (<a href="https://github.com/decaporg/decap-cms">repo</a>) - Configurable admin panel for static site generator authoring.</li> </ul> <h3>Mobile</h3> <ul> <li><a href="https://gitnex.com/">GitNex</a> (<a href="https://codeberg.org/gitnex/GitNex">repo</a>) - Android client for Forgejo and Gitea (<code>Android – GPL</code>)</li> <li>👻 <a href="https://github.com/git-touch/git-touch">GitTouch</a> - Mobile client for Forgejo, Gitea, GitHub, GitLab, Bitbucket (<code>Android, iOS – Apache</code>)</li> </ul> <h3>Command-line</h3> <ul> <li><a href="https://codeberg.org/Aviac/codeberg-cli">codeberg-cli</a> - CLI tool for Forgejo similar to <code>gh</code>, <code>glab</code> and <code>tea</code> (<code>cross-platform - AGPL</code>)</li> <li><a href="https://codeberg.org/forgejo-contrib/forgejo-cli">forgejo-cli</a> - CLI tool for interacting with Forgejo (<code>cross-platform - Apache/MIT</code>)</li> <li><a href="https://code.forgejo.org/forgejo/forgejo-curl">![][forgejo] forgejo-curl</a> - a thin curl wrapper that helps with Forgejo authentication</li> <li><a href="https://gitea.com/gitea/tea">tea</a> - the official Gitea CLI client, works with Forgejo (<code>cross-platform – MIT</code>)</li> </ul> <h3>Emacs</h3> <ul> <li><a href="https://codeberg.org/martianh/fj.el">fj.el</a> - a basic Emacs client for Forgejo.</li> </ul> <h2>Tools</h2> <h3>Infrastructure as Code</h3> <ul> <li><a href="https://forgejo.dev/forgejo.dev/infrastructure-as-code/">Terraform &amp; Ansible playbook</a> - Deploy Forgejo, Minio, Nginx &amp; Woodpecker in the cloud via <a href="https://docs.docker.com/compose/">Docker Compose</a>, depends on related <a href="https://github.com/geerlingguy">geerlingguy</a> playbooks.</li> <li><a href="https://registry.terraform.io/providers/svalabs/forgejo/latest/docs">Terraform Provider for Forgejo</a> - Allows managing resources (organizations, repositories, users) within Forgejo.</li> <li><a href="https://galaxy.ansible.com/ui/repo/published/l3d/git/content/role/gitea/">Ansible Collection l3d.git.gitea</a> - Ansible collection to install and configure forgejo.</li> </ul> <h3>Monitoring</h3> <ul> <li><a href="https://gitea.zionetrix.net/bn8/check_forgejo_upgrade">Monitoring plugin to check Forgejo instance upgrade status</a> has the sources and is published in the <a href="https://exchange.icinga.com/brenard/check_forgejo_upgrade">Icinga exchange repository</a>.</li> </ul> <h3>Bots</h3> <ul> <li>👻 <a href="https://codeberg.org/spla/gitcat.git">gitcat</a> - Python script that allows sign-up to a Forgejo instance to all local users of a Mastodon server. (<code>Python - GPLv3</code>)</li> <li><a href="https://docs.renovatebot.com/modules/platform/gitea/#gitea-and-forgejo">Renovate</a> - Dependency update tool (similar to dependabot) with Forgejo support. (<code>TypeScript - AGPLv3</code>)</li> <li><a href="https://codeberg.org/kita/nayrah">Nayrah</a> - A Discord bot that retrieves various information from a Forgejo instance like Codeberg. (<code>Python - AGPLv3</code>)</li> <li>👻 <a href="https://git.4rs.nl/awiteb/forgejo-guardian">forgejo-guardian</a> - Simple Forgejo instance guardian, banning users and alerting admins based on certain regular expressions. (<code>Rust - AGPLv3</code>)</li> <li><a href="https://codeberg.org/kemitix/git-next">git-next</a> - Trunk-based development manager (with Forgejo &amp; Github support).</li> </ul> <h3>Scripts</h3> <ul> <li><a href="https://github.com/maxkratz/github2gitea-mirror">github2gitea/github2forgejo</a> - Bash script that creates mirrors for various GitHub resources like orgs, users and starred repos (incl. private repos). (<code>Bash - AGPLv3</code>)</li> <li><a href="https://git.kaki87.net/KaKi87/userscripts/src/branch/master/enhancementsForForgejo/README.md">Enhancements for Forgejo</a> - Violentmonkey userscript that adds minor UX improvements to the web UI. (<code>JavaScript - MIT</code>)</li> </ul> <h3>Package deployment</h3> <ul> <li><a href="https://codeberg.org/rome-user/lein-forgejo-wagon">lein-forgejo-wagon</a> - Leiningen plugin for deployment and downloading of JARs in Maven repositories hosted on Forgejo packages.</li> </ul> <h3>API client libraries</h3> <ul> <li><a href="https://codeberg.org/Cyborus/forgejo-api">Cyborus/forgejo-api</a> - Rust crate to interact with the Forgejo API (<code>Rust - Apache or MIT</code>)</li> <li><a href="https://codeberg.org/harabat/pyforgejo">harabat/pyforgejo</a> - Python client library to interact with the Forgejo API (<code>Python - MIT</code>)</li> <li><a href="https://codeberg.org/mvdkleijn/forgejo-sdk">mvdkleijn/forgejo-sdk</a> - Go client library to interact with the Forgejo API (fork of <a href="https://gitea.com/gitea/go-sdk">gitea/go-sdk</a>) (<code>Go - MIT</code>)</li> <li><a href="https://codeberg.org/anbraten/forgejo-js">forgejo-js</a> - JavaScript / Typescript<br> library to interact with the Forgejo API (fork of <a href="https://github.com/anbraten/gitea-js">gitea-js</a>) (<code>JavaScript / Typescript - MIT</code>)</li> </ul> <h2>Forks</h2> <p>Actively maintained Forgejo forks.</p> <ul> <li><a href="https://fordj.org/">Fordj</a> - a git-based design-platform for the AEC (Architecture, Engineering and Construction).</li> <li><a href="https://codeberg.org/forgejo-aneksajo/forgejo-aneksajo/">Forgejo-aneksajo</a> - (temporary) fork of Forgejo that adds an integration with git-annex.</li> </ul> https://tedneward.github.io/Research/tools/fsnotes/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fsnotes/index.html <p><a href="https://fsnot.es/">Website</a> | <a href="https://github.com/glushchenko/fsnotes">Source</a></p> <h3>Features</h3> <ul> <li>Markdown-first. Also supports plaintext and RTF files.</li> <li>Fast and lightweight. Works smoothly with 10k+ files.</li> <li>Access anywhere. Sync with iCloud Drive or Dropbox.</li> <li>Multi-folder storage.</li> <li>Keyboard-centric. nvalt-inspired controls and shortcuts.</li> <li>Syntax highlighting within code blocks. Supports over 170 programming languages.</li> <li>In-line image support.</li> <li>Organize with tags.</li> <li>Cross-note links using [[double brackets]].</li> <li>Elastic two-pane view. Choose a vertical or horizontal layout.</li> <li>External editor support (changes seamless live sync with UI).</li> <li>Pin important notes.</li> <li>Quickly copy notes to the clipboard.</li> <li>Dark mode.</li> <li>Lock sensitive notes with AES-256 encryption.</li> <li>Mermaid and MathJax support.</li> <li>Optional Git versioning and backups.</li> </ul> https://tedneward.github.io/Research/tools/fzf/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fzf/index.html <p><a href="https://github.com/junegunn/fzf">Source</a></p> https://tedneward.github.io/Research/tools/gcc/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gcc/index/index.html <p><a href="https://gcc.gnu.org/">Website</a> | <a href="https://web.archive.org/web/20170326232435/http://www.network-theory.co.uk/docs/gccintro/index.html">An Introduction to GCC</a> - Brian Gough</p> <ul> <li>GCC online documentation - <a href="https://gcc.gnu.org/onlinedocs/">https://gcc.gnu.org/onlinedocs/</a> <ul> <li>Cynbe's GCC Debugging Hints - <a href="http://muq.org/~cynbe/gcc/">http://muq.org/~cynbe/gcc/</a> <ul> <li>Debugging Resources - <a href="http://muq.org/~cynbe/gcc/offsite-resources.html">http://muq.org/~cynbe/gcc/offsite-resources.html</a></li> <li>Comments on the Internals Manual - <a href="http://muq.org/~cynbe/gcc/gccint.html">http://muq.org/~cynbe/gcc/gccint.html</a></li> </ul> </li> <li>GCC Wiki - <a href="https://gcc.gnu.org/wiki">https://gcc.gnu.org/wiki</a></li> </ul> </li> </ul> <p><a href="https://gcc.gnu.org/projects/cxx-status.html">C++ Status</a></p> <p><a href="/tools/gcc/static-analyzer">Static analyzer options</a></p> <p><a href="https://tldp.org/HOWTO/GCC-Frontend-HOWTO.html">GCC Front-end HOWTO</a>: How to build a GCC front-end (so GCC can take care of the rest)</p> <h2>Viewing exported symbols from executables</h2> <h3><code>readelf</code></h3> <pre><code>$ readelf -s lib.so Symbol table '.dynsym' contains 8 entries: Num: Value Size Type Bind Vis Ndx Name 0: 0000000000000000 0 NOTYPE LOCAL DEFAULT UND 1: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_deregisterT[...] 2: 0000000000000000 0 FUNC GLOBAL DEFAULT UND puts@GLIBC_2.2.5 (2) 3: 0000000000000000 0 NOTYPE WEAK DEFAULT UND __gmon_start__ 4: 0000000000000000 0 NOTYPE WEAK DEFAULT UND _ITM_registerTMC[...] 5: 0000000000000000 0 FUNC WEAK DEFAULT UND [...]@GLIBC_2.2.5 (2) 6: 000000000000111f 22 FUNC GLOBAL DEFAULT 12 lib_exported2 7: 0000000000001109 22 FUNC GLOBAL DEFAULT 12 lib_exported1 ... </code></pre> <h3><code>objdump</code></h3> <pre><code>$ objdump -T lib.so lib.so: file format elf64-x86-64 DYNAMIC SYMBOL TABLE: 0000000000000000 w D *UND* 0000000000000000 Base _ITM_deregisterTMCloneTable 0000000000000000 DF *UND* 0000000000000000 GLIBC_2.2.5 puts 0000000000000000 w D *UND* 0000000000000000 Base __gmon_start__ 0000000000000000 w D *UND* 0000000000000000 Base _ITM_registerTMCloneTable 0000000000000000 w DF *UND* 0000000000000000 GLIBC_2.2.5 __cxa_finalize 000000000000111f g DF .text 0000000000000016 Base lib_exported2 0000000000001109 g DF .text 0000000000000016 Base lib_exported1 </code></pre> <p>Does not demangle C++ names unless passed the `--demangle`` flag.</p> <h3><code>nm</code></h3> <pre><code>$ nm -D --demangle lib.so w __cxa_finalize@GLIBC_2.2.5 w __gmon_start__ w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable U puts@GLIBC_2.2.5 0000000000001109 T lib_exported1() 000000000000111f T lib_exported2() </code></pre> <p>Does not demangle C++ names unless passed `--demangle`` flag.</p> https://tedneward.github.io/Research/tools/exa/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/exa/index.html <p><a href="https://the.exa.website/">Website</a> | <a href="https://the.exa.website/introduction">Docs</a> | <a href="https://the.exa.website/install/source">Sources</a></p> https://tedneward.github.io/Research/tools/fileformat/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fileformat/index.html <p>COFF, ELF, Intel HEX, Mach-O, PE, raw data</p> <p><a href="https://github.com/avast-tl/fileformat">Github</a></p> https://tedneward.github.io/Research/tools/flubucore/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/flubucore/index.html <p><a href="https://flubucore.dotnetcore.xyz/">Website</a> | <a href="https://github.com/dotnetcore/FlubuCore/">Source</a></p> <p>Examples:</p> <pre><code>[FromArg("nugetKey", "Nuget api key for publishing Flubu nuget packages.")] public string NugetApiKey { get; set; } protected override void ConfigureTargets(ITaskContext context) { var pack = context.CreateTarget("Pack") .SetDescription("Prepare's nuget package.") .AddCoreTask(x =&gt; x.Pack() .NoBuild() .OutputDirectory(OutputDirectory) .WithArguments("--force"); //you can add your own custom arguments on each task var branch = context.BuildSystems().Travis().Branch; //// Examine travis.yaml to see how to pass api key from travis to FlubuCore build script. var nugetPush = context.CreateTarget("Nuget.publish") .SetDescription("Publishes nuget package.") .DependsOn(pack) .AddCoreTask(x =&gt; x.NugetPush($"{OutputDirectory}/NetCoreOpenSource.nupkg") .ServerUrl("https://www.nuget.org/api/v2/package") .ApiKey(NugetApiKey) ) .When((c) =&gt; c.BuildSystems().RunningOn == BuildSystemType.TravisCI &amp;&amp; !string.IsNullOrEmpty(branch) &amp;&amp; branch.EndsWith("stable", StringComparison.OrdinalIgnoreCase)); } </code></pre> <pre><code>context.CreateTarget("build") .AddTask(x =&gt; x.GitVersionTask()) .AddTask(x =&gt; x.CompileSolutionTask("MySolution.sln").BuildConfiguration("Release"); context.CreateTarget("run.tests") .AddTask(x =&gt; x.XunitTaskByProjectName("MyProject").StopOnFail()) .AddTask(x =&gt; x.NUnitTask(NunitCmdOptions.V3, "MyProject2").ExcludeCategory("Linux")) .AddCoreTask(x =&gt; x.CoverletTask("MyProject.dll")); </code></pre> <pre><code>[NugetPackage("Newtonsoft.json", "11.0.2")] [Assembly(".\Lib\EntityFramework.dll")] public class BuildScript : DefaultBuildScript { public void NuGetPackageReferencingExample(ITaskContext context) { JsonConvert.SerializeObject("Example"); } } </code></pre> <pre><code>public class SimpleScript : DefaultBuildScript { protected override void ConfigureTargets(ITaskContext context) { context.CreateTarget("Run.Libz") .AddTask(x =&gt; x.RunProgramTask(@"packages\LibZ.Tool\1.2.0\tools\libz.exe") .WorkingFolder(@".\src") .WithArguments("add") .WithArguments("--libz", "Assemblies.libz")); } } </code></pre> <pre><code>public class SimpleScript : DefaultBuildScript { [FromArg("c", "The configuration to use for building the project.")] public string Configuration { get; set; } = "Release" [FromArg("sn", "If true app is deployed on second node. Otherwise not.")] public bool deployOnSecondNode { get; set; } protected override void ConfigureTargets(ITaskContext context) { context.CreateTarget("build") .AddCoreTask(x =&gt; x.Build() .Configuration(Configuration) .ForMember(x =&gt; x.Framework("net462"), "f", "The target framework to build for.")); } } </code></pre> <pre><code>dotnet tool install --global FlubuCore.GlobalTool flubu compile </code></pre> <pre><code>context.WaitForDebugger(); </code></pre> https://tedneward.github.io/Research/tools/formae/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/formae/index.html <p><a href="https://platform.engineering/formae">Website</a> | <a href="https://github.com/platform-engineering-labs/formae">Source</a></p> <p><strong>formae</strong> doesn't require its user to maintain any secondary artifacts such as state files, and keeps the infrastructure code automatically in sync with the reality.</p> <h2><strong>formae</strong> capabilities</h2> <ul> <li><strong>The single source of truth is code:</strong> It unifies every infrastructure resource and change into fully versioned infrastructure code.</li> <li><strong>Always up-to-data infrastructure code:</strong> It sees when things change outside the tool. It merges these changes into your infrastructure code instead of just ignoring or undoing them. This way, important outside changes are never lost, and are immediately incorporated into the infrastructure code.</li> <li><strong>Prevents avoidable mistakes:</strong> It is built around a very robust, enforced schema.</li> <li><strong>Made for everyone:</strong> It welcomes all kinds of engineers, whether they are new or experienced, be they in Ops, DevOps, SRE or Platform Engineering.</li> <li><strong>Perfect for Platform Engineering:</strong> It allows platform engineers to work on the low level of detail, and developers consume reusable services by just providing a few predefined, schema-safe properties.</li> <li><strong>Built for co-existence:</strong> It is not necessary to migrate or to import anything - <strong>formae</strong> will automatically discover and update resources and happily co-exist with other IaC and Infrastructure Management tools and even ClickOps.</li> <li><strong>Perfect for Day 0 and onward:</strong> It is equally great for setting up new systems and for making small, safe changes with minimal blast radius as you go.</li> </ul> <h2>How to Use <strong>formae</strong> in Your Organization?</h2> <p>Using <strong>formae</strong> is straightforward and designed to fit various team roles, needs and daily operation scenarios and situations:</p> <ul> <li> <p><strong>Core Platform Engineers:</strong> A small group of engineers with broad responsibilities can manage the main infrastructure code. They can make large, system-wide changes. They often use GitOps (managing infrastructure through Git), but <strong>formae</strong> doesn't enforce this.</p> </li> <li> <p><strong>Engineers with Specific Roles &amp; Developers:</strong> Engineers who are newer, or those who focus on smaller parts of the system, can make small changes. Developers needing to adjust specific resources can also apply these small patches. This keeps the risk of impact (blast radius) very small.</p> </li> <li> <p><strong>On-Call Engineers (Emergency Fixes):</strong> An engineer fixing an urgent issue, even at night, works like those making small changes. No matter their experience, they focus on fixing the problem with minimal risk to other parts of the system. This helps them solve issues quickly and safely.</p> </li> <li> <p><strong>Specialized Teams (Security, Cost Optimization):</strong> Teams that work across different areas, like security or cost-saving groups, can also apply changes as patches. They might not change the core infrastructure, but their changes can affect many parts of the system. <strong>formae</strong> handles these wider-reaching but targeted changes easily.</p> </li> <li> <p><strong>Working with Other Tools:</strong> In many companies, people use different tools to make changes directly in cloud accounts. These could be special security tools or other IaC tools like Terraform, or even ClickOps. <strong>formae</strong> works well alongside them. It detects these external changes and merges them, giving you a consistent, version-controlled and always up-to-date view of your entire infrastructure - <strong>entirely as code</strong>.</p> </li> </ul> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.infoq.com/news/2025/10/iac-formae/">New Infrastructure-as-Code Tool "formae" Takes Aim at Terraform</a></li> </ul> https://tedneward.github.io/Research/tools/fuite/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/fuite/index.html <p><a href="https://github.com/nolanlawson/fuite">Sources</a> | <a href="https://nolanlawson.com/2021/12/17/introducing-fuite-a-tool-for-finding-memory-leaks-in-web-apps/">Blog post introducing it</a></p> https://tedneward.github.io/Research/tools/gaia/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/gaia/index.html <p><a href="https://gaia-pipeline.io/">Website</a> | <a href="https://github.com/gaia-pipeline/gaia">Source</a></p> <p>Use code to build pipelines (Job objects). GUI dashboard.</p> <p>Supports Go, Java, Python, C++, NodeJS, Ruby</p> <ul> <li>Programming-language independent</li> <li>First class support for tests</li> <li>Simple and efficient</li> </ul> https://tedneward.github.io/Research/tools/curl/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/curl/index.html <p><a href="https://curl.se/">Website</a> | <a href="https://everything.curl.dev/">Book: Everything Curl</a></p> https://tedneward.github.io/Research/tools/dbeaver/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dbeaver/index.html <p><a href="https://dbeaver.io/">Website</a> | <a href="https://github.com/dbeaver/dbeaver">Source</a></p> <p>Supports MySQL, PostgreSQL, SQLite, Oracle, DB2, SQL Server, Sybase, MS Access, Teradata, Firebird, Apache Hive, Phoenix, Presto, etc.</p> <p>(Written in Java, so probably supports anything with a JDBC driver.)</p> https://tedneward.github.io/Research/tools/detect-it-easy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/detect-it-easy/index.html <p><a href="https://github.com/horsicq/Detect-It-Easy">Github</a></p> https://tedneward.github.io/Research/tools/diataxis/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/diataxis/index.html <p><a href="https://diataxis.fr/">Website</a></p> <p>"It prescribes approaches to content, architecture and form that emerge from a systematic approach to understanding the needs of documentation users.</p> <p>"Diátaxis identifies four distinct needs, and four corresponding forms of documentation - tutorials, how-to guides, technical reference and explanation. It places them in a systematic relationship, and proposes that documentation should itself be organised around the structures of those needs."</p> https://tedneward.github.io/Research/tools/doctoc/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/doctoc/index.html <p><a href="https://github.com/thlorenz/doctoc">Source</a></p> <p>Links are compatible with anchors generated by github or other sites via a command line flag.</p> <p>Source repo has examples of how to use it in a git hook.</p> https://tedneward.github.io/Research/tools/drmemory/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/drmemory/index.html <p><a href="http://drmemory.org/">Website</a> | <a href="https://github.com/DynamoRIO/drmemory">Source</a></p> <ul> <li>Practical memory checking with Dr. Memory <ul> <li>Code Generation and Optimization (CGO) 2011</li> <li>Derek Bruening, Qin Zhao</li> <li><a href="https://www.researchgate.net/publication/224236404_Practical_memory_checking_with_Dr_Memory">https://www.researchgate.net/publication/224236404_Practical_memory_checking_with_Dr_Memory</a></li> <li><a href="http://www.burningcutlery.com/derek/docs/drmem-CGO11.pdf">http://www.burningcutlery.com/derek/docs/drmem-CGO11.pdf</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/dagger/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dagger/index.html <p><a href="https://dagger.io/">Website</a> | <a href="https://github.com/dagger/dagger">Source</a></p> <p>Features:</p> <ul> <li>Use <a href="/languages/cuelang">CUE</a> (a config language), not Yaml</li> <li>CUE is all about actions</li> <li> <h1>Dagger helps create custom actions in any language</h1> </li> <li> <h1>Dagger provides a huge collection of reusable actions</h1> </li> <li>Run "dagger do deploy" to test &amp; debug on your local machine</li> <li>Run Dagger pipelines on any Docker compatible runtimes</li> </ul> <p>Run sample TODO app</p> <ul> <li> <p>Clone &amp; run the following</p> <pre><code>git clone https://github.com/dagger/dagger cd dagger git checkout v0.2.4 cd pkg/universe.dagger.io/examples/todoapp dagger do build </code></pre> </li> <li> <p>You should see the output similar to below</p> <pre><code>[✔] actions.deps 62.1s [✔] actions.build.run.script 0.4s [✔] actions.test.script 0.5s [✔] client.filesystem."./".read 0.6s [✔] actions.test 2.0s [✔] actions.build.run 12.4s [✔] actions.build.contents 0.1s [✔] client.filesystem."./_build".write 0.2s </code></pre> </li> <li> <p>Run the following command to see your application up &amp; running</p> <pre><code>start _build/index.html </code></pre> </li> <li> <p>Update the application and see changes</p> <ul> <li>In the todoapp directory, edit line 25 of src/components/Form.js by updating to "What must be done today?" &amp; save the file</li> <li>Now run "dagger do build" again &amp; then start _build/index.html to see latest changes</li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/ddbug/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ddbug/index.html <p>ddbug is a utility that can extract useful information from DWARF/PDB debugging data. Its goal is to use the debugging information to provide insights into the code generation. Supports: ELF files with DWARF, Mach-O files with DWARF, Windows PDB files (mimimal)</p> <p><a href="https://github.com/gimli-rs/ddbug">Github</a></p> https://tedneward.github.io/Research/tools/devenv/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/devenv/index.html <p><a href="https://devenv.sh/">Website</a></p> https://tedneward.github.io/Research/tools/dify/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dify/index.html <p><a href="https://dify.ai/">Website</a> | <a href="https://github.com/langgenius/dify/">Source</a></p> <p>Dify is an open-source LLM app development platform. Its intuitive interface combines AI workflow, RAG pipeline, agent capabilities, model management, observability features (including <a href="https://www.comet.com/docs/opik/integrations/dify">Opik</a>, <a href="https://docs.langfuse.com/">Langfuse</a>, and <a href="https://docs.arize.com/phoenix">Arize Phoenix</a>) and more, letting you quickly go from prototype to production.</p> <h2>Quick start</h2> <blockquote> <p>Before installing Dify, make sure your machine meets the following minimum system requirements:</p> <ul> <li>CPU &gt;= 2 Core</li> <li>RAM &gt;= 4 GiB</li> </ul> </blockquote> <p>The easiest way to start the Dify server is through <a href="https://github.com/langgenius/dify/blob/main/docker/docker-compose.yaml">Docker Compose</a>. Before running Dify with the following commands, make sure that <a href="https://docs.docker.com/get-docker/">Docker</a> and <a href="https://docs.docker.com/compose/install/">Docker Compose</a> are installed on your machine:</p> <pre><code> cd dify cd docker cp .env.example .env docker compose up -d </code></pre> <p>After running, you can access the Dify dashboard in your browser at <a href="http://localhost/install">http://localhost/install</a> and start the initialization process.</p> https://tedneward.github.io/Research/tools/dog/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dog/index.html <p><a href="https://dog.ramfield.net/">Website</a> | <a href="https://github.com/Dj-Codeman/dog_community">Source</a></p> <p><code>dog-community</code> is a maintained fork of the beloved terminal DNS client (dog/doge). It keeps the human-readable output you know, adds steady fixes, and supports DNS over HTTPS/TLS with JSON output for scripting.</p> <h2>Usage</h2> <p>Basics</p> <pre><code>doge example.net # default query doge example.net MX # specific record type doge example.net MX @1.1.1.1 # specific nameserver doge example.net MX @1.1.1.1 -T # TCP instead of UDP </code></pre> <p>Arguments style</p> <pre><code>doge -q example.net -t MX -n 1.1.1.1 -T # explicit flags </code></pre> <p>JSON &amp; scripting</p> <pre><code>doge example.com AAAA -J | jq . </code></pre> <p>Protocol modes</p> <pre><code>-U/--udp # DNS over UDP -T/--tcp # DNS over TCP -S/--tls # DNS over TLS -H/--https # DNS over HTTPS </code></pre> https://tedneward.github.io/Research/tools/duneshell/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/duneshell/index.html <p><a href="https://adam-mcdaniel.github.io/dune-website/">Website</a> | <a href="https://github.com/adam-mcdaniel/dune">Source</a></p> https://tedneward.github.io/Research/tools/darke/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/darke/index.html <p><a href="https://darke.handmade.network/">Website</a> | Not sure where the files are</p> <p>Darke Files is a version control and file synchronization system. Its goal is to scale seamlessly between the two extremes:</p> <ul> <li>a full-featured version control system. It lets you work with branches and inspect and act on a rich version history. It works like version control systems programmers use today, for example Git.</li> <li>a file synchronization tool. It uploads your changed files to cloud storage so that you can forget it exists in the day-to-day operation. It works like cloud synchronization services everyone else uses, for example Dropbox or Google Drive.</li> </ul> <p>Darke Files can scale between those two extremes in a single repository. This enables teams with a wide range of requirements for such a system to work on the same project.</p> https://tedneward.github.io/Research/tools/deck/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/deck/index.html <p><a href="https://github.com/k1LoW/deck">Source</a></p> <h2>Installation</h2> <p><strong>Homebrew:</strong></p> <pre><code class="language-console">$ brew install deck </code></pre> <p><strong>go install:</strong></p> <pre><code class="language-console">$ go install github.com/k1LoW/deck/cmd/deck@latest </code></pre> <p><strong>Manual installation:</strong></p> <p>Download the binary from the <a href="https://github.com/k1LoW/deck/releases">releases page</a></p> <h2>Usage</h2> <h3>Setup</h3> <h4>Get and set your OAuth client credentials</h4> <ul> <li>Create (or reuse) a developer project at <a href="https://console.cloud.google.com">https://console.cloud.google.com</a>.</li> <li>Enable <a href="https://console.cloud.google.com/apis/library/slides.googleapis.com"><code>Google Slides API</code></a> and <a href="https://console.cloud.google.com/apis/library/drive.googleapis.com"><code>Google Drive API</code></a> at the <a href="https://console.cloud.google.com/apis/dashboard"><code>API &amp; Services</code> page</a>.</li> <li>Go to the <a href="https://console.cloud.google.com/apis/credentials"><code>Credentials</code> page</a> and click <a href="https://console.cloud.google.com/auth/clients/create"><code>+ CREATE CREDENTIALS</code></a> at the top.</li> <li>Create an <code>OAuth client ID</code> type of credentials.</li> <li>Choose the type <code>Desktop app</code>. <ul> <li>Since there is no need to publish the app, add your email address as a test user from <a href="https://console.cloud.google.com/auth/audience">Google Auth Platform / Audience</a>.</li> </ul> </li> <li>Download the credentials file to <code>${XDG_DATA_HOME:-~/.local/share}/deck/credentials.json</code>.</li> </ul> <h4>For CI/CD automation (Service Account)</h4> <p>If you're setting up <code>deck</code> for automated workflows (GitHub Actions, CI/CD pipelines), see <a href="docs/setup-service-account.md">Service Account Setup Guide</a>.</p> <h4>Check your setup with <code>deck doctor</code></h4> <p>You can verify if <code>deck</code> is ready to use and diagnose any configuration issues with the <code>deck doctor</code> command.</p> <h3>Prepare presentation ID and markdown file with <code>deck new</code></h3> <p><code>deck</code> requires two main components:<br> - <strong>Presentation ID</strong>: A unique identifier for your Google Slides presentation (e.g., <code>xxxxxXXXXxxxxxXXXXxxxxxxxxxx</code> from the URL <code>https://docs.google.com/presentation/d/xxxxxXXXXxxxxxXXXXxxxxxxxxxx/edit</code>)<br> - <strong>Markdown file</strong>: Your slide content written in markdown format</p> <h4>When creating a new presentation</h4> <p>You can create a new presentation with the <code>deck new</code> command:</p> <pre><code class="language-console">$ deck new deck.md --title "Talk about deck" Applied frontmatter to deck.md xxxxxXXXXxxxxxXXXXxxxxxxxxxx </code></pre> <p>This will create (or update) the given markdown file with frontmatter containing the presentation ID and title.</p> <h5>Reusing theme from an existing presentation</h5> <p>To reuse the theme from an existing presentation, you have two options:</p> <p><strong>Option 1: Use the <code>--base</code> flag</strong></p> <pre><code class="language-console">$ deck new deck.md --base yyyyyyyYYYYyYYYYYYYyyyyyyyyy --title "Talk about deck" xxxxxXXXXxxxxxXXXXxxxxxxxxxx </code></pre> <p><strong>Option 2: Set a default base presentation in your configuration file</strong></p> <pre><code class="language-yaml"># ~/.config/deck/config.yml basePresentationID: "yyyyyyyYYYYyYYYYYYYyyyyyyyyy" </code></pre> <p>With this configuration, you can reuse the theme from the base presentation without using the <code>--base</code> flag. If both the configuration and <code>--base</code> flag are present, the <code>--base</code> flag takes precedence.</p> <h4>When using an existing presentation</h4> <p>Get the presentation ID you want to work with. You can list all presentations with <code>deck ls</code>.</p> <pre><code class="language-console">$ deck ls xxxxxXXXXxxxxxXXXXxxxxxxxxxx My Presentation yyyyyYYYYyyyyyYYYYyyyyyyyyyy Team Project Slides </code></pre> <blockquote> <p>[!NOTE]<br> <code>deck</code> fully supports Google Shared Drives (Team Drives). Presentations stored in shared drives are automatically included in listings and can be operated on just like personal drive presentations.</p> </blockquote> <p>To use this presentation, specify it with the <code>--presentation-id</code> flag or add it to your markdown file's frontmatter as <code>presentationID</code>.</p> <h3>Write your slides in markdown</h3> <p>Edit your markdown file with your favorite editor. Among horizontal rule syntaxes, three or more consecutive hyphens at the beginning of a line (e.g. <code>---</code>) are treated as slide page separators. See <a href="#markdown-file-format-for-deck">Markdown file format for <code>deck</code></a> for details.</p> <h3>Apply markdown content to Google Slides with <code>deck apply</code></h3> <pre><code class="language-console">$ deck apply deck.md </code></pre> <h4>Watch mode</h4> <p>You can use the <code>--watch</code> flag to continuously monitor changes to your markdown file and automatically apply them to the presentation:</p> <pre><code class="language-console">$ deck apply --watch deck.md </code></pre> <p>This is useful during the content creation process as it allows you to see your changes reflected in the presentation in real-time as you edit the markdown file.</p> <blockquote> <p>[!NOTE]<br> The <code>--watch</code> flag cannot be used together with the <code>--page</code> flag.</p> </blockquote> <h3>Open presentation in your browser with <code>deck open</code></h3> <p>You can open your Google Slides presentation in your default web browser:</p> <pre><code class="language-console">$ deck open deck.md </code></pre> <h2>Markdown file format for <code>deck</code></h2> <p>The Markdown used by <code>deck</code> consists of YAML frontmatter and a body section.</p> <h3>YAML Frontmatter</h3> <p><code>deck</code> accepts YAML frontmatter at the beginning of your markdown file.</p> <pre><code class="language-markdown">--- presentationID: xxxxxXXXXxxxxxXXXXxxxxxxxxxx title: Talk about deck --- # First Slide Content... </code></pre> <p>The frontmatter must be:<br> - At the very beginning of the file<br> - Enclosed between <code>---</code> delimiters<br> - Valid YAML syntax<br> - Use <code>camelCase</code> for fields used in <code>deck</code> settings</p> <h4>Available fields</h4> <ul> <li><code>presentationID</code> (string): Google Slides presentation ID. When specified, you can use the simplified command syntax.</li> <li><code>title</code> (string): The title of the presentation. When specified, you can use the simplified command syntax.</li> <li><code>breaks</code> (boolean): Control how line breaks are rendered. Default (<code>false</code> or omitted) renders line breaks as spaces. When <code>true</code>, line breaks in markdown are rendered as actual line breaks in slides. Can also be configured globally in <code>config.yml</code>.</li> <li><code>codeBlockToImageCommand</code> (string): Command to convert code blocks to images. When specified, code blocks in the presentation will be converted to images using this command. Can also be configured globally in <code>config.yml</code>.</li> <li><code>defaults</code> (array): Define conditional actions using CEL (Common Expression Language) expressions. Actions are automatically applied to pages based on page structure and content. Only applies to pages without explicit page configuration. Can also be configured globally in <code>config.yml</code>.</li> </ul> <h3>Supported Markdown syntax</h3> <p><code>deck</code> supports CommonMark and selected GitHub Flavored Markdown extensions. For comprehensive documentation, see <a href="docs/markdown.md">Markdown Support Documentation</a>.</p> <p><strong>Key supported features:</strong><br> - Bold ( <code>**bold**</code> )<br> - Italic ( <code>*italic*</code> <code>__italic__</code> )<br> - Strikethrough ( <code>~~strikethrough~~</code> )<br> - List ( <code>-</code> <code>*</code> )<br> - Ordered list ( <code>1.</code> <code>1)</code> )<br> - Link ( <code>[Link](https://example.com)</code> )<br> - Angle bracket autolinks ( <code>&lt;https://example.com&gt;</code> )<br> - Code ( <code>`code`</code> )<br> - <code>&lt;br&gt;</code> (for newline)<br> - Image (<code></code> )<br> - Block quote ( <code>&gt; block quote</code> )<br> - Table (GitHub Flavored Markdown tables)<br> - RAW inline HTML (e.g., <code>&lt;mark&gt;</code>, <code>&lt;small&gt;</code>, <code>&lt;kbd&gt;</code>, <code>&lt;cite&gt;</code>, <code>&lt;q&gt;</code>, <code>&lt;span&gt;</code>, <code>&lt;u&gt;</code>, <code>&lt;s&gt;</code>, <code>&lt;del&gt;</code>, <code>&lt;ins&gt;</code>, <code>&lt;sub&gt;</code>, <code>&lt;sup&gt;</code>, <code>&lt;var&gt;</code>, <code>&lt;samp&gt;</code>, <code>&lt;data&gt;</code>, <code>&lt;dfn&gt;</code>, <code>&lt;time&gt;</code>, <code>&lt;abbr&gt;</code>)</p> <h4>Line break handling</h4> <p><code>deck</code> provides configurable line break behavior through the <code>breaks</code> setting:</p> <ul> <li><strong>Default (<code>breaks: false</code>)</strong>: Single line breaks become spaces (per CommonMark/GFM specs)</li> <li><strong>With <code>breaks: true</code></strong>: Line breaks are preserved (GitHub-style rendering)</li> <li><strong>For explicit breaks</strong>: Use hard line break syntax (two spaces at line end per CommonMark standard) or <code>&lt;br&gt;</code> tag</li> </ul> <p>Example with <code>breaks: true</code>:</p> <pre><code class="language-markdown">--- breaks: true --- Text with line breaks preserved </code></pre> <h4>Comments</h4> <p>HTML comments <code>&lt;!--</code> <code>--&gt;</code> are used for speaker notes or <a href="#page-configuration">page configuration</a>.</p> <h2>How markdown maps to slide placeholders</h2> <p><code>deck</code> inserts values according to the following rules regardless of the slide layout.</p> <ul> <li>The shallowest heading level within each slide content is treated as the title and inserted into the title placeholder ( <code>CENTERED_TITLE</code> or <code>TITLE</code> ) in order.</li> <li>In most cases, this will be H1 (<code>#</code>), which is the standard for slide titles</li> <li>The next heading level (minimum level + 1) is treated as the subtitle and inserted into the subtitle placeholder ( <code>SUBTITLE</code> ) in order.</li> <li>When H1 is used as the title, H2 (<code>##</code>) becomes the subtitle</li> <li>All other items are inserted into the body placeholder ( <code>BODY</code> ) in order. <ul> <li>The remaining contents are divided into one or more bodies by headings corresponding to the title or subtitle in the slide.</li> </ul> </li> </ul> <p>For example:<br> - <strong>Standard case</strong>: If a slide contains <code>#</code> (H1), then <code>#</code> becomes the title and <code>##</code> becomes the subtitle<br> - <strong>Alternative case</strong>: If a slide only contains <code>##</code> (H2) or deeper, then <code>##</code> becomes the title and <code>###</code> becomes the subtitle</p> <blockquote> <p>[!NOTE]<br> They are inserted in the order they appear in the markdown document, <strong>from the placeholder at the top of the slide</strong> (or from the placeholder on the left if placeholders are at the same height).</p> <p>Also, if there are not enough placeholders, the remaining contents will not be rendered.</p> </blockquote> <h3>Example</h3> <p><strong>Input markdown document:</strong></p> <pre><code class="language-markdown"># CAP theorem ## In Database theory ## Consistency Every read receives the most recent write or an error. ## Availability Every request received by a non-failing node in the system must result in a response. ## Partition tolerance The system continues to operate despite an arbitrary number of messages being dropped (or delayed) by the network between nodes. </code></pre> <p><strong>Layout and placeholders:</strong></p> <p><img src="https://tedneward.github.io/Research/tools/img/layout.png" alt="img"></p> <p><strong>Result of applying:</strong></p> <p><img src="https://tedneward.github.io/Research/tools/img/result.png" alt="img"></p> <h2>Configuration File</h2> <p><code>deck</code> supports global configuration files that provide default settings for all presentations. Configuration files are loaded in the following order:</p> <ol> <li><code>${XDG_CONFIG_HOME:-~/.config}/deck/config-{profile}.yml</code> (when using <code>--profile</code> option)</li> <li><code>${XDG_CONFIG_HOME:-~/.config}/deck/config.yml</code> (default config file)</li> </ol> <p>The configuration file uses YAML format and supports the same fields as frontmatter. Settings in frontmatter take precedence over configuration file settings, which in turn take precedence over built-in defaults.</p> <h3>Configuration file example</h3> <pre><code class="language-yaml"># Global configuration for deck basePresentationID: "1wIik04tlp1U4SBHTLrSu20dPFlAGTbRHxnqdRFF9nPo" breaks: true codeBlockToImageCommand: "go run testdata/txt2img/main.go" folderID: "1aBcDeFgHiJkLmNoPqRsTuVwXyZ" defaults: # First page should always use title layout - if: page == 1 layout: title # Pages with only one title and one H2 heading use section layout - if: titles.size() == 1 &amp;&amp; headings[2].size() == 1 layout: section-purple # Skip pages with TODO in speaker notes - if: speakerNote.contains("TODO") skip: true # Default layout for all other pages - if: true layout: title-and-body </code></pre> <h3>Available configuration fields</h3> <ul> <li><strong><code>basePresentationID</code></strong> (string): Base presentation ID to use as a template when creating new presentations</li> <li><strong><code>breaks</code></strong> (boolean): Global line break rendering behavior</li> <li><strong><code>codeBlockToImageCommand</code></strong> (string): Global command to convert code blocks to images</li> <li><strong><code>folderID</code></strong> (string): Default folder ID to create presentations and upload temporary images to</li> <li><strong><code>defaults</code></strong> (array): A series of conditions and actions written in CEL expressions for default page configs</li> </ul> <h3>Configuration precedence</h3> <p>Settings are applied in the following order (highest to lowest priority):</p> <ol> <li><strong>Command-line options</strong> - Takes highest precedence when available</li> <li><strong>Frontmatter settings</strong> - Takes precedence over config file</li> <li><strong>Configuration file settings</strong> - Applied when neither command-line nor frontmatter specify the setting</li> <li><strong>Built-in defaults</strong> - Used when no other source specifies a setting</li> </ol> <p>This allows you to set organization-wide or project-wide defaults while still maintaining the flexibility to override them on a per-file basis using frontmatter or command-line options.</p> <h2>Advanced features</h2> <h3>Style for syntax</h3> <p>Create a layout named <code>style</code> and add a <code>Text box</code> to enter specific words. The styles (<code>bold</code>, <code>italic</code>, <code>underline</code>, <code>backgroundColor</code>, <code>foregroundColor</code>, <code>fontFamily</code>) will be applied as the style for each Markdown syntax.</p> <p><img src="https://tedneward.github.io/Research/tools/img/style.png" alt="img"></p> <table> <thead> <tr> <th> Word </th> <th> </th> </tr> </thead> <tbody> <tr> <td> <code>bold</code> </td> <td> style for <strong>bold</strong>. </td> </tr> <tr> <td> <code>italic</code> </td> <td> style for <em>italic</em>. </td> </tr> <tr> <td> <code>link</code> </td> <td> style for <a href="#">link</a>. </td> </tr> <tr> <td> <code>code</code> </td> <td> style for <code>code</code>. </td> </tr> <tr> <td> <code>del</code> </td> <td> style for ~~strikethrough~~ (also applies to <code>&lt;del&gt;</code> tag). </td> </tr> <tr> <td> <code>blockquote</code> </td> <td> style for block quote. </td> </tr> <tr> <td> HTML element names </td> <td> style for content of inline HTML elements ( e.g. <code>&lt;cite&gt;</code>, <code>&lt;q&gt;</code>, <code>&lt;s&gt;</code>, <code>&lt;ins&gt;</code>, etc. ) </td> </tr> <tr> <td> (other word) </td> <td> style for content of inline HTML elements with matching class name ( e.g. <code>&lt;span class="notice"&gt;THIS IS NOTICE&lt;/span&gt;</code> ) </td> </tr> </tbody> </table> <h3>Code blocks to images</h3> <p>You can convert <a href="testdata/codeblock.md">Markdown code blocks</a> to images by specifying a command that outputs image data (PNG, JPEG, GIF) to standard output or to a file by using the <code>{{output}}</code> placeholder for the output file path.</p> <pre><code class="language-console">$ deck apply --code-block-to-image-command "some-command" -i xxxxxXXXXxxxxxXXXXxxxxxxxxxx deck.md </code></pre> <p>Alternatively, you can configure it in <code>config.yml</code> or frontmatter:</p> <pre><code class="language-yaml">codeBlockToImageCommand: "some-command" </code></pre> <h4>How to receive values</h4> <p>From code blocks like the following, you can obtain the optional language identifier <code>go</code> and the content within the code block.</p> <pre><code>```go package main import "fmt" func main() { fmt.Println("Hello, 世界") } ``` </code></pre> <p>There are three ways to receive code block information within the command:</p> <ol> <li><strong>Receive from standard input</strong></li> </ol> <ul> <li>The content of the code block is passed as standard input</li> <li>The optional language identifier cannot be obtained, so use it in combination with other methods</li> </ul> <ol> <li><strong>Receive as environment variables</strong></li> </ol> <ul> <li><code>CODEBLOCK_LANG</code>: Optional language identifier of the code block (e.g., <code>go</code>, <code>python</code>)</li> <li><code>CODEBLOCK_CONTENT</code>: Content of the code block</li> <li><code>CODEBLOCK_OUTPUT</code>: Path to a temporary output file</li> </ul> <ol> <li><strong>Receive with CEL template syntax</strong></li> </ol> <ul> <li><code>{{lang}}</code>: Optional language identifier of the code block</li> <li><code>{{content}}</code>: Content of the code block</li> <li><code>{{output}}</code>: Path to a temporary output file</li> <li><code>{{env.XXX}}</code>: Value of environment variable XXX</li> </ul> <p>The template expansion uses CEL (Common Expression Language) for evaluating expressions within <code>{{ }}</code> delimiters. This supports:<br> - Ternary operators: <code>{{ lang == "" ? "md" : lang }}</code><br> - String concatenation: <code>{{ "prefix_" + lang }}</code><br> - Boolean logic: <code>{{ lang != "" &amp;&amp; content.contains("main") }}</code><br> - Arithmetic operations: <code>{{ count + 1 }}</code></p> <p>These methods can be used in combination, and you can choose the appropriate method according to the command requirements.</p> <blockquote> <p>[!NOTE]<br> When <code>{{output}}</code> is not specified, <code>deck</code> reads the image data from the command's stdout. When <code>{{output}}</code> is specified, the command should write the image to that file path, and <code>deck</code> will read the image data from that file.</p> </blockquote> <h4>Examples</h4> <pre><code class="language-console"># Convert Mermaid diagrams to images $ deck apply -c 'mmdc -i - -o {{output}} --quiet' deck.md </code></pre> <pre><code class="language-console"># Generate code images using the built-in text-to-image tool $ deck apply -c 'go run testdata/txt2img/main.go' deck.md </code></pre> <pre><code class="language-console"># Use different tools depending on the language $ deck apply -c 'if [ {{lang}} = "mermaid" ]; then mmdc -i - -o {{output}} --quiet; else go run testdata/txt2img/main.go; fi' deck.md # Alternatively, you can use Songmu/laminate to use the appropriate tool for each language. $ deck apply -c 'laminate' deck.md </code></pre> <h2>Page configuration</h2> <p>You can configure individual pages using JSON comments. Available settings:</p> <ul> <li><strong><code>"layout"</code></strong>: Specifies which slide layout to use from your presentation template. Different layouts have different placeholder arrangements (title, subtitle, body, etc.)</li> <li><strong><code>"freeze"</code></strong>: Prevents <code>deck</code> from modifying the page (useful for slides with completed designs)</li> <li><strong><code>"ignore"</code></strong>: Excludes the page from slide generation (for drafts, notes, or unused content)</li> <li><strong><code>"skip"</code></strong>: Creates the slide but skips it during presentation playback (automatically advances to next slide)</li> </ul> <pre><code class="language-markdown">&lt;!-- {"layout": "title-and-body"} --&gt; # Your slide content --- &lt;!-- {"freeze": true} --&gt; # This slide won't be modified --- &lt;!-- {"ignore": true} --&gt; # This content won't appear in slides --- &lt;!-- {"skip": true} --&gt; # This slide will be skipped during presentation </code></pre> <blockquote> <p>[!TIP]<br> Use <code>deck ls-layouts</code> to see all available layout names for your presentation:</p> <pre><code class="language-console">$ deck ls-layouts deck.md title section section-dark title-and-body title-and-body-half title-and-body-2col title-and-body-3col </code></pre> </blockquote> <p><img src="https://tedneward.github.io/Research/tools/img/layout_name.png" alt="img"></p> <h2>Default page configs with CEL expressions</h2> <p>The <code>defaults</code> field in Frontmatter or configuration file allows you to define default page configs using CEL (Common Expression Language) expressions. This feature automatically sets layouts and controls page behavior based on their structure and content, eliminating the need for manual configuration on each page.</p> <h3>Available actions</h3> <p>The following actions can be applied to pages through the <code>defaults</code> configuration:</p> <ul> <li><strong><code>layout</code></strong>: Set layout automatically</li> <li><strong><code>freeze</code></strong>: Freeze page from modifications</li> <li><strong><code>ignore</code></strong>: Exclude page from generation</li> <li><strong><code>skip</code></strong>: Skip page during presentation</li> </ul> <pre><code class="language-yaml">--- defaults: # First page should always use title layout - if: page == 1 layout: title # Pages with only one title and one H2 heading use section layout - if: titles.size() == 1 &amp;&amp; headings[2].size() == 1 layout: section-purple # Skip pages with TODO in speaker notes - if: speakerNote.contains("TODO") skip: true # Default layout for all other pages - if: true layout: title-and-body --- </code></pre> <h3>Available CEL variables</h3> <table> <thead> <tr> <th> Variable </th> <th> Type </th> <th> Description </th> </tr> </thead> <tbody> <tr> <td> <code>page</code> </td> <td> <code>int</code> </td> <td> Current page number (1-based) </td> </tr> <tr> <td> <code>pageTotal</code> </td> <td> <code>int</code> </td> <td> Total number of pages </td> </tr> <tr> <td> <code>titles</code> </td> <td> <code>[]string</code> </td> <td> List of titles in the page </td> </tr> <tr> <td> <code>subtitles</code> </td> <td> <code>[]string</code> </td> <td> List of subtitles in the page </td> </tr> <tr> <td> <code>bodies</code> </td> <td> <code>[]string</code> </td> <td> List of body texts in the page </td> </tr> <tr> <td> <code>blockQuotes</code> </td> <td> <code>[]string</code> </td> <td> List of block quotes in the page </td> </tr> <tr> <td> <code>codeBlocks</code> </td> <td> <code>[]CodeBlock</code> </td> <td> List of code blocks in the page </td> </tr> <tr> <td> <code>images</code> </td> <td> <code>[]Image</code> </td> <td> List of images in the page </td> </tr> <tr> <td> <code>comments</code> </td> <td> <code>[]string</code> </td> <td> List of comments in the page </td> </tr> <tr> <td> <code>headings</code> </td> <td> <code>map[int][]string</code> </td> <td> Headings grouped by level </td> </tr> <tr> <td> <code>speakerNote</code> </td> <td> <code>string</code> </td> <td> Speaker note </td> </tr> <tr> <td> <code>topHeadingLevel</code> </td> <td> <code>int</code> </td> <td> The highest heading level in the content </td> </tr> </tbody> </table> <h3>CEL condition examples</h3> <ul> <li><code>page == 1</code> - First page only</li> <li><code>titles.size() == 0</code> - Pages without titles</li> <li><code>codeBlocks.size() &gt; 0</code> - Pages containing code blocks</li> <li><code>headings[3].size() &gt;= 2</code> - Pages with 2 or more H3 headings</li> <li><code>bodies[0].contains("TODO")</code> - Pages with TODO in first body text</li> <li><code>page &gt; pageTotal - 3</code> - Last 3 pages</li> <li><code>images.size() &gt;= 2</code> - Pages with 2 or more images</li> </ul> <h3>Important notes</h3> <ul> <li><strong>Evaluation order</strong>: Conditions are evaluated in order, and the first matching condition's action is applied</li> <li><strong>Priority</strong>: Default actions only apply to pages without explicit page configuration (via JSON comments like <code>&lt;!-- {"layout": "title"} --&gt;</code>)</li> <li><strong>Performance</strong>: Using <code>ignore</code> for unnecessary content improves processing speed</li> <li><strong>Workflow</strong>: This feature enables automatic page management based on content patterns, reducing manual configuration overhead</li> </ul> <h2>Profile support</h2> <p><code>deck</code> supports multiple profiles through the <code>--profile</code> option. This feature allows you to manage separate profiles (authentication Google accounts or environments).</p> <pre><code class="language-console">$ deck apply deck.md --profile work $ deck ls --profile personal $ deck new presentation.md --profile project-a </code></pre> <p>When using profiles, authentication files are managed as follows:<br> - <strong>Credentials file</strong>: <code>credentials-{profile}.json</code> - Create this file manually to use profile-specific credentials. If this file exists, it will be automatically used for the specified profile.<br> - <strong>Token file</strong>: <code>token-{profile}.json</code> - This file is automatically generated when you use the <code>--profile</code> option and complete the OAuth authentication process.</p> <h2>FAQ</h2> <h3>A setting permission error occurs during image upload, as shown below</h3> <pre><code>Error: failed to apply page: failed to upload image: failed to set permission for image: googleapi: Error 403: The user does not have sufficient permissions for this file., insufficientFilePermissions </code></pre> <p>Please verify that you can grant public read access to files in Google Drive. Organizational policies may prevent this. If restricted, create a folder with appropriate permissions and specify its ID using the <code>--folder-id</code> flag during <code>deck apply</code>.</p> <p>To insert images into slides, <code>deck</code> temporarily uploads image files to Google Drive, obtains a publicly accessible URL from there, and passes it to the API. Therefore, you must be able to grant reader permissions to anyone for image files on Google Drive.</p> <h2>Integration</h2> <ul> <li><a href="https://github.com/zonuexe/deck-slides.el">zonuexe/deck-slides.el</a> ... Emacs integration for creating presentations using Markdown and Google Slides</li> <li><a href="https://github.com/Songmu/laminate">Songmu/laminate</a> ... Selects image generation commands based on code block language. Useful for converting code blocks to images</li> </ul> <h2>With AI agent</h2> <p>By collaborating with AI agents to create Markdown-formatted slides, you may be able to create effective presentations.</p> <details> <summary>It is a good idea to provide the following rules for creating presentations with `deck` in the prompt. (Click to expand)</summary> <pre><code>Create a presentation in Markdown according to the following rules. # Rules for describing presentations using Markdown Unless otherwise specified, please follow the rules below. ## Basic Structure - Use a line containing only three or more consecutive hyphens (`---`, `----`, etc.) from the beginning to the end of the line to indicate page breaks between slides. - Other horizontal rule elements (like `- - -`, `***`, `___`) remain in the content as visual separators and can be used to separate multiple body placeholders. - Within each slide, the minimum heading level will be treated as the title, and the next level as the subtitle. Higher level headings will be treated as body content. It is recommended to use only one title heading per slide. ## YAML Frontmatter You can include YAML frontmatter at the beginning of the file: ```yaml --- title: "Presentation Title" presentationID: "presentation_id" breaks: true author: "Author Name" date: "2024-01-01" tags: ["tag1", "tag2"] custom: nested: "value" number: 42 --- ``` ## Supported Markdown Syntax The following syntax can be used in the slide content: ### Text Formatting - **Bold** (`**bold**`) - *Italic* (`*italic*` or `__italic__`) - `Inline code` (&lt;code&gt;\`code\`&lt;/code&gt;) - Combined formatting (e.g., ***bold and italic***) ### Lists - Bullet lists (`-` or `*`) - Numbered lists (`1.` or `1)`) - Nested lists (with proper indentation) - Alphabetical lists (a. b. c.) ### Links and Images - Links (`[Link text](https://example.com)`) - Angle bracket autolinks (`&lt;https://example.com&gt;`) - Images (``) - Supports PNG, JPEG, GIF formats - Supports both local files and URLs (HTTP/HTTPS) ### Block Elements - Block quotes (`&gt; quoted text`) - Nested block quotes - Code blocks with language specification: ```language code content ``` - Mermaid diagrams (in code blocks with `mermaid` language) ### Tables - GitHub Flavored Markdown (GFM) tables - Supports table headers with automatic bold formatting - Cell content can include inline formatting (bold, italic, code) - Example: ```markdown | Header 1 | Header 2 | Header 3 | |----------|----------|----------| | Cell 1 | **Bold** | `code` | | Cell 2 | *Italic* | Normal | ``` - Header rows are automatically styled with bold text and gray background - Tables created by users in Google Slides are preserved ### HTML Elements You can use the following HTML inline elements: - `&lt;strong&gt;`, `&lt;em&gt;`, `&lt;b&gt;`, `&lt;i&gt;`, `&lt;mark&gt;`, `&lt;small&gt;` - `&lt;code&gt;`, `&lt;kbd&gt;`, `&lt;cite&gt;`, `&lt;q&gt;`, `&lt;ruby&gt;`, `&lt;rt&gt;` - `&lt;span&gt;`, `&lt;u&gt;`, `&lt;s&gt;`, `&lt;sub&gt;`, `&lt;sup&gt;`, `&lt;var&gt;` - `&lt;samp&gt;`, `&lt;data&gt;`, `&lt;dfn&gt;`, `&lt;time&gt;`, `&lt;abbr&gt;`, `&lt;rp&gt;` - `&lt;br&gt;` (for line breaks) - Use `class` attribute for custom styling ### Line Break Handling - Default (`breaks: false`): Soft line breaks become spaces - With `breaks: true`: Soft line breaks become actual line breaks - Use `&lt;br&gt;` tags for explicit line breaks ## Page Configuration Use HTML comments for page settings and speaker notes: - Page settings: `&lt;!-- {"layout": "title-and-body"} --&gt;` - Available settings: `"freeze": true`, `"ignore": true`, `"skip": true` - Speaker notes: `&lt;!-- This is a speaker note --&gt;` (use separate comments for notes) ## Important Notes - If a comment (`&lt;!-- --&gt;`) contains JSON, it's a page setting - do not overwrite it - If `"freeze": true` is present in page settings, do not modify that page content at all - Write speaker notes in separate comments, not in JSON configuration comments - Code blocks can be converted to images using the `--code-block-to-image-command` option </code></pre> </details> <h2>Alternatives</h2> <ul> <li><a href="https://github.com/googleworkspace/md2googleslides">googleworkspace/md2googleslides</a>: Generates Google Slides from markdown</li> </ul> https://tedneward.github.io/Research/tools/devknife/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/devknife/index.html <p><a href="https://devknife.app/">Website</a> | Commercial</p> <p>JSON Formatter &amp; Editor | JSON/YAML/TOML/XML/CSV Converter | Regex Tester | Text Compare | JWT Decoder | Port Scanner | Time Inspector | Index Now | Markdown Editor | Encoder &amp; Decoder | Domain &amp; IP WHOIS | Network Speed Test | Color Picker | Data Generator | IP Location | Hasher | Text Inspector | SVG Editor | Password Generator | Tailwind Colors</p> https://tedneward.github.io/Research/tools/dnspy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dnspy/index.html <p><a href="https://github.com/dnSpy/dnSpy">Source</a></p> https://tedneward.github.io/Research/tools/dotnetreleaser/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dotnetreleaser/index.html <p><a href="https://github.com/xoofx/dotnet-releaser">Source</a></p> https://tedneward.github.io/Research/tools/dynamorio/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/dynamorio/index.html <p><a href="https://dynamorio.org">Website</a> | <a href="https://github.com/DynamoRIO/dynamorio">Source</a></p> <p><a href="https://github.com/lgeek/dynamorio_pin_escape">dynamorio_pin_escape</a>: Escaping DynamoRIO and Pin - or why it's a worse-than-you-think idea to run untrusted code or to input untrusted data<br> - Cosmin Gorgovan (2014)</p> https://tedneward.github.io/Research/tools/daytona/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/daytona/index.html <p><a href="https://www.daytona.io/">Website</a> | <a href="https://github.com/daytonaio/daytona">Source</a></p> https://tedneward.github.io/Research/tools/delinker/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/delinker/index.html <p><a href="https://github.com/jnider/delinker">Github</a></p> <p>Reverse the Linking Process<br> - Compiler, Architecture, and Tools Conference (CATC) 2018<br> - Joel Nider<br> - <a href="https://software.intel.com/en-us/download/reverse-the-linking-process">https://software.intel.com/en-us/download/reverse-the-linking-process</a></p> https://tedneward.github.io/Research/tools/diagrams/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/diagrams/index.html <p><a href="https://diagrams.mingrammer.com/">Website</a> | <a href="https://github.com/mingrammer/diagrams">Source</a></p> https://tedneward.github.io/Research/tools/docmd/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/docmd/index.html <p><a href="https://docmd.mgks.dev/">Website</a> | <a href="https://github.com/mgks/docmd">Source</a></p> https://tedneward.github.io/Research/tools/drawio/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/drawio/index.html <p><a href="https://drawio-app.com/">Website</a></p> <hr> <p><a href="https://drawio-app.com/github-support/">Integrating Draw.io into Github repository</a></p> https://tedneward.github.io/Research/tools/easy2boot/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/easy2boot/index.html <p><a href="https://easy2boot.xyz/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.xda-developers.com/multi-boot-usb-drive-for-almost-any-os-new-or-old/">I made a multi-boot USB drive for almost any OS, new or old</a></li> </ul> https://tedneward.github.io/Research/tools/crap4j/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/crap4j/index.html <p><a href="http://www.crap4j.org/">Website</a> | <a href="https://code.google.com/archive/p/crap4j/">Source</a></p> <p><strong>Q:</strong> What Are Change Risk Anti Patterns?</p> <p><strong>A:</strong> They are code patterns that are associated with an increased risk of breaking an existing piece of code when it’s being modified.</p> <p><strong>Q:</strong> What are some examples of Change Risk Anti-Patterns?</p> <p><strong>A:</strong> A good example of a Java change risk anti-pattern, and one used in CRAP, is a method with high cyclomatic complexity (i.e. a lot of branches). Several studies show a definite correlation between excessive code complexity and an increased probability of introducing defects during maintenance. Another change risk anti-pattern used in CRAP is lack of automated tests. Modifying existing code without having some tests you can run to ensure that the changes have not introduced regressions is, not surprisingly, quite risky – especially if the person modifying the code is not the original developer.</p> <p><strong>Q:</strong> How is CRAP calculated?</p> <p><strong>A:</strong> The current version of CRAP combines the two change risk anti-patterns we just discussed: excessive method complexity and lack of automated tests for those methods.</p> <p>Given a Java method m, CRAP for m is calculated as follows:</p> <pre><code>CRAP(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m) </code></pre> <p>Where comp(m) is the cyclomatic complexity of method m, and cov(m) is the test code coverage provided by automated tests (e.g. JUnit tests, not manual QA). Cyclomatic complexity is a well-known and widely used metric and it’s calculated as one plus the number of unique decisions in the method. For code coverage we use basis path coverage. Low CRAP numbers indicate code with relatively low change and maintenance risk – because it’s not too complex and/or it’s well-protected by automated and repeatable tests. High CRAP numbers indicate code that’s risky to change because of a hazardous combination of high complexity and low, or no, automated test coverage.</p> <p>Generally speaking, you can lower your CRAP score either by adding automated tests or by refactoring to reduce complexity. Preferably both; and it’s a good idea to write the tests firsts so you can refactor more safely.</p> <p><strong>Q:</strong> You said “current version of CRAP”, does that mean that CRAP might change?</p> <p><strong>A:</strong> Yes, we believe that the current version of CRAP is a very good start and it has proven useful and rather accurate in detecting risky code. But we also believe that metrics should be open to change and evolve as we gain experience using them and to keep up with changes in programming practices.</p> <p><strong>Q:</strong> I understand how CRAP is calculated for a given method, but how do I interpret the CRAP number I get?</p> <p><strong>A:</strong> Bob Evans and I have looked at a lot of examples (using our code and many open source projects) and listened to a LOT of opinions. After much debate, we decided to initially use a CRAP score of 30 as the threshold for crappiness. Below is a table that shows the amount of test coverage required to stay below the CRAP threshold based on the complexity of a method:</p> <p>Complexity vs Coverage<br> Method's Cyclomatic Complexity | % of coverage required to be below CRAPpy threshold<br> ------------------------------ | ---------------------------------------------------<br> 0–5 | 0%<br> 6-10 | 42%<br> 11-15 | 57%<br> 16-20 | 71%<br> 21-25 | 80%<br> 26-30 | 100%<br> 31+ | Time to refactor</p> <p><strong>Q:</strong> What Are Change Risk Anti Patterns?</p> <p><strong>A:</strong> They are code patterns that are associated with an increased risk of breaking an existing piece of code when it’s being modified.</p> <p><strong>Q:</strong> What are some examples of Change Risk Anti-Patterns?</p> <p><strong>A:</strong> A good example of a Java change risk anti-pattern, and one used in CRAP, is a method with high cyclomatic complexity (i.e. a lot of branches). Several studies show a definite correlation between excessive code complexity and an increased probability of introducing defects during maintenance. Another change risk anti-pattern used in CRAP is lack of automated tests. Modifying existing code without having some tests you can run to ensure that the changes have not introduced regressions is, not surprisingly, quite risky – especially if the person modifying the code is not the original developer.</p> <p><strong>Q:</strong> How is CRAP calculated?</p> <p><strong>A:</strong> The current version of CRAP combines the two change risk anti-patterns we just discussed: excessive method complexity and lack of automated tests for those methods.</p> <p>Given a Java method m, CRAP for m is calculated as follows:</p> <pre><code>CRAP(m) = comp(m)^2 * (1 – cov(m)/100)^3 + comp(m) </code></pre> <p>Where comp(m) is the cyclomatic complexity of method m, and cov(m) is the test code coverage provided by automated tests (e.g. JUnit tests, not manual QA). Cyclomatic complexity is a well-known and widely used metric and it’s calculated as one plus the number of unique decisions in the method. For code coverage we use basis path coverage. Low CRAP numbers indicate code with relatively low change and maintenance risk – because it’s not too complex and/or it’s well-protected by automated and repeatable tests. High CRAP numbers indicate code that’s risky to change because of a hazardous combination of high complexity and low, or no, automated test coverage.</p> <p>Generally speaking, you can lower your CRAP score either by adding automated tests or by refactoring to reduce complexity. Preferably both; and it’s a good idea to write the tests firsts so you can refactor more safely.</p> <p><strong>Q:</strong> You said “current version of CRAP”, does that mean that CRAP might change?</p> <p><strong>A:</strong> Yes, we believe that the current version of CRAP is a very good start and it has proven useful and rather accurate in detecting risky code. But we also believe that metrics should be open to change and evolve as we gain experience using them and to keep up with changes in programming practices.</p> <p><strong>Q:</strong> I understand how CRAP is calculated for a given method, but how do I interpret the CRAP number I get?</p> <p><strong>A:</strong> Bob Evans and I have looked at a lot of examples (using our code and many open source projects) and listened to a LOT of opinions. After much debate, we decided to initially use a CRAP score of 30 as the threshold for crappiness. Below is a table that shows the amount of test coverage required to stay below the CRAP threshold based on the complexity of a method:</p> <p>In other words, you can have high-complexity methods BUT you better have a lot of tests for them. Please note that we don’t recommend having zero tests – even for simple methods of complexity 5 and below. Every piece of code that does something non-trivial should have some tests but – after many discussions – we believe that the CRAP metric will be more useful if it highlights the truly crappy and risky code, not just the code that could be made better by refactoring or adding some tests</p> <p><strong>Q:</strong> OK. Crap4J has labeled one of my methods as crappy, what do I do now?</p> <p><strong>A:</strong> First of all, take a look at the CRAP details page in the report (the link is at the bottom of the report). The CRAP page shows all methods sorted by CRAP (from highest to lowest) and displays both complexity and test coverage information.</p> <p>If a method is crappy because it has acceptable complexity (e.g. 8) but inadequate tests (e.g. 0%), the obvious thing is to write some tests for it to increase coverage.</p> <p>If the method is well tested by high-complexity, it’s a good idea to refactor. Method extraction is a very safe refactoring and can really improve code legibility if you give the extracted method a name that describes what it does (e.g. validateCardNumber(String num) is a good method extraction name, check(String num) is not.)</p> <p><strong>Q:</strong> What is the CRAPload number?</p> <p><strong>A:</strong> Crap load represents the "minimum" amount of work that could be done to get below the crap threshold of 30 for a method. The summary Crapload number is for the whole project and is just the sum of crap load for all methods. The idea is that this gives you a floor value for the amount of work to get rid of crap. Usually, the amount of work will be greater.</p> <p>Examining this in more detail, to lower the Crap score, there are two options: reduce cyclomatic complexity in a method, or write tests that cover more of the paths in the method. In the case of the Crapload number, we take the simplest possible restructuring of code, which is to refactor it into methods that are half the size of the previous method, say using the Extract Method refactoring. For example, I have a method of complexity 60. if I refactor by extracting half the method into a new method, then I would have two methods of complexity 30.</p> <p>Obviously, this blind refactoring is a pretty poor way to "fix" a system, and in reality, some other, more appropriate, restructuring of the system would be the action to take. In lieu of being able to determine that programmatically, we can at least compute the minimum amount of work that could possibly be done to remove Crap from the system.</p> <p>Crap load for a method is calculated as follows:</p> <pre><code> public int getCrapLoad(float crapThreshold) { int crapLoad = 0; if (getCrap() &gt;= crapThreshold) { int complexity = getComplexity(); float coverage = getCoverage(); crapLoad += complexity * (1.0 - coverage); crapLoad += complexity / crapThreshold; } return crapLoad; } </code></pre> <p>So, interpreting that, if the CRAP score for a method is above the threshold, 30, then for every point of uncovered complexity, add 1 for a test to cover that path. Then for every bit of complexity over the threshold, figure out the number of extract methods dividing in half that need to be done to get below the threshold.</p> https://tedneward.github.io/Research/tools/8mblocal/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/8mblocal/index.html <p><a href="https://8mb.campuscal.tech/">Website</a> | <a href="https://github.com/JMS1717/8mb.local">Source</a></p> <p>8mb.local is a self‑hosted, fire‑and‑forget video compressor. Drop a file, choose a target size (e.g., 8MB, 25MB, 50MB, 100MB), and let GPU-accelerated encoding produce compact outputs with AV1/HEVC/H.264. Supports NVIDIA NVENC, Intel/AMD VAAPI (Linux), and CPU fallback. The stack includes a SvelteKit UI, FastAPI backend, Celery worker, Redis broker, and real‑time progress via Server‑Sent Events (SSE).</p> <h2>Features</h2> <ul> <li>Multi-vendor GPU support: Auto-detects NVIDIA NVENC, Intel/AMD VAAPI (Linux), or falls back to CPU</li> <li>Robust encoder validation: Tests actual encoder initialization, not just availability listing</li> <li>Automatic CPU fallback: Gracefully handles missing drivers, permission issues, or hardware access problems</li> <li>Drag‑and‑drop UI with helpful presets and advanced options (codec, container, tune, audio bitrate)</li> <li>Configurable codec visibility: Enable/disable specific codecs in Settings page</li> <li>Resolution control: Set max width/height while maintaining aspect ratio</li> <li>Video trimming: Specify start/end times (seconds or HH:MM:SS format)</li> <li>ffprobe analysis on upload for instant estimates and warnings</li> <li>Real‑time progress tracking: Multi-signal progress using actual output size, time processed, bitrate, and wall-clock estimates for smooth, accurate updates</li> <li>Real‑time FFmpeg logs: Streaming logs during compression for instant feedback</li> <li>Live Queue Management: View all active jobs with real-time progress, cancel individual jobs, or clear entire queue</li> <li>One‑click Cancel: Stop an in‑flight encode; worker interrupts FFmpeg immediately</li> <li>Queue Clear: Remove all jobs (cancel running, remove queued/completed) with one click</li> <li>Automatic file size optimization: If output exceeds target by &gt;2%, automatically re-encodes with adjusted bitrate</li> <li>Smart retry notifications: Audio alerts and visual notifications when auto-retry occurs</li> <li>History tracking enabled by default: Recent jobs stored in /app/history.json</li> <li>Auto‑download enabled by default</li> <li>Hardware encoders: AV1, HEVC (H.265), H.264 (GPU-accelerated when available)</li> <li>Software fallback: libx264, libx265, libaom-av1 for CPU-only systems</li> <li>Output container choice: MP4 or MKV, with compatibility safeguards</li> <li>Version tracking: UI displays current version (v125+), backend provides /api/version endpoint</li> </ul> https://tedneward.github.io/Research/tools/ansible/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ansible/index.html <p><a href="https://www.redhat.com/en/ansible-collaborative">Website</a> | <a href="https://github.com/ansible/ansible">Source</a></p> <h2>Reading</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://www.xda-developers.com/automated-my-entire-home-network-with-ansible/">I'm automating my entire home network with Ansible</a></li> <li><a href="https://www.xda-developers.com/ansible-made-my-entire-homelab-reproducible-with-one-command/">Ansible made my entire homelab reproducible with one command</a></li> </ul> https://tedneward.github.io/Research/tools/apache-drill/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/apache-drill/index.html <p><a href="https://drill.apache.org/">Website</a></p> https://tedneward.github.io/Research/tools/autotools/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/autotools/index.html <p>Website? Github?</p> <ul> <li>Autoconf / Automake Basics <ul> <li>Mark K. Kim; Linux Users' Group of Davis. March 2, 2004.</li> <li><a href="https://www.lugod.org/presentations/autotools/presentation/autotools.pdf">https://www.lugod.org/presentations/autotools/presentation/autotools.pdf</a></li> </ul> </li> <li>Autotools Mythbuster <ul> <li>"Autotools Mythbuster is a no-nonsense guide to Autotools, written with the idea of providing a full, integrated view of the tools in the GNU build chain: autoconf, automake, libtool, pkg-config, and so on."</li> <li><a href="https://autotools.io/">https://autotools.io/</a></li> </ul> </li> <li>Autotools Tutorial <ul> <li><a href="https://www.lrde.epita.fr/~adl/autotools.html">https://www.lrde.epita.fr/~adl/autotools.html</a></li> </ul> </li> <li>Autotools Tutorial for Beginners <ul> <li><a href="http://markuskimius.wikidot.com/programming:tut:autotools">http://markuskimius.wikidot.com/programming:tut:autotools</a></li> </ul> </li> <li>Autotools, 2nd Edition <ul> <li>A Practitioner's Guide to GNU Autoconf, Automake, and Libtool</li> <li>2019; John Calcote</li> <li><a href="https://nostarch.com/autotools2e">https://nostarch.com/autotools2e</a></li> <li>Chapter 2: A Brief Introduction to the GNU Autotools <ul> <li><a href="https://nostarch.com/download/samples/Autotools2e_Sample_Ch2.pdf">https://nostarch.com/download/samples/Autotools2e_Sample_Ch2.pdf</a></li> </ul> </li> </ul> </li> <li>Autotools: A Demystification Tutorial <ul> <li>Embedded Linux Conference 2016; Thomas Petazzoni</li> <li><a href="https://www.youtube.com/watch?v=_zX8LJ9Xjyk">https://www.youtube.com/watch?v=_zX8LJ9Xjyk</a></li> <li><a href="http://events.linuxfoundation.org/sites/events/files/slides/petazzoni-autotools-tutorial.pdf">http://events.linuxfoundation.org/sites/events/files/slides/petazzoni-autotools-tutorial.pdf</a></li> </ul> </li> <li>Autotools: a practitioner's guide to Autoconf, Automake and Libtool <ul> <li><a href="http://freesoftwaremagazine.com/books/autotools_a_guide_to_autoconf_automake_libtool/">http://freesoftwaremagazine.com/books/autotools_a_guide_to_autoconf_automake_libtool/</a></li> </ul> </li> <li>Four Languages and Lots of Macros: Analyzing Autotools Build Systems <ul> <li>GPCE 2017</li> <li>Jafar M. Al-Kofahi, Suresh Kothari, Christian Kästner</li> <li><a href="https://conf.researchr.org/event/gpce-2017/gpce-2017-gpce-2017-four-languages-and-lots-of-macros-analyzing-autotools-build-systems">https://conf.researchr.org/event/gpce-2017/gpce-2017-gpce-2017-four-languages-and-lots-of-macros-analyzing-autotools-build-systems</a></li> <li><a href="https://www.cs.cmu.edu/~ckaestne/pdf/gpce17.pdf">https://www.cs.cmu.edu/~ckaestne/pdf/gpce17.pdf</a></li> </ul> </li> <li>GNU Autoconf: A uniform means of creating makefiles at build time <ul> <li>Ethan McCallum - C/C++ Users Journal, January 2006</li> <li><a href="http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/2006/0601/0601mccallum/0601mccallum.html">http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/2006/0601/0601mccallum/0601mccallum.html</a></li> <li><a href="https://web.archive.org/http://www.drdobbs.com/gnu-autoconf/184402060">https://web.archive.org/http://www.drdobbs.com/gnu-autoconf/184402060</a></li> <li><a href="https://web.archive.org/http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/2006/0601/0601mccallum/0601mccallum.html">https://web.archive.org/http://collaboration.cmc.ec.gc.ca/science/rpn/biblio/ddj/Website/articles/CUJ/2006/0601/0601mccallum/0601mccallum.html</a></li> </ul> </li> <li>GNU Autoconf, Automake and Libtool <ul> <li><a href="https://www.sourceware.org/autobook/">https://www.sourceware.org/autobook/</a></li> </ul> </li> <li>Minimal autotools template for C++11/14 projects <ul> <li><a href="https://github.com/mpoullet/autotools-cpp-template">https://github.com/mpoullet/autotools-cpp-template</a></li> </ul> </li> <li>Minimal autotools template for C/C++ library projects <ul> <li><a href="https://github.com/mpoullet/autotools-lib-template">https://github.com/mpoullet/autotools-lib-template</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/bdeverify/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bdeverify/index.html <p><a href="https://github.com/bloomberg/bde_verify">Source</a> | <a href="http://bloomberg.github.io/bde_verify/bde_verify_build/html/">Website</a></p> https://tedneward.github.io/Research/tools/blint/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/blint/index.html <p><a href="https://github.com/owasp-dep-scan/blint">Source</a></p> <p>Several source code analysis tools can examine a code repository and generate an SBOM. But what about a binary executable, where the code repository may not be available and the executable is the only artifact to work with? Enter BLint, which aids in generating an SBOM for a binary executable. Creating BLint as open source and adopting it into the OWASP family means that it will be available to all and will grow and evolve.</p> <ul> <li>Can be used to detect overlooked security weaknesses and code signing or authenticode issues of compiled binaries in CI/CD pipelines. Third-party or proprietary dependencies in the form of binaries should be analyzed as part of software supply chain security efforts</li> <li>Does not rely on signatures but instead focuses on capabilities</li> <li>Does not execute the target, preserving resources and avoiding the need for a live environment</li> </ul> <p>Supported binary formats:</p> <ul> <li>Android (apk, aab)</li> <li>ELF (GNU, musl)</li> <li>PE (exe, dll)</li> <li>Mach-O (x64, arm64)</li> </ul> <p>The SBOM feature is supported for these types:</p> <ul> <li>Android (apk/aab)</li> <li>.NET executable binaries</li> <li>Go binaries</li> </ul> <p>They are looking at adding the following capabilities:</p> <ol> <li>Detecting libraries dynamically loaded during runtime</li> <li>Offering a deep mode that will list symbols within static libraries.</li> <li>Add additional annotations and refine existing ones.</li> <li>Add CycloneDx 1.6 support for the SBOM feature.</li> </ol> <p><a href="https://www.helpnetsecurity.com/2024/05/14/blint-open-source-check-security-properties-executables/">"BLint: Open-source tool to check the security properties of your executables"</a></p> https://tedneward.github.io/Research/tools/boomerang/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/boomerang/index.html <p><a href="http://boomerang.sourceforge.net/index.php">Website</a></p> <blockquote> <p>This project is an attempt to develop a real decompiler for machine code programs through the open source community. A decompiler takes as input an executable file, and attempts to create a high level, compilable, possibly even maintainable source file that does the same thing. It is therefore the opposite of a compiler, which takes a source file and makes an executable. However, a general decompiler does not attempt to reverse every action of the decompiler, rather it transforms the input program repeatedly until the result is high level source code. It therefore won't recreate the original source file; probably nothing like it. It does not matter if the executable file has symbols or not, or was compiled from any particular language. (However, declarative languages like ML are not considered.)</p> </blockquote> <p>No updates since 2012, Sourceforge page lists a number of complaints. Probably dead.</p> https://tedneward.github.io/Research/tools/browser-extensions/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/browser-extensions/index.html <p><a href="https://ardalis.com/chrome-browser-tabs-to-links/">Bookmarks to Tabs</a></p> https://tedneward.github.io/Research/tools/byte-me/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/byte-me/index.html <p><a href="https://byte-me.dev/">Website</a></p> https://tedneward.github.io/Research/tools/cave-miner/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cave-miner/index.html <p><a href="https://github.com/Antonin-Deniau/cave_miner">Github</a></p> https://tedneward.github.io/Research/tools/cherry-bomb/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cherry-bomb/index.html <p><a href="https://www.blstsecurity.com/cherrybomb">Website</a> | <a href="https://github.com/blst-security/cherrybomb">Source</a></p> https://tedneward.github.io/Research/tools/clover/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/clover/index.html <p><a href="https://bitbucket.org/atlassian/clover/src/master/">Source</a> | <a href="http://confluence.atlassian.com/display/CLOVER/Clover+Documentation+Home">Docs</a></p> https://tedneward.github.io/Research/tools/coala/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/coala/index.html <p><a href="https://coala.io/#/home">Website</a> | <a href="https://github.com/coala/coala">GitHub</a></p> <p>coala provides a unified interface for linting and fixing code with a single configuration file, regardless of the programming languages used. You can use coala from within your favorite editor, integrate it with your CI, get the results as JSON, or customize it to your needs with its flexible configuration syntax.</p> <p>coala has a set of official bears (plugins) for several languages, including popular languages such as C/C++, Python, JavaScript, CSS, Java and many more, in addition to some generic language independent algorithms. To learn more about the different languages supported and the bears themselves, click below.</p> https://tedneward.github.io/Research/tools/codesize/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/codesize/index.html <p><a href="https://github.com/zeux/codesize">Github</a></p> https://tedneward.github.io/Research/tools/cppcheck/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cppcheck/index.html <p><a href="http://cppcheck.sourceforge.net/">Website</a> | <a href="https://github.com/danmar/cppcheck">Source</a></p> https://tedneward.github.io/Research/tools/act/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/act/index.html <p><a href="https://nektosact.com/">Website</a> | <a href="https://github.com/nektos/act">Source</a></p> <p><code>brew install act</code> (macOS) | <code>choco install act-cli</code> (Windows)</p> https://tedneward.github.io/Research/tools/ant/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ant/index.html <p><a href="http://ant.apache.org/">Website</a> | <a href="http://ant.apache.org/git.html">Source</a></p> <p>The main known usage of Ant is the build of Java applications. Ant supplies a number of built-in tasks allowing to compile, assemble, test and run Java applications. Ant can also be used effectively to build non Java applications, for instance C or C++ applications. More generally, Ant can be used to pilot any type of process which can be described in terms of targets and tasks.</p> https://tedneward.github.io/Research/tools/ast-grep/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ast-grep/index.html <p><a href="https://ast-grep.github.io/">Website</a> | <a href="https://github.com/ast-grep/ast-grep">GitHub</a></p> <p><strong>Search and Rewrite</strong><br> ast-grep is a code tool for structural search and replace. It is like syntax-aware grep/sed! You can write code patterns to locate and modify code, based on AST, in thousands of files, interactively.</p> <pre><code>ast-grep -p '$A &amp;&amp; $A()' -r '$A?.()' </code></pre> <p><strong>Scan as Linter</strong><br> ast-grep is a versatile and flexible tool for linting code with AST patterns. You can easily add new customized rules with intuitive syntax and enjoy pretty error reporting out of box.</p> <pre><code>ast-grep scan </code></pre> <p><strong>Programmatic Usage</strong><br> ast-grep also provides node-js binding to access syntax trees programmatically. You can use jQuery like utility methods to traverse syntax tree nodes. Node API also has opt-in type safety.</p> <pre><code>npm install @ast-grep/napi </code></pre> <p><strong>Polyglot</strong><br> Supports C, C#, Go, Java, Javascript, Kotlin, Python, Rust, TypeScript, and many more.</p> <p>Rules are written in YAML files</p> https://tedneward.github.io/Research/tools/avatar/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/avatar/index.html <p><a href="https://github.com/avatartwo/avatar2">Source</a></p> <ul> <li>Avatar²: Towards an open source binary firmware analysis framework <ul> <li>34C3 (2017)</li> <li><a href="https://media.ccc.de/v/34c3-9195-avatar">https://media.ccc.de/v/34c3-9195-avatar</a></li> <li><a href="https://www.youtube.com/watch?v=cRxmxapS8N4">https://www.youtube.com/watch?v=cRxmxapS8N4</a></li> <li><a href="https://events.ccc.de/congress/2017/Fahrplan/events/9195.html">https://events.ccc.de/congress/2017/Fahrplan/events/9195.html</a></li> </ul> </li> <li>Avatar²: A Multi-target Orchestration Platform <ul> <li><a href="http://s3.eurecom.fr/docs/bar18_muench.pdf">http://s3.eurecom.fr/docs/bar18_muench.pdf</a></li> </ul> </li> <li>Dynamic Binary Firmware Analysis: Challenges &amp; Solutions <ul> <li>2019 PhD Dissertation; Marius Muench</li> <li><a href="http://www.eurecom.fr/en/publication/5969/detail/dynamic-binary-firmware-analysis-challenges-and-solutions">http://www.eurecom.fr/en/publication/5969/detail/dynamic-binary-firmware-analysis-challenges-and-solutions</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/bedit/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bedit/index.html <p><a href="https://kipt.itch.io/bedit">Website</a> | Closed source</p> https://tedneward.github.io/Research/tools/bloaty-mcbloatface/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bloaty-mcbloatface/index.html <p><a href="http://blog.reverberate.org/2016/11/07/introducing-bloaty-mcbloatface.html">Website</a> | <a href="https://github.com/google/bloaty">Github</a></p> <p><a href="https://github.com/CppCon/CppCon2019/tree/master/Presentations/abusing_compiler_tools">Talk: “(Ab)using Compiler Tools”</a>; CppCon 2019; Reka Kovacs (<a href="https://www.youtube.com/watch?v=5Lke40ywMaU">YouTube</a>)</p> https://tedneward.github.io/Research/tools/boostbuild/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/boostbuild/index.html <p><a href="http://www.boost.org/build/">Website</a></p> <p><a href="https://github.com/boostcon/cppnow_presentations_2016/blob/master/03_friday/understanding_boost_build.pdf">Understanding Boost.Build</a> (<a href="https://www.youtube.com/watch?v=8yc7ZvU0yOw">YouTube</a>) -- C++Now 2016; Steven Watanabe</p> <p><a href="https://github.com/boostcon/2011_presentations/raw/master/mon/Boost.Build.pdf">Complete Overview on Boost.Jam and Boost.Build</a> (<a href="https://www.youtube.com/watch?v=OgYwvzUUupM">YouTube</a>) -- BoostCon 2011; Boris Schaeling</p> https://tedneward.github.io/Research/tools/build2/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/build2/index.html <p><a href="https://build2.org/">Website</a> | <a href="https://github.com/build2/">Github</a></p> <p><a href="https://www.youtube.com/watch?v=Nni2Qu2WitY">C++ Dependency Management: from Package Consumption to Project Development</a> -- CppCon 2018; Boris Kolpackov</p> https://tedneward.github.io/Research/tools/cake/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cake/index.html <p><a href="https://cakebuild.net/">Website</a> | <a href="https://github.com/cake-build">Source</a> | <a href="https://cakebuild.net/docs">Docs</a></p> https://tedneward.github.io/Research/tools/cfr/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cfr/index.html <p><a href="https://www.benf.org/other/cfr/">Website</a> | <a href="https://github.com/leibnitz27/cfr">Source</a></p> https://tedneward.github.io/Research/tools/clamshell/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/clamshell/index.html <p><a href="https://github.com/benrutter/clamshell">Source</a></p> https://tedneward.github.io/Research/tools/cmake/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cmake/index.html <p>CMake is an open-source, cross-platform family of tools designed to build, test and package software. CMake is used to control the software compilation process using simple platform and compiler independent configuration files, and generate native makefiles and workspaces that can be used in the compiler environment of your choice.</p> <p><a href="https://cmake.org/">Website</a> | <a href="https://gitlab.kitware.com/cmake/cmake">Github</a></p> <h3>Reading</h3> <ul> <li><a href="https://cliutils.gitlab.io/modern-cmake/">An Introduction to Modern CMake</a> - Henry Schreiner (HTML)</li> <li><a href="https://cmake.org/cmake/help/latest/guide/tutorial/index.html">CMake Tutorial</a> (HTML)</li> <li><a href="https://www.jetbrains.com/help/clion/quick-cmake-tutorial.html">Quick CMake tutorial</a> (HTML)</li> </ul> <h2><a href="https://learnxinyminutes.com/docs/cmake/">Learn X in Y Minutes</a></h2> <p>CMake is a cross-platform, open-source build system. This tool allows you to test, compile, and create packages of your source code.</p> <p>The problem that CMake tries to solve is the problem of Makefiles and Autoconfigure on cross-platforms (different make interpreters have different commands) and the ease-of-use on linking 3rd party libraries.</p> <p>CMake is an extensible, open-source system that manages the build process in an operating system and compiler-agnostic manner. Unlike many cross-platform systems, CMake is designed to be used in conjunction with the native build environment. Simple configuration files placed in each source directory (called CMakeLists.txt files) are used to generate standard build files (e.g., makefiles on Unix and projects/workspaces in Windows MSVC) which are used in the usual way.</p> <pre><code class="language-cmake"># In CMake, this is a comment # To run our code, please perform the following commands: # - mkdir build &amp;&amp; cd build # - cmake .. # - make # # With those steps, we will follow the best practice to compile into a subdir # and the second line will request to CMake to generate a new OS-dependent # Makefile. Finally, run the native Make command. #------------------------------------------------------------------------------ # Basic #------------------------------------------------------------------------------ # # The CMake file MUST be named as "CMakeLists.txt". # Setup the minimum version required of CMake to generate the Makefile cmake_minimum_required (VERSION 2.8) # Raises a FATAL_ERROR if version &lt; 2.8 cmake_minimum_required (VERSION 2.8 FATAL_ERROR) # We define the name of our project, and this changes some directories # naming convention generated by CMake. We can send the LANG of code # as the second param project (learncmake C) # Set the project source dir (just convention) set( LEARN_CMAKE_SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR} ) set( LEARN_CMAKE_BINARY_DIR ${CMAKE_CURRENT_BINARY_DIR} ) # It's useful to set up the current version of our code in the build system # using a `semver` style set (LEARN_CMAKE_VERSION_MAJOR 1) set (LEARN_CMAKE_VERSION_MINOR 0) set (LEARN_CMAKE_VERSION_PATCH 0) # Send the variables (version number) to the source code header configure_file ( "${PROJECT_SOURCE_DIR}/TutorialConfig.h.in" "${PROJECT_BINARY_DIR}/TutorialConfig.h" ) # Include Directories # In GCC, this will invoke the "-I" command include_directories( include ) # Where are the additional libraries installed? Note: provide includes # path here, subsequent checks will resolve everything else set( CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/CMake/modules/" ) # Conditions if ( CONDITION ) # Output! # Incidental information message(STATUS "My message") # CMake Warning, continue processing message(WARNING "My message") # CMake Warning (dev), continue processing message(AUTHOR_WARNING "My message") # CMake Error, continue processing, but skip generation message(SEND_ERROR "My message") # CMake Error, stop processing and generation message(FATAL_ERROR "My message") endif() if( CONDITION ) elseif( CONDITION ) else( CONDITION ) endif( CONDITION ) # Loops foreach(loop_var arg1 arg2 ...) COMMAND1(ARGS ...) COMMAND2(ARGS ...) ... endforeach(loop_var) foreach(loop_var RANGE total) foreach(loop_var RANGE start stop [step]) foreach(loop_var IN [LISTS [list1 [...]]] [ITEMS [item1 [...]]]) while(condition) COMMAND1(ARGS ...) COMMAND2(ARGS ...) ... endwhile(condition) # Logic Operations if(FALSE AND (FALSE OR TRUE)) message("Don't display!") endif() # Set a regular, cache, or environment variable to a given value. # If the PARENT_SCOPE option is given, the variable will be set in the scope # above the current scope. # `set(&lt;variable&gt; &lt;value&gt;... [PARENT_SCOPE])` # How to reference variables inside quoted and unquoted arguments? # A variable reference is replaced by either the variable value or by the # empty string if the variable is not set. ${variable_name} # Lists # Setup the list of source files set( LEARN_CMAKE_SOURCES src/main.c src/imagem.c src/pather.c ) # Calls the compiler # # ${PROJECT_NAME} refers to Learn_CMake add_executable( ${PROJECT_NAME} ${LEARN_CMAKE_SOURCES} ) # Link the libraries target_link_libraries( ${PROJECT_NAME} ${LIBS} m ) # Where are the additional libraries installed? Note: provide includes # path here, subsequent checks will resolve everything else set( CMAKE_MODULE_PATH ${CMAKE_MODULE_PATH} "${CMAKE_SOURCE_DIR}/CMake/modules/" ) # Compiler Condition (gcc ; g++) if ( "${CMAKE_C_COMPILER_ID}" STREQUAL "GNU" ) message( STATUS "Setting the flags for ${CMAKE_C_COMPILER_ID} compiler" ) add_definitions( --std=c99 ) endif() # Check for OS if( UNIX ) set( LEARN_CMAKE_DEFINITIONS "${LEARN_CMAKE_DEFINITIONS} -Wall -Wextra -Werror -Wno-deprecated-declarations -Wno-unused-parameter -Wno-comment" ) endif() </code></pre> <h3>More Resources</h3> <ul> <li><a href="https://cmake.org/cmake-tutorial/">CMake tutorial</a></li> <li><a href="https://cmake.org/documentation/">CMake documentation</a></li> <li><a href="http://amzn.com/1930934319/">Mastering CMake</a></li> <li><a href="https://cliutils.gitlab.io/modern-cmake/">An Introduction to Modern CMake</a></li> <li><a href="https://github.com/friendlyanon/cmake-init">cmake-init</a>: an opinionated CMake project initializer that generates CMake projects which are FetchContent ready, separate consumer and developer targets, provide install rules with proper relocatable CMake packages and use modern CMake (3.14+). (Install with <code>[pip3](/languaegs/python) install cmake-init</code>; generated scripts require <code>[conan](/tools/conan)</code>)</li> </ul> <hr> <h2>Links</h2> <p><a href="https://github.com/onqtam/awesome-cmake">Awesome CMake</a>: A curated list of awesome CMake scripts, modules, examples and others</p> <h2>Examples</h2> <ul> <li>C++/CMake modern boilerplate <ul> <li><a href="https://github.com/Lectem/cpp-boilerplate">https://github.com/Lectem/cpp-boilerplate</a></li> </ul> </li> <li>CMake: config mode of find_package command (examples) <ul> <li><a href="https://github.com/forexample/package-example">https://github.com/forexample/package-example</a></li> </ul> </li> <li>CMake cookbook recipes <ul> <li><a href="https://github.com/dev-cafe/cmake-cookbook">https://github.com/dev-cafe/cmake-cookbook</a></li> </ul> </li> <li>CMake Examples <ul> <li><a href="https://github.com/ttroy50/cmake-examples">https://github.com/ttroy50/cmake-examples</a></li> </ul> </li> <li>learning-cmake: a simple CMake tutorial project which contains some different scenarios <ul> <li><a href="https://github.com/Akagi201/learning-cmake">https://github.com/Akagi201/learning-cmake</a></li> </ul> </li> <li>Modern CMake Examples <ul> <li><a href="https://github.com/pr0g/cmake-examples">https://github.com/pr0g/cmake-examples</a></li> </ul> </li> <li>Polly: Collection of CMake toolchain files and scripts for cross-platform build and CI testing (GCC, Visual Studio, iOS, Android, Clang analyzer, sanitizers etc.) <ul> <li><a href="https://github.com/ruslo/polly">https://github.com/ruslo/polly</a></li> </ul> </li> </ul> <h2>Reading</h2> <ul> <li>Basic CMake usage <ul> <li><a href="https://codingnest.com/basic-cmake/">https://codingnest.com/basic-cmake/</a></li> </ul> </li> <li>CGold: The Hitchhiker’s Guide to the CMake <ul> <li><a href="https://cgold.readthedocs.io/en/latest/">https://cgold.readthedocs.io/en/latest/</a></li> </ul> </li> <li>CMake 3.16 added support for precompiled headers &amp; unity builds - what you need to know <ul> <li><a href="http://onqtam.com/programming/2019-12-20-pch-unity-cmake-3-16/">http://onqtam.com/programming/2019-12-20-pch-unity-cmake-3-16/</a></li> </ul> </li> <li>CMake Coding Style <ul> <li><a href="https://community.kde.org/Policies/CMake_Coding_Style">https://community.kde.org/Policies/CMake_Coding_Style</a></li> </ul> </li> <li>Dev Santa Claus - Nick Sarten <ul> <li>Part 1: Adding Clang Sanitizers to a CMake Build <ul> <li><a href="https://genbattle.bitbucket.io/blog/2018/01/05/Dev-Santa-Claus-Part-1/">https://genbattle.bitbucket.io/blog/2018/01/05/Dev-Santa-Claus-Part-1/</a></li> </ul> </li> <li>Part 2: C++ code coverage metrics with gcov <ul> <li>setting up code coverage metrics for a C++ codebase built using Bamboo, CMake, and GCC</li> <li><a href="https://genbattle.bitbucket.io/blog/2018/01/19/Dev-Santa-Claus-Part-2/">https://genbattle.bitbucket.io/blog/2018/01/19/Dev-Santa-Claus-Part-2/</a></li> </ul> </li> </ul> </li> <li>Effective CMake by Kai Wolf <ul> <li><a href="http://effective-cmake.com">http://effective-cmake.com</a></li> <li><a href="https://leanpub.com/effective-cmake">https://leanpub.com/effective-cmake</a></li> </ul> </li> <li>Effective Modern CMake - Manuel Binna <ul> <li><a href="https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1">https://gist.github.com/mbinna/c61dbb39bca0e4fb7d1f73b0d66a4fd1</a></li> </ul> </li> <li>Everything You Never Wanted to Know About CMake <ul> <li><a href="https://izzys.casa/2019/02/everything-you-never-wanted-to-know-about-cmake/">https://izzys.casa/2019/02/everything-you-never-wanted-to-know-about-cmake/</a></li> </ul> </li> <li>How to Build a CMake-Based Project <ul> <li><a href="http://preshing.com/20170511/how-to-build-a-cmake-based-project/">http://preshing.com/20170511/how-to-build-a-cmake-based-project/</a></li> </ul> </li> <li>It's Time To Do CMake Right <ul> <li><a href="https://pabloariasal.github.io/2018/02/19/its-time-to-do-cmake-right/">https://pabloariasal.github.io/2018/02/19/its-time-to-do-cmake-right/</a></li> </ul> </li> <li>Learn CMake's Scripting Language in 15 Minutes <ul> <li><a href="http://preshing.com/20170522/learn-cmakes-scripting-language-in-15-minutes/">http://preshing.com/20170522/learn-cmakes-scripting-language-in-15-minutes/</a></li> </ul> </li> <li>Professional CMake: A Practical Guide <ul> <li><a href="https://crascit.com/professional-cmake/">https://crascit.com/professional-cmake/</a></li> </ul> </li> <li>siliceum - Clément Grégoire <ul> <li>CMake basics <ul> <li><a href="https://www.siliceum.com/en/blog/post/cmake_01_cmake-basics">https://www.siliceum.com/en/blog/post/cmake_01_cmake-basics</a></li> </ul> </li> <li>CMake customization points <ul> <li><a href="https://www.siliceum.com/en/blog/post/cmake_02_customization-points">https://www.siliceum.com/en/blog/post/cmake_02_customization-points</a></li> </ul> </li> </ul> </li> <li>Speed up your C++ unit tests cycles with CMake/CTest (and the right testing framework) <ul> <li><a href="https://a4z.bitbucket.io/blog/2018/05/17/Speed-up-your-test-cycles-with-CMake.html">https://a4z.bitbucket.io/blog/2018/05/17/Speed-up-your-test-cycles-with-CMake.html</a></li> </ul> </li> <li>The Architecture of Open Source Applications: CMake <ul> <li><a href="http://www.aosabook.org/en/cmake.html">http://www.aosabook.org/en/cmake.html</a></li> </ul> </li> <li>The Ultimate Guide to Modern CMake <ul> <li><a href="https://rix0r.nl/blog/2015/08/13/cmake-guide/">https://rix0r.nl/blog/2015/08/13/cmake-guide/</a></li> </ul> </li> <li>Tutorial: Easy dependency management for C++ with CMake and Git <ul> <li><a href="http://foonathan.net/blog/2016/07/07/cmake-dependency-handling.html">http://foonathan.net/blog/2016/07/07/cmake-dependency-handling.html</a></li> </ul> </li> <li>Using ccache with CMake <ul> <li><a href="https://crascit.com/2016/04/09/using-ccache-with-cmake/">https://crascit.com/2016/04/09/using-ccache-with-cmake/</a></li> </ul> </li> </ul> <h2>Talks</h2> <ul> <li>Building C++ <ul> <li>C++ Edinburgh 2018; Morris Hafner</li> <li><a href="https://www.youtube.com/watch?v=n_f-2p5eDBo">https://www.youtube.com/watch?v=n_f-2p5eDBo</a></li> </ul> </li> <li>C++ Weekly - Jason Turner <ul> <li>Ep 78 - Intro to CMake <ul> <li><a href="https://www.youtube.com/watch?v=HPMvU64RUTY">https://www.youtube.com/watch?v=HPMvU64RUTY</a></li> <li><a href="https://github.com/lefticus/cpp_starter_project">https://github.com/lefticus/cpp_starter_project</a></li> </ul> </li> <li>Ep 82 - Intro To CTest <ul> <li><a href="https://www.youtube.com/watch?v=ZlMbqFcJEzA">https://www.youtube.com/watch?v=ZlMbqFcJEzA</a></li> </ul> </li> <li>Ep 208 - The Ultimate CMake / C++ Quick Start <ul> <li>"CMake, sanitizers, clang-tidy, conan, cmake-format, clang-format, cppcheck, doxygen, ctest, catch, {fmt}, and more!"</li> <li><a href="https://www.youtube.com/watch?v=YbgH7yat-Jo">https://www.youtube.com/watch?v=YbgH7yat-Jo</a></li> </ul> </li> </ul> </li> <li>Deep CMake for Library Authors <ul> <li>CppCon 2019; Craig Scott</li> <li><a href="https://www.youtube.com/watch?v=m0DwB4OvDXk">https://www.youtube.com/watch?v=m0DwB4OvDXk</a></li> </ul> </li> <li>Effective CMake <ul> <li>Daniel Pfeifer</li> <li>C++Now 2017 <ul> <li><a href="https://www.youtube.com/watch?v=bsXLMQ6WgIk">https://www.youtube.com/watch?v=bsXLMQ6WgIk</a></li> <li><a href="https://github.com/boostcon/cppnow_presentations_2017/raw/master/05-19-2017_friday/effective_cmake__daniel_pfeifer__cppnow_05-19-2017.pdf">https://github.com/boostcon/cppnow_presentations_2017/raw/master/05-19-2017_friday/effective_cmake__daniel_pfeifer__cppnow_05-19-2017.pdf</a></li> </ul> </li> <li>MUCplusplus 2017 <ul> <li><a href="https://www.youtube.com/watch?v=rLopVhns4Zs">https://www.youtube.com/watch?v=rLopVhns4Zs</a></li> </ul> </li> </ul> </li> <li>Effective dependency management with CMake <ul> <li>MUCplusplus 2017; Kai Wolf</li> <li><a href="https://www.youtube.com/watch?v=QayyhI-36os">https://www.youtube.com/watch?v=QayyhI-36os</a></li> </ul> </li> <li>Embracing Modern CMake <ul> <li>Stephen Kelly</li> <li><a href="https://steveire.wordpress.com/2017/11/05/embracing-modern-cmake/">https://steveire.wordpress.com/2017/11/05/embracing-modern-cmake/</a></li> <li>Dublin C++ User Group 2017 <ul> <li><a href="https://www.youtube.com/watch?v=JsjI5xr1jxM">https://www.youtube.com/watch?v=JsjI5xr1jxM</a></li> </ul> </li> <li>NDC TechTown 2019 <ul> <li><a href="https://www.youtube.com/watch?v=mn1ZnO3MtVk">https://www.youtube.com/watch?v=mn1ZnO3MtVk</a></li> </ul> </li> </ul> </li> <li>How to CMake Good <ul> <li><a href="https://vector-of-bool.github.io/2018/08/12/cmake-good.html">https://vector-of-bool.github.io/2018/08/12/cmake-good.html</a></li> <li><a href="https://www.youtube.com/playlist?list=PLK6MXr8gasrGmIiSuVQXpfFuE1uPT615s">https://www.youtube.com/playlist?list=PLK6MXr8gasrGmIiSuVQXpfFuE1uPT615s</a></li> </ul> </li> <li>Introduction to CMake <ul> <li>SwedenCpp 2018; Florent Castelli</li> <li>Examples: <ul> <li><a href="https://bit.ly/swedencpp-cmake-workshop">https://bit.ly/swedencpp-cmake-workshop</a></li> <li><a href="https://github.com/Orphis/cmake-workshop">https://github.com/Orphis/cmake-workshop</a></li> </ul> </li> <li>Slides: <ul> <li><a href="https://bit.ly/swedencpp-cmake-slides">https://bit.ly/swedencpp-cmake-slides</a></li> <li><a href="https://docs.google.com/presentation/d/1vIEfCE33-1BEMT1nNQQbhe-1va-3m2pL9c6_WrIIjdU/">https://docs.google.com/presentation/d/1vIEfCE33-1BEMT1nNQQbhe-1va-3m2pL9c6_WrIIjdU/</a></li> </ul> </li> <li>Video: <ul> <li><a href="https://www.youtube.com/watch?v=jt3meXdP-QI">https://www.youtube.com/watch?v=jt3meXdP-QI</a></li> </ul> </li> </ul> </li> <li>Modern CMake for modular design <ul> <li>Meeting C++ 2017; Mathieu Ropert</li> <li><a href="https://www.youtube.com/watch?v=ztrnb-bVVPo">https://www.youtube.com/watch?v=ztrnb-bVVPo</a></li> </ul> </li> <li>More Modern CMake - Working with CMake 3.12 and later <ul> <li>Meeting C++ 2018; Deniz Bahadir</li> <li><a href="https://meetingcpp.com/2018/Talks/items/More_Modern_CMake___Working_with_CMake_3_12_and_later.html">https://meetingcpp.com/2018/Talks/items/More_Modern_CMake___Working_with_CMake_3_12_and_later.html</a></li> <li><a href="https://meetingcpp.com/mcpp/slides/2018/MoreModernCMake.pdf">https://meetingcpp.com/mcpp/slides/2018/MoreModernCMake.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=y7ndUhdQuU8">https://www.youtube.com/watch?v=y7ndUhdQuU8</a></li> </ul> </li> <li>Oh No! More Modern CMake <ul> <li>Meeting C++ 2019; Deniz Bahadir</li> <li><a href="https://www.youtube.com/watch?v=y9kSr5enrSk">https://www.youtube.com/watch?v=y9kSr5enrSk</a></li> <li><a href="https://github.com/Bagira80/More-Modern-CMake">https://github.com/Bagira80/More-Modern-CMake</a></li> </ul> </li> <li>Using Modern CMake Patterns to Enforce a Good Modular Design <ul> <li>CppCon 2017; Mathieu Ropert</li> <li><a href="https://www.youtube.com/watch?v=eC9-iRN2b04">https://www.youtube.com/watch?v=eC9-iRN2b04</a></li> </ul> </li> </ul> <h2>Additional software</h2> <ul> <li>BLT (Build, Link and Triumph): A streamlined CMake build system foundation for developing HPC software <ul> <li>composition of CMake macros and several widely used open source tools assembled to simplify HPC software development.</li> <li><a href="https://github.com/llnl/blt">https://github.com/llnl/blt</a></li> <li><a href="http://llnl-blt.readthedocs.io/en/latest/">http://llnl-blt.readthedocs.io/en/latest/</a></li> </ul> </li> <li>CMake Unit <ul> <li>A unit testing framework for CMake.</li> <li><a href="https://github.com/polysquare/cmake-unit">https://github.com/polysquare/cmake-unit</a></li> </ul> </li> <li>CMakeBuildPackage: Automatic project build script with CMake <ul> <li><a href="https://github.com/berenm/CMakeBuildPackage">https://github.com/berenm/CMakeBuildPackage</a></li> </ul> </li> <li>cmake format: Source code formatter for cmake listfiles. <ul> <li><a href="https://github.com/cheshirekow/cmake_format">https://github.com/cheshirekow/cmake_format</a></li> </ul> </li> <li>CMake Modules <ul> <li>Additional CMake Modules <ul> <li><a href="https://github.com/bilke/cmake-modules">https://github.com/bilke/cmake-modules</a></li> </ul> </li> <li>Ryan's CMake Modules <ul> <li><a href="https://github.com/rpavlik/cmake-modules">https://github.com/rpavlik/cmake-modules</a></li> </ul> </li> </ul> </li> <li>cmany: CMake build tree batching tool <ul> <li><a href="https://github.com/biojppm/cmany">https://github.com/biojppm/cmany</a></li> </ul> </li> <li>configure-cmake: autotools-style configure script wrapper around CMake <ul> <li><a href="https://github.com/nemequ/configure-cmake">https://github.com/nemequ/configure-cmake</a></li> </ul> </li> <li>CPM.cmake <ul> <li>CMake's missing package manager. A small CMake script for setup-free, cross-platform and reproducible dependency management.</li> <li><a href="https://github.com/TheLartians/CPM.cmake">https://github.com/TheLartians/CPM.cmake</a></li> </ul> </li> <li>Izzy's eXtension Modules: Make CMake less painful when trying to write Modern Flexible CMake <ul> <li><a href="https://github.com/slurps-mad-rips/ixm">https://github.com/slurps-mad-rips/ixm</a></li> </ul> </li> <li>ucm - useful cmake macros <ul> <li><a href="https://github.com/onqtam/ucm">https://github.com/onqtam/ucm</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/cocor/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cocor/index.html <p>"Coco/R is a compiler generator, which takes an attributed grammar of a source language and generates a scanner and a parser for this language. The scanner works as a deterministic finite automaton. The parser uses recursive descent. LL(1) conflicts can be resolved by a multi-symbol lookahead or by semantic checks. Thus the class of accepted grammars is LL(k) for an arbitrary k. There are versions of Coco/R for different languages (see below). The latest versions from the University of Linz are those for C#, Java and C++."</p> <p><a href="http://www.ssw.uni-linz.ac.at/Coco/">Website</a></p> https://tedneward.github.io/Research/tools/codespeak/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/codespeak/index.html <p><a href="https://codespeak.dev/">Website</a></p> <h2>Getting Started</h2> <p><code>uv tool install codespeak-cli</code></p> https://tedneward.github.io/Research/tools/cppnet/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cppnet/index.html <p><a href="https://github.com/xtravar/CppNet">Source</a></p> https://tedneward.github.io/Research/tools/affine/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/affine/index.html <p><a href="https://affine.pro/">Website</a> | <a href="https://github.com/toeverything/AFFiNE">Source</a></p> <p>Obsidian/Notion competitor.</p> <h2>Installation/Self-Hosting</h2> <p><a href="https://docs.affine.pro/self-host-affine/install/docker-compose-recommend">Docker Compose</a></p> <h3>1. Create folders to put persisted data</h3> <pre><code>mkdir affine cd affine </code></pre> <p>2. Get latest docker-compose.yml</p> <p>You can choose to download the latest compose file tagged together with AFFiNE releases.</p> <pre><code>wget -O docker-compose.yml https://github.com/toeverything/affine/releases/latest/download/docker-compose.yml </code></pre> <p>or go to <a href="https://docs.affine.pro/self-host-affine/references/docker-compose-yml">docker-compose.yml</a> and copy the content, put it in manually created <code>docker-compose.yml</code> file in the folder created in <a href="https://docs.affine.pro/self-host-affine/install/docker-compose-recommend#step1">Step 1</a>.</p> <p>3. Get .env file</p> <p>A <code>.env</code> file is required to configure docker volumes mapping to specific where the user data will be persisted at and other required environment variables.</p> <pre><code>wget -O .env https://github.com/toeverything/affine/releases/latest/download/default.env.example </code></pre> <p>or go to <a href="https://docs.affine.pro/self-host-affine/references/env">.env</a> and copy the content, put it in manually created <code>.env</code> file in the folder created in <a href="https://docs.affine.pro/self-host-affine/install/docker-compose-recommend#step1">Step 1</a>.</p> <p>Then, you need to update the <code>.env</code> file to correct the fields to your configurations.</p> <p>.env</p> <pre><code>The folder for Postgres data, usually to be the one in step 1 with '/postgres' suffix DB_DATA_LOCATION=./postgres The folder for uploaded blobs, usually to be the one in step 1 with '/storage' suffix UPLOAD_LOCATION=./storage The folder for custom configurations, usually to be the one in step with '/config' suffix CONFIG_LOCATION=./config DATABASE credentials and names to initialize Postgres DB_USERNAME=affine DB_PASSWORD= DB_DATABASE=affine </code></pre> <p>WARNING</p> <p>Most of the values in <code>.env</code> file shouldn't be changed once your host initialized and start to have data written.</p> <p>If you do want to, for example, change the location of the <code>affine</code> folder, please check out <a href="https://docs.affine.pro/self-host-affine/administer/backup-and-restore">Backup and Restore</a> for help.</p> <p>4. Start the containers</p> <pre><code>docker compose up -d </code></pre> <p>5. Validate the deploy in browser</p> <p>If everything goes perfect, you will be able to visit your AFFiNE in browser with <a href="http://localhost:3010/">http://localhost:3010</a>. You might update the port to what you configured in <code>.env</code> file.</p> https://tedneward.github.io/Research/tools/antigravity/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/antigravity/index.html <p><a href="https://antigravity.google/">Website</a></p> https://tedneward.github.io/Research/tools/atom/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/atom/index.html <p><a href="http://atom.io/">Website</a> | <a href="https://github.com/atom">Source</a></p> <p>Built on <a href="../presentation/electron">Electron</a>.</p> https://tedneward.github.io/Research/tools/backdoor-factory/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/backdoor-factory/index.html <p><a href="https://github.com/secretsquirrel/the-backdoor-factory">Github</a></p> https://tedneward.github.io/Research/tools/bingrep/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bingrep/index.html <p><a href="https://github.com/m4b/bingrep">Github</a></p> https://tedneward.github.io/Research/tools/booklore/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/booklore/index.html <p><a href="https://github.com/adityachandelgit/BookLore">Source</a> | <a href="https://hub.docker.com/r/booklore/booklore">Docker Hub</a></p> <p>Provides an intuitive interface to browse, read, and track your progress across PDFs and eBooks. With robust metadata management, multi-user support, and a sleek, modern UI, BookLore makes it easy to build and explore your personal library.</p> <pre><code>``` services: booklore: # Official Docker Hub image: image: booklore/booklore:latest # Or the GHCR image: # image: ghcr.io/booklore-app/booklore:latest container_name: booklore environment: - PUID=1000 - PGID=1000 - TZ=Etc/UTC - DATABASE_URL=jdbc:mariadb://mariadb:3306/booklore # Only modify this if you're familiar with JDBC and your database setup - DATABASE_USERNAME=booklore # Must match MYSQL_USER defined in the mariadb container - DATABASE_PASSWORD=your_secure_password # Use a strong password; must match MYSQL_PASSWORD defined in the mariadb container - SWAGGER_ENABLED=false # Enable or disable Swagger UI (API docs). Set to 'true' to allow access; 'false' to block access (recommended for production). depends_on: mariadb: condition: service_healthy ports: - "6060:6060" volumes: - /your/local/path/to/booklore/data:/app/data # Internal app data (settings, metadata, cache) - /your/local/path/to/booklore/books1:/books1 # Book library folder — point to one of your collections - /your/local/path/to/booklore/books2:/books2 # Another book library — you can mount multiple library folders this way - /your/local/path/to/booklore/bookdrop:/bookdrop # Bookdrop folder — drop new files here for automatic import into libraries restart: unless-stopped mariadb: image: lscr.io/linuxserver/mariadb:11.4.5 container_name: mariadb environment: - PUID=1000 - PGID=1000 - TZ=Etc/UTC - MYSQL_ROOT_PASSWORD=super_secure_password # Use a strong password for the database's root user, should be different from MYSQL_PASSWORD - MYSQL_DATABASE=booklore - MYSQL_USER=booklore # Must match DATABASE_USERNAME defined in the booklore container - MYSQL_PASSWORD=your_secure_password # Use a strong password; must match DATABASE_PASSWORD defined in the booklore container volumes: - /your/local/path/to/mariadb/config:/config restart: unless-stopped healthcheck: test: ["CMD", "mariadb-admin", "ping", "-h", "localhost"] interval: 5s timeout: 5s retries: 10 ``` </code></pre> https://tedneward.github.io/Research/tools/bore/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bore/index.html <p><a href="http://bore.pub/">Website</a> | <a href="https://github.com/ekzhang/bore">Source</a></p> https://tedneward.github.io/Research/tools/building/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/building/index.html <p><a href="autotools.html">Autotools</a> | <a href="bazel.html">Bazel</a> | <a href="boostbuild.html">Boost.Build</a> | <a href="build2.html">build2</a> | <a href="cmake.html">CMake</a> | <a href="fastbuild.html">FASTBuild</a> | <a href="gradle.html">Gradle</a> | <a href="make.html">Make and GNUMake</a> | <a href="meson.html">Meson</a> | <a href="msbuild.html">MSBuild</a> <a href="ninja.html">Ninja</a> | <a href="tundra.html">Tundra</a> | <a href="tup.html">Tup</a> | <a href="xcode.html">Xcode</a> | <a href="xmake.html">xmake</a></p> <h1>Readings</h1> <ul> <li>Build Systems à la Carte <ul> <li>Microsoft Research 2018</li> <li>Andrey Mokhov, Neil Mitchell, Simon Peyton Jones</li> <li><a href="https://www.microsoft.com/en-us/research/publication/build-systems-la-carte/">https://www.microsoft.com/en-us/research/publication/build-systems-la-carte/</a></li> <li><a href="https://github.com/snowleopard/build">https://github.com/snowleopard/build</a></li> <li><a href="https://blogs.ncl.ac.uk/andreymokhov/the-task-abstraction/">https://blogs.ncl.ac.uk/andreymokhov/the-task-abstraction/</a></li> <li><a href="https://blogs.ncl.ac.uk/andreymokhov/build-systems-a-la-carte/">https://blogs.ncl.ac.uk/andreymokhov/build-systems-a-la-carte/</a></li> <li><a href="https://icfp18.sigplan.org/event/icfp-2018-papers-build-systems-a-la-carte">https://icfp18.sigplan.org/event/icfp-2018-papers-build-systems-a-la-carte</a></li> <li><a href="https://www.youtube.com/watch?v=BQVT6wiwCxM">https://www.youtube.com/watch?v=BQVT6wiwCxM</a></li> <li>Build Systems à La Carte: Theory and Practice <ul> <li>Journal of Functional Programming 30 (2020)</li> <li>Andrey Mokhov, Neil Mitchell, Simon Peyton Jones</li> <li><a href="https://doi.org/10.1017/S0956796820000088">https://doi.org/10.1017/S0956796820000088</a></li> </ul> </li> </ul> </li> <li>Build System Rules and Algorithms <ul> <li>Mike Shal (2009)</li> <li><a href="http://gittup.org/tup/build_system_rules_and_algorithms.pdf">http://gittup.org/tup/build_system_rules_and_algorithms.pdf</a></li> <li>Build System Partial Updates - <a href="http://gittup.org/blog/2014/09/11-build-system-partial-updates/">http://gittup.org/blog/2014/09/11-build-system-partial-updates/</a></li> </ul> </li> <li>Clobber Builds - Mike Shal <ul> <li>Part 1 - Missing Dependencies - <a href="http://gittup.org/blog/2014/03/6-clobber-builds-part-1---missing-dependencies/">http://gittup.org/blog/2014/03/6-clobber-builds-part-1---missing-dependencies/</a></li> <li>Part 2 - Fixing Missing Dependencies - <a href="http://gittup.org/blog/2014/05/7-clobber-builds-part-2---fixing-missing-dependencies/">http://gittup.org/blog/2014/05/7-clobber-builds-part-2---fixing-missing-dependencies/</a></li> <li>Part 3 - Other Clobber Causes - <a href="http://gittup.org/blog/2014/06/8-clobber-builds-part-3---other-clobber-causes/">http://gittup.org/blog/2014/06/8-clobber-builds-part-3---other-clobber-causes/</a></li> <li>Part 4 - Fixing Other Clobber Causes - <a href="http://gittup.org/blog/2015/03/13-clobber-builds-part-4---fixing-other-clobber-causes/">http://gittup.org/blog/2015/03/13-clobber-builds-part-4---fixing-other-clobber-causes/</a></li> </ul> </li> <li>Correct, Efficient, and Tailored: The Future of Build Systems <ul> <li>IEEE Software, vol. 35, no. 2, 2018</li> <li>G. Maudoux and K. Mens</li> <li><a href="https://doi.ieeecomputersociety.org/10.1109/MS.2018.111095025">https://doi.ieeecomputersociety.org/10.1109/MS.2018.111095025</a></li> </ul> </li> <li>The C++ Build Process Explained <ul> <li><a href="https://github.com/green7ea/cpp-compilation">https://github.com/green7ea/cpp-compilation</a></li> </ul> </li> </ul> <h2>Caching</h2> <ul> <li>Caching Function Calls Using Precise Dependencies <ul> <li>Programming Language Design and Implementation (PLDI) 2000</li> <li>Allan Heydon, Roy Levin, Yuan Yu</li> <li><a href="http://www.vestasys.org/doc/pubs/pldi-00-04-20.pdf">http://www.vestasys.org/doc/pubs/pldi-00-04-20.pdf</a></li> <li>Vesta Configuration Management System - <a href="http://www.vestasys.org/#publications">http://www.vestasys.org/#publications</a></li> </ul> </li> <li>cHash: Detection of Redundant Compilations via AST Hashing <ul> <li>USENIX Annual Technical Conference 2017</li> <li>Christian Dietrich, Valentin Rothberg, Ludwig Füracker, Andreas Ziegler, Daniel Lohmann</li> <li><a href="https://www.usenix.org/conference/atc17/technical-sessions/presentation/dietrich">https://www.usenix.org/conference/atc17/technical-sessions/presentation/dietrich</a></li> <li>cHash Compiler Plugins and related tools - <a href="https://github.com/luhsra/chash">https://github.com/luhsra/chash</a></li> </ul> </li> </ul> <h2>Correctness</h2> <ul> <li>Detecting Incorrect Build Rules <ul> <li>International Conference on Software Engineering (ICSE) 2019</li> <li>Nandor Licker, Andrew Rice</li> <li><a href="https://2019.icse-conferences.org/details/icse-2019-Technical-Papers/82/Detecting-Incorrect-Build-Rules">https://2019.icse-conferences.org/details/icse-2019-Technical-Papers/82/Detecting-Incorrect-Build-Rules</a></li> <li><a href="https://www.repository.cam.ac.uk/handle/1810/288468">https://www.repository.cam.ac.uk/handle/1810/288468</a></li> <li>mkcheck: Incremental Build Verification <ul> <li><a href="https://github.com/nandor/mkcheck">https://github.com/nandor/mkcheck</a></li> </ul> </li> </ul> </li> <li>Oops, My Tests Broke the Build: An Explorative Analysis of Travis CI with GitHub <ul> <li>Mining Software Repositories (MSR) 2017</li> <li>M. Beller, G. Gousios, A. Zaidman</li> <li><a href="http://www.gousios.gr/pub/tests-broke-build-explorative-analysis-travis-ci-github.pdf">http://www.gousios.gr/pub/tests-broke-build-explorative-analysis-travis-ci-github.pdf</a></li> <li><a href="https://www.slideshare.net/inventitech/oops-my-tests-broke-the-build-an-explorative-analysis-of-travis-ci-with-github">https://www.slideshare.net/inventitech/oops-my-tests-broke-the-build-an-explorative-analysis-of-travis-ci-with-github</a></li> </ul> </li> <li>Programmers’ Build Errors: A Case Study (at Google) <ul> <li>International Conference on Software Engineering (ICSE) 2014</li> <li>Hyunmin Seo, Caitlin Sadowski, Sebastian Elbaum, Edward Aftandilian, Robert Bowdidge</li> <li><a href="https://research.google.com/pubs/pub42184.html">https://research.google.com/pubs/pub42184.html</a></li> </ul> </li> </ul> <h2>Dependencies</h2> <ul> <li>Automatic Object Linkage, with Include Graphs (Source code sharing without static libraries) <ul> <li>2018; Thomas Young</li> <li><a href="https://upcoder.com/19/automatic-object-linkage-with-include-graphs">https://upcoder.com/19/automatic-object-linkage-with-include-graphs</a></li> </ul> </li> <li>Build Predictor: More Accurate Missed Dependency Prediction in Build Configuration Files <ul> <li>Computer Software and Applications Conference (COMPSAC 2014)</li> <li>Bo Zhou, Xia Xin, David Lo, Xinyu Wang</li> <li><a href="http://www.mysmu.edu/faculty/davidlo/papers/compsac14-dependency.pdf">http://www.mysmu.edu/faculty/davidlo/papers/compsac14-dependency.pdf</a></li> <li><a href="https://www.semanticscholar.org/paper/Build-Predictor%3A-More-Accurate-Missed-Dependency-in-Zhou-Xia/a4d4b05c8594fc7358a89f0afffb7e405b65fa0d">https://www.semanticscholar.org/paper/Build-Predictor%3A-More-Accurate-Missed-Dependency-in-Zhou-Xia/a4d4b05c8594fc7358a89f0afffb7e405b65fa0d</a></li> </ul> </li> <li>Program Repository <ul> <li>LLVM with Program Repository Support<br> - <a href="https://github.com/SNSystems/llvm-project-prepo">https://github.com/SNSystems/llvm-project-prepo</a></li> <li>Program Repository: What’s the Idea? <ul> <li><a href="https://github.com/SNSystems/llvm-project-prepo/wiki">https://github.com/SNSystems/llvm-project-prepo/wiki</a></li> </ul> </li> <li>Early Overview <ul> <li><a href="https://github.com/SNSystems/llvm-project-prepo/wiki/Early-Overview">https://github.com/SNSystems/llvm-project-prepo/wiki/Early-Overview</a></li> </ul> </li> <li>Toy programming demo of a repository for statically compiled programs <ul> <li>2016 US LLVM Developers' Meeting; Paul Bowen-Huggett</li> <li><a href="https://llvm.org/devmtg/2016-11/Slides/Bowen-Hugett-ToyProgrammingDemo.pdf">https://llvm.org/devmtg/2016-11/Slides/Bowen-Hugett-ToyProgrammingDemo.pdf</a></li> <li><a href="https://youtu.be/-pL94rqyQ6c">https://youtu.be/-pL94rqyQ6c</a></li> </ul> </li> <li>Targeting a statically compiled program repository with LLVM <ul> <li>2019 EuroLLVM Developers’ Meeting; Phil Camp , Russell Gallop</li> <li><a href="https://www.youtube.com/watch?v=mlQyEBDnDJE">https://www.youtube.com/watch?v=mlQyEBDnDJE</a></li> <li><a href="http://llvm.org/devmtg/2019-04/slides/Lightning-Camp-Program_Repo.pdf">http://llvm.org/devmtg/2019-04/slides/Lightning-Camp-Program_Repo.pdf</a></li> </ul> </li> <li>LLVM Build Times Using a Program Repository <ul> <li><a href="https://www.snsystems.com/technology/tech-blog/llvm-build-times-using-a-program-repository">https://www.snsystems.com/technology/tech-blog/llvm-build-times-using-a-program-repository</a></li> </ul> </li> <li>Compile Faster with the Program Repository and ccache <ul> <li>2020 European LLVM Developers Meeting; Ying Yi, Paul Bowen-Huggett</li> <li><a href="http://www.llvm.org/devmtg/2020-04/talks.html#LightningTalk_51">http://www.llvm.org/devmtg/2020-04/talks.html#LightningTalk_51</a></li> </ul> </li> </ul> </li> <li>mtime comparison considered harmful <ul> <li>"tl;dr: Rebuilding a target because its mtime is older than the mtimes of its dependencies, like make does, is very error prone. redo does it better, and so can you."</li> <li><a href="https://apenwarr.ca/log/20181113">https://apenwarr.ca/log/20181113</a></li> </ul> </li> <li>Non-recursive Make Considered Harmful <ul> <li>Proceedings of the 9th International Symposium on Haskell, 2016</li> <li>Andrey Mokhov, Neil Mitchell, Simon Peyton Jones, Simon Marlow</li> <li><a href="https://www.microsoft.com/en-us/research/publication/non-recursive-make-considered-harmful/">https://www.microsoft.com/en-us/research/publication/non-recursive-make-considered-harmful/</a></li> <li><a href="https://simonmar.github.io/bib/shake-2016_abstract.html">https://simonmar.github.io/bib/shake-2016_abstract.html</a></li> </ul> </li> <li>Recursive Make Considered Harmful <ul> <li>Journal of AUUG Inc, 19(1):14–25, 1998</li> <li>Peter Miller</li> <li><a href="http://aegis.sourceforge.net/auug97.pdf">http://aegis.sourceforge.net/auug97.pdf</a></li> <li><a href="http://sites.e-advies.nl/nonrecursive-make.html">http://sites.e-advies.nl/nonrecursive-make.html</a></li> </ul> </li> </ul> <h2>Distributed</h2> <ul> <li>Build in the Cloud <ul> <li>Accessing Source Code: <a href="http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html">http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-source-code.html</a></li> <li>How the Build System works: <a href="http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html">http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-system-works.html</a></li> <li>Distributing Build Steps: <a href="http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributing-build-steps.html">http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributing-build-steps.html</a></li> <li>Distributing Build Outputs: <a href="http://google-engtools.blogspot.com/2011/10/build-in-cloud-distributing-build.html">http://google-engtools.blogspot.com/2011/10/build-in-cloud-distributing-build.html</a></li> </ul> </li> <li>CloudBuild: Microsoft’s Distributed and Caching Build Service <ul> <li>International Conference on Software Engineering (ICSE) 2016</li> <li>Hamed Esfahani, Jonas Fietz, Qi Ke, Alexei Kolomiets, Erica Lan, Erik Mavrinac, Wolfram Schulte, Newton Sanches, Srikanth Kandula</li> <li><a href="https://dl.acm.org/citation.cfm?id=2889222">https://dl.acm.org/citation.cfm?id=2889222</a></li> <li><a href="https://www.microsoft.com/en-us/research/publication/cloudbuild-microsofts-distributed-and-caching-build-service/">https://www.microsoft.com/en-us/research/publication/cloudbuild-microsofts-distributed-and-caching-build-service/</a></li> </ul> </li> <li>From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Transient Functional Containers <ul> <li>2019 USENIX Annual Technical Conference</li> <li>Sadjad Fouladi, Francisco Romero, Dan Iter, Qian Li, Shuvo Chatterjee, Christos Kozyrakis, Matei Zaharia, Keith Winstein</li> <li><a href="https://www.usenix.org/conference/atc19/presentation/fouladi">https://www.usenix.org/conference/atc19/presentation/fouladi</a></li> <li>Outsourcing Everyday Jobs to Thousands of Cloud Functions with gg <ul> <li><a href="https://www.usenix.org/system/files/login/articles/login_fall19_02_fouladi.pdf">https://www.usenix.org/system/files/login/articles/login_fall19_02_fouladi.pdf</a></li> </ul> </li> <li>gg: The Stanford Builder <ul> <li><a href="https://github.com/stanfordsnr/gg">https://github.com/stanfordsnr/gg</a></li> </ul> </li> </ul> </li> </ul> <h2>Incremental Building</h2> <ul> <li>A Sound and Optimal Incremental Build System with Dynamic Dependencies <ul> <li>SPLASH 2015 OOPSLA</li> <li>Sebastian Erdweg, Moritz Lichter, Manuel Weiel</li> <li><a href="https://www.youtube.com/watch?v=QsgLSDMLLTo">https://www.youtube.com/watch?v=QsgLSDMLLTo</a></li> <li><a href="https://2015.splashcon.org/event/oopsla2015-a-sound-and-optimal-incremental-build-system">https://2015.splashcon.org/event/oopsla2015-a-sound-and-optimal-incremental-build-system</a></li> </ul> </li> <li>Bringing Incremental Builds to Continuous Integration <ul> <li>SaTToSE (Seminar Series on Advanced Techniques &amp; Tools for Software Evolution) 2017</li> <li>Guillaume Maudoux and Kim Mens</li> <li><a href="https://dial.uclouvain.be/pr/boreal/object/boreal:189543">https://dial.uclouvain.be/pr/boreal/object/boreal:189543</a></li> <li><a href="http://sattose.wdfiles.com/local--files/2017:schedule/SATToSE_2017_paper_3.pdf">http://sattose.wdfiles.com/local--files/2017:schedule/SATToSE_2017_paper_3.pdf</a></li> </ul> </li> <li>Constructing Hybrid Incremental Compilers for Cross-Module Extensibility with an Internal Build System <ul> <li>‹Programming› 2020</li> <li>Jeff Smits, Gabriël Konat, Eelco Visser</li> <li><a href="https://2020.programming-conference.org/details/programming-2020-papers/16/Constructing-Hybrid-Incremental-Compilers-for-Cross-Module-Extensibility-with-an-Inte">https://2020.programming-conference.org/details/programming-2020-papers/16/Constructing-Hybrid-Incremental-Compilers-for-Cross-Module-Extensibility-with-an-Inte</a></li> </ul> </li> <li>Scalable Incremental Building with Dynamic Task Dependencies <ul> <li>Automated Software Engineering (ASE) 2018</li> <li>Gabriël Konat, Sebastian Erdweg, Eelco Visser</li> <li><a href="https://doi.org/10.1145/3238147.3238196">https://doi.org/10.1145/3238147.3238196</a></li> <li><a href="https://eelcovisser.org/post/306/scalable-incremental-building-with-dynamic-task-dependencies">https://eelcovisser.org/post/306/scalable-incremental-building-with-dynamic-task-dependencies</a></li> <li><a href="https://www.student.informatik.tu-darmstadt.de/~xx00seba/publications/pie-scalable-incremental-build.pdf">https://www.student.informatik.tu-darmstadt.de/~xx00seba/publications/pie-scalable-incremental-build.pdf</a></li> </ul> </li> </ul> <h2>Reproducibility</h2> <ul> <li>An introduction to deterministic builds with C/C++ <ul> <li><a href="https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html">https://blog.conan.io/2019/09/02/Deterministic-builds-with-C-C++.html</a></li> </ul> </li> <li>Automated Localization for Unreproducible Builds <ul> <li>ICSE 2018</li> <li>Zhilei Ren, He Jiang, Jifeng Xuan, Zijiang Yang</li> <li><a href="https://arxiv.org/abs/1803.06766">https://arxiv.org/abs/1803.06766</a></li> <li><a href="https://blog.acolyer.org/2018/06/22/automated-localization-for-unreproducible-builds/">https://blog.acolyer.org/2018/06/22/automated-localization-for-unreproducible-builds/</a></li> </ul> </li> <li>Deterministic builds with Clang and LLD <ul> <li><a href="http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html">http://blog.llvm.org/2019/11/deterministic-builds-with-clang-and-lld.html</a></li> </ul> </li> <li>Reproducible Builds — a set of software development practices that create an independently-verifiable path from source to binary code <ul> <li><a href="https://reproducible-builds.org/">https://reproducible-builds.org/</a></li> </ul> </li> </ul> <hr> <h1>Build Performance</h1> <h2>Build Performance Readings</h2> <ul> <li>2 tips to make your C++ projects compile 3 times faster <ul> <li>Tip #1: Distributing compilation load</li> <li>Tip #2: Using a distcc server container</li> <li><a href="https://developers.redhat.com/blog/2019/05/15/2-tips-to-make-your-c-projects-compile-3-times-faster/">https://developers.redhat.com/blog/2019/05/15/2-tips-to-make-your-c-projects-compile-3-times-faster/</a></li> </ul> </li> <li>30% faster Windows builds with clang-cl and the new <code>/Zc:dllexportInlines-</code> flag <ul> <li><a href="http://blog.llvm.org/2018/11/30-faster-windows-builds-with-clang-cl_14.html">http://blog.llvm.org/2018/11/30-faster-windows-builds-with-clang-cl_14.html</a></li> </ul> </li> <li>Anders Schau Knatten <ul> <li>Another Reason to Avoid #includes in Headers <ul> <li><a href="https://blog.knatten.org/2012/11/09/another-reason-to-avoid-includes-in-headers/">https://blog.knatten.org/2012/11/09/another-reason-to-avoid-includes-in-headers/</a></li> </ul> </li> <li>How to avoid includes in headers <ul> <li><a href="https://blog.knatten.org/2012/11/30/how-to-avoid-includes-in-headers/">https://blog.knatten.org/2012/11/30/how-to-avoid-includes-in-headers/</a></li> </ul> </li> </ul> </li> <li>Aras Pranckevičius <ul> <li>Unreasonable Effectiveness of Profilers (profiling the build system) - <a href="http://aras-p.info/blog/2017/08/08/Unreasonable-Effectiveness-of-Profilers/">http://aras-p.info/blog/2017/08/08/Unreasonable-Effectiveness-of-Profilers/</a></li> <li>Forced Inlining Might Be Slow - <a href="https://aras-p.info/blog/2017/10/09/Forced-Inlining-Might-Be-Slow/">https://aras-p.info/blog/2017/10/09/Forced-Inlining-Might-Be-Slow/</a></li> <li>Best unknown MSVC flag: d2cgsummary - <a href="https://aras-p.info/blog/2017/10/23/Best-unknown-MSVC-flag-d2cgsummary/">https://aras-p.info/blog/2017/10/23/Best-unknown-MSVC-flag-d2cgsummary/</a></li> <li>Slow to Compile Table Initializers - <a href="https://aras-p.info/blog/2017/10/24/Slow-to-Compile-Table-Initializers/">https://aras-p.info/blog/2017/10/24/Slow-to-Compile-Table-Initializers/</a></li> <li>Investigating compile times, and Clang -ftime-report - <a href="http://aras-p.info/blog/2019/01/12/Investigating-compile-times-and-Clang-ftime-report/">http://aras-p.info/blog/2019/01/12/Investigating-compile-times-and-Clang-ftime-report/</a></li> <li>Another cool MSVC flag: /d1reportTime <ul> <li><a href="http://aras-p.info/blog/2019/01/21/Another-cool-MSVC-flag-d1reportTime/">http://aras-p.info/blog/2019/01/21/Another-cool-MSVC-flag-d1reportTime/</a></li> <li>Reports where the compiler frontend spends time.</li> <li>Passing /d1reportTime to the MSVC compiler (cl.exe) will make it print: <ul> <li>Which header files are included (hierarchically), with time taken for each,</li> <li>Which classes are being parsed, with time taken for each,</li> <li>Which functions are being parsed, with time taken for each.</li> </ul> </li> </ul> </li> </ul> </li> <li>Big Project Build Times–Chromium <ul> <li><a href="https://randomascii.wordpress.com/2020/03/30/big-project-build-times-chromium/">https://randomascii.wordpress.com/2020/03/30/big-project-build-times-chromium/</a></li> </ul> </li> <li>C++ Compilation Speed <ul> <li>Walter Bright - DDJ, August 17, 2010</li> <li><a href="https://digitalmars.com/articles/b54.html">https://digitalmars.com/articles/b54.html</a></li> <li><a href="http://www.drdobbs.com/cpp/c-compilation-speed/228701711">http://www.drdobbs.com/cpp/c-compilation-speed/228701711</a></li> </ul> </li> <li>C++ Compilation: Fixing It <ul> <li><a href="http://virtuallyrandom.com/c-compilation-fixing-it/">http://virtuallyrandom.com/c-compilation-fixing-it/</a></li> </ul> </li> <li>Faster C++ builds <ul> <li><a href="http://www.bitsnbites.eu/faster-c-builds/">http://www.bitsnbites.eu/faster-c-builds/</a></li> </ul> </li> <li>Identifying and Understanding Header File Hotspots in C/C++ Build Processes <ul> <li>Automated Software Engineering, Vol. 23, No. 4, 2016</li> <li>Shane McIntosh, Bram Adams, Meiyappan Nagappan, Ahmed E. Hassan</li> <li><a href="http://rebels.ece.mcgill.ca/journalpaper/2015/07/08/identifying-and-understanding-header-file-hotspots-in-c-cpp-build-processes.html">http://rebels.ece.mcgill.ca/journalpaper/2015/07/08/identifying-and-understanding-header-file-hotspots-in-c-cpp-build-processes.html</a></li> </ul> </li> <li>Improving C++ Builds with Split DWARF <ul> <li><a href="https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/">https://www.productive-cpp.com/improving-cpp-builds-with-split-dwarf/</a></li> </ul> </li> <li>Improving Compilation Time of C/C++ Projects <ul> <li><a href="https://interrupt.memfault.com/blog/improving-compilation-times-c-cpp-projects">https://interrupt.memfault.com/blog/improving-compilation-times-c-cpp-projects</a></li> </ul> </li> <li>Investigating C++ compile times <ul> <li>My journey investigating slow compile times in C++ <ul> <li><a href="https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-0">https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-0</a></li> </ul> </li> <li>Useful tools to investigate C++ compile times <ul> <li><a href="https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-1">https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-1</a></li> </ul> </li> <li>Understanding MSBuild to create flame graphs <ul> <li><a href="https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-2">https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-2</a></li> </ul> </li> <li>Improving C++ compile times using flame graphs <ul> <li><a href="https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-3">https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-3</a></li> </ul> </li> <li>Getting data from C++ Build Insights SDK <ul> <li><a href="https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-4">https://metanokid.github.io/coding-scars/investigating-cpp-compile-times-4</a></li> </ul> </li> </ul> </li> <li>Physical Design of The Machinery <ul> <li><a href="http://ourmachinery.com/post/physical-design/">http://ourmachinery.com/post/physical-design/</a></li> </ul> </li> <li>Reducing Build Time through Precompilations for Evolving Large Software <ul> <li>International Conference on Software Maintenance (ICSM) 2005</li> <li>Yu, Yijun; Dayani-Fard, Homayoun; Mylopoulos, John and Andritsos, Periklis</li> <li><a href="http://www.cs.toronto.edu/%7Eperiklis/pubs/icsm05.pdf">http://www.cs.toronto.edu/%7Eperiklis/pubs/icsm05.pdf</a></li> <li><a href="http://oro.open.ac.uk/6944/">http://oro.open.ac.uk/6944/</a></li> </ul> </li> <li>To Unify or Not to Unify: A Case Study on Unified Builds (in WebKit) <ul> <li>Compiler Construction (CC) 2019</li> <li>Takafumi Kubota, Yusuke Suzuki, and Kenji Kono</li> <li><a href="https://doi.org/10.1145/3302516.3307347">https://doi.org/10.1145/3302516.3307347</a></li> </ul> </li> </ul> <h2>Build Performance Software</h2> <ul> <li>Cotire (compile time reducer) <ul> <li>a CMake module that speeds up the build process of CMake based build systems by fully automating techniques as precompiled header usage and single compilation unit builds for C and C++.</li> <li><a href="https://github.com/sakra/cotire">https://github.com/sakra/cotire</a></li> </ul> </li> </ul> <h3>Benchmarking and Profiling</h3> <ul> <li>C++ Compile Health Watchdog <ul> <li>Benchmark and accountability tool for C++ compile-time overhead / compile-time health.</li> <li><a href="https://github.com/Philip-Trettner/cpp-compile-overhead">https://github.com/Philip-Trettner/cpp-compile-overhead</a></li> <li>Results: <a href="https://artificial-mind.net/projects/compile-health/">https://artificial-mind.net/projects/compile-health/</a></li> </ul> </li> <li>Clang -ftime-trace and ftime-trace-granularity=N <ul> <li><a href="http://releases.llvm.org/9.0.0/tools/clang/docs/ReleaseNotes.html#new-compiler-flags">http://releases.llvm.org/9.0.0/tools/clang/docs/ReleaseNotes.html#new-compiler-flags</a></li> <li>Clang Time Trace Feature <ul> <li><a href="https://www.snsystems.com/technology/tech-blog/clang-time-trace-feature">https://www.snsystems.com/technology/tech-blog/clang-time-trace-feature</a></li> </ul> </li> <li>Clang Build Analyzer <ul> <li>Clang build analysis tool using -ftime-trace</li> <li><a href="https://github.com/aras-p/ClangBuildAnalyzer">https://github.com/aras-p/ClangBuildAnalyzer</a></li> <li><a href="https://aras-p.info/blog/2019/09/28/Clang-Build-Analyzer/">https://aras-p.info/blog/2019/09/28/Clang-Build-Analyzer/</a></li> <li>time-trace: timeline / flame chart profiler for Clang <ul> <li><a href="https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/">https://aras-p.info/blog/2019/01/16/time-trace-timeline-flame-chart-profiler-for-Clang/</a></li> </ul> </li> </ul> </li> </ul> </li> <li>CTMark (Compile Time Mark) <ul> <li><a href="https://github.com/llvm-mirror/test-suite/tree/master/CTMark">https://github.com/llvm-mirror/test-suite/tree/master/CTMark</a></li> </ul> </li> <li>Metabench: A simple framework for compile-time microbenchmarks <ul> <li><a href="http://metaben.ch/">http://metaben.ch/</a></li> <li><a href="https://github.com/ldionne/metabench">https://github.com/ldionne/metabench</a></li> </ul> </li> <li>Templight: Template Instantiation Profiler and Debugger <ul> <li>a Clang-based tool to profile the time and memory consumption of template instantiations and to perform interactive debugging sessions to gain introspection into the template instantiation process</li> <li><a href="https://github.com/mikael-s-persson/templight">https://github.com/mikael-s-persson/templight</a></li> <li>Use templight and Templar to debug C++ templates <ul> <li><a href="https://baptiste-wicht.com/posts/2016/02/use-templight-and-templar-to-debug-cpp-templates.html">https://baptiste-wicht.com/posts/2016/02/use-templight-and-templar-to-debug-cpp-templates.html</a></li> </ul> </li> <li>Templight: A Clang Extension for Debugging and Profiling C++ Template Metaprograms <ul> <li>2015 EuroLLVM Developers’ Meeting, Zoltan Porkolab</li> <li><a href="https://llvm.org/devmtg/2015-04/slides/EuroLLVM2015Templight.pdf">https://llvm.org/devmtg/2015-04/slides/EuroLLVM2015Templight.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=djAPtopWhRU">https://www.youtube.com/watch?v=djAPtopWhRU</a></li> </ul> </li> </ul> </li> <li>Visual C++ <ul> <li>vcperf: A build analysis tool for the MSVC toolchain <ul> <li><a href="https://github.com/microsoft/vcperf">https://github.com/microsoft/vcperf</a></li> </ul> </li> <li>MSBuild Flame Graph: Turns MSBuild executions into flame graphs <ul> <li><a href="https://github.com/MetanoKid/msbuild-flame-graph">https://github.com/MetanoKid/msbuild-flame-graph</a></li> </ul> </li> <li>C++ Build Insights <ul> <li>Get started with C++ Build Insights <ul> <li><a href="https://docs.microsoft.com/en-us/cpp/build-insights/get-started-with-cpp-build-insights">https://docs.microsoft.com/en-us/cpp/build-insights/get-started-with-cpp-build-insights</a></li> </ul> </li> <li>Introducing C++ Build Insights <ul> <li><a href="https://devblogs.microsoft.com/cppblog/introducing-c-build-insights/">https://devblogs.microsoft.com/cppblog/introducing-c-build-insights/</a></li> </ul> </li> <li>Analyze your builds programmatically with the C++ Build Insights SDK <ul> <li><a href="https://devblogs.microsoft.com/cppblog/analyze-your-builds-programmatically-with-the-c-build-insights-sdk/">https://devblogs.microsoft.com/cppblog/analyze-your-builds-programmatically-with-the-c-build-insights-sdk/</a></li> </ul> </li> <li>Finding build bottlenecks with C++ Build Insights <ul> <li><a href="https://devblogs.microsoft.com/cppblog/finding-build-bottlenecks-with-cpp-build-insights/">https://devblogs.microsoft.com/cppblog/finding-build-bottlenecks-with-cpp-build-insights/</a></li> </ul> </li> </ul> </li> </ul> </li> </ul> <h3>Caching</h3> <ul> <li>ccache: A Fast C/C++ Compiler Cache <ul> <li><a href="https://ccache.samba.org">https://ccache.samba.org</a></li> </ul> </li> <li>distcc: A free distributed C/C++ compiler system <ul> <li><a href="https://github.com/distcc/distcc">https://github.com/distcc/distcc</a></li> </ul> </li> <li>icecream: Distributed compiler with a central scheduler to share build load <ul> <li><a href="https://github.com/icecc/icecream">https://github.com/icecc/icecream</a></li> </ul> </li> <li>sccache - Shared Compilation Cache <ul> <li><a href="https://github.com/mozilla/sccache">https://github.com/mozilla/sccache</a></li> </ul> </li> </ul> <h3>Dependencies Analysis and Optimization</h3> <ul> <li>cpp-dependencies: Tool to check C++ #include dependencies (dependency graphs created in .dot format) <ul> <li><a href="https://github.com/tomtom-international/cpp-dependencies">https://github.com/tomtom-international/cpp-dependencies</a></li> </ul> </li> <li>Header Hero: optimizing C++ codebase header #include dependencies <ul> <li>A tool for optimizing C++ header files and reducing build times.</li> <li><a href="https://bitbucket.org/bitsquid/header_hero">https://bitbucket.org/bitsquid/header_hero</a></li> <li><a href="https://bitsquid.blogspot.com/2011/10/caring-by-sharing-header-hero.html">https://bitsquid.blogspot.com/2011/10/caring-by-sharing-header-hero.html</a></li> <li><a href="https://aras-p.info/blog/2018/01/17/Header-Hero-Improvements/">https://aras-p.info/blog/2018/01/17/Header-Hero-Improvements/</a></li> </ul> </li> <li>include-what-you-use <ul> <li>A tool for use with clang to analyze #includes in C and C++ source files</li> <li><a href="https://include-what-you-use.org/">https://include-what-you-use.org/</a></li> <li><a href="https://github.com/include-what-you-use/include-what-you-use">https://github.com/include-what-you-use/include-what-you-use</a></li> </ul> </li> </ul> <h2>Build Performance Talks</h2> <ul> <li>Common-sense acceleration of your MLOC build <ul> <li>CppCon 2014; Matt Hargett</li> <li><a href="https://www.youtube.com/watch?v=t4M3yG1dWho">https://www.youtube.com/watch?v=t4M3yG1dWho</a></li> </ul> </li> <li>Compiling C++ is slow - let's go faster <ul> <li>StockholmCpp 2020; Tobias Hieta</li> <li><a href="https://www.youtube.com/watch?v=X4pyOtawqjg">https://www.youtube.com/watch?v=X4pyOtawqjg</a></li> <li><a href="https://gist.github.com/tru/ad677ba2b1d543e238b0e09eb3aad3ac">https://gist.github.com/tru/ad677ba2b1d543e238b0e09eb3aad3ac</a></li> </ul> </li> <li>LLVM Compile-Time: Challenges. Improvements. Outlook. <ul> <li>2017 LLVM Developers’ Meeting; Michael Zolotukhin</li> <li><a href="https://www.youtube.com/watch?v=bYHMwyyZ6Mk">https://www.youtube.com/watch?v=bYHMwyyZ6Mk</a></li> </ul> </li> <li>Optimizing builds on Windows <ul> <li>2019 LLVM Developers’ Meeting; Alexandre Ganea</li> <li><a href="https://www.youtube.com/watch?v=usPL_DROn4k">https://www.youtube.com/watch?v=usPL_DROn4k</a></li> <li><a href="https://llvm.org/devmtg/2019-10/slides/Ganea-OptimizingBuildesOnWindows.pdf">https://llvm.org/devmtg/2019-10/slides/Ganea-OptimizingBuildesOnWindows.pdf</a></li> </ul> </li> <li>Optimizing compilation times with Templates <ul> <li>CppCon 2017: Eddie Elizondo</li> <li><a href="https://www.youtube.com/watch?v=NPWQ7xKfIHQ">https://www.youtube.com/watch?v=NPWQ7xKfIHQ</a></li> <li><a href="https://github.com/eduardo-elizondo/cppcon2017">https://github.com/eduardo-elizondo/cppcon2017</a></li> </ul> </li> <li>Practical Techniques for Improving C++ Build Times <ul> <li>CppCon 2017; Dmitry Panin</li> <li><a href="https://www.youtube.com/watch?v=h8UoYG4dvH8">https://www.youtube.com/watch?v=h8UoYG4dvH8</a></li> </ul> </li> <li>The Hitchhiker's Guide to Faster Builds <ul> <li>Viktor Kirilov</li> <li>slides: <a href="https://slides.com/onqtam/faster_builds">https://slides.com/onqtam/faster_builds</a></li> <li>CppOnSea 2019 - <a href="https://www.youtube.com/watch?v=anbOy47fBYI">https://www.youtube.com/watch?v=anbOy47fBYI</a></li> <li>C++ Russia 2019 - <a href="https://www.youtube.com/watch?v=5rRLHRRqg5A">https://www.youtube.com/watch?v=5rRLHRRqg5A</a></li> <li>code::dive 2018 <ul> <li>Part 1: <a href="https://www.youtube.com/watch?v=WSFbNhCbdJM">https://www.youtube.com/watch?v=WSFbNhCbdJM</a></li> <li>Part 2: <a href="https://www.youtube.com/watch?v=9-Y5JnJlypM">https://www.youtube.com/watch?v=9-Y5JnJlypM</a></li> </ul> </li> </ul> </li> <li>Visual Studio <ul> <li>clcache.py - a compiler cache for Microsoft Visual Studio <ul> <li><a href="https://github.com/frerich/clcache">https://github.com/frerich/clcache</a></li> </ul> </li> <li>CompileTimer: Set of tests to benchmark the compile time of C++ constructs <ul> <li><a href="https://github.com/janwilmans/CompileTimer">https://github.com/janwilmans/CompileTimer</a></li> </ul> </li> <li>Make VC++ Compiles Fast Through Parallel Compilation <ul> <li><a href="https://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/">https://randomascii.wordpress.com/2014/03/22/make-vc-compiles-fast-through-parallel-compilation/</a></li> </ul> </li> <li>Visual Studio 2017 Throughput Improvements and Advice <ul> <li><a href="https://blogs.msdn.microsoft.com/vcblog/2018/01/04/visual-studio-2017-throughput-improvements-and-advice/">https://blogs.msdn.microsoft.com/vcblog/2018/01/04/visual-studio-2017-throughput-improvements-and-advice/</a></li> </ul> </li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>Build System Shootout: Comparison of build program expressive power <ul> <li><a href="https://github.com/ndmitchell/build-shootout">https://github.com/ndmitchell/build-shootout</a></li> </ul> </li> <li>TraceCode: Trace a build to find out which source files are built in a binary <ul> <li><a href="https://github.com/nexB/tracecode-toolkit">https://github.com/nexB/tracecode-toolkit</a></li> <li>Debug your build by tracing and reversing: stracing your build from sources to binaries <ul> <li><a href="https://fosdem.org/2018/schedule/event/debugging_tools_stracing_build/">https://fosdem.org/2018/schedule/event/debugging_tools_stracing_build/</a></li> </ul> </li> </ul> </li> </ul> <hr> <h1>Talks</h1> <h2>2019</h2> <ul> <li>Behind the Scenes of a C++ Build System <ul> <li>CppCon 2019; Jussi Pakkanen</li> <li><a href="https://www.youtube.com/watch?v=34KzT2yvQuM">https://www.youtube.com/watch?v=34KzT2yvQuM</a></li> <li><a href="https://github.com/CppCon/CppCon2019/tree/master/Presentations/behind_the_scenes_of_a_cpp_build_system">https://github.com/CppCon/CppCon2019/tree/master/Presentations/behind_the_scenes_of_a_cpp_build_system</a></li> </ul> </li> <li>Building Modules <ul> <li>CppCon 2019; Michael Spencer</li> <li><a href="https://www.youtube.com/watch?v=L0SHHkBenss">https://www.youtube.com/watch?v=L0SHHkBenss</a></li> </ul> </li> </ul> <h2>2018</h2> <ul> <li>Build Systems: a Simple Solution to a Complicated Problem <ul> <li>CppCon 2018; Peter Bindels</li> <li><a href="https://www.youtube.com/watch?v=mWOmkwv8N_U">https://www.youtube.com/watch?v=mWOmkwv8N_U</a></li> </ul> </li> <li>Creating the Complete Build Package <ul> <li>CppCon 2018</li> <li>Panelists: Boris Kolpackov, Titus Winter, Robert Schumacher, Paddy McDonald, Manuel Klimek</li> <li><a href="https://www.youtube.com/watch?v=TjdCxXdjaSA">https://www.youtube.com/watch?v=TjdCxXdjaSA</a></li> </ul> </li> <li>Don't package your libraries, write packagable libraries! <ul> <li>CppCon 2018; Robert Schumacher</li> <li><a href="https://www.youtube.com/watch?v=sBP17HQAQjk">https://www.youtube.com/watch?v=sBP17HQAQjk</a></li> </ul> </li> <li>What to Expect from a Next-Generation C++ Build System <ul> <li>CppCon 2018; Boris Kolpackov</li> <li><a href="https://www.youtube.com/watch?v=cJP7SSLjvSI">https://www.youtube.com/watch?v=cJP7SSLjvSI</a></li> </ul> </li> </ul> <h2>2016</h2> <ul> <li>A New Architecture for Building Software <ul> <li>2016 LLVM Developers’ Meeting; Daniel Dunbar</li> <li><a href="https://www.youtube.com/watch?v=b_T-eCToX1I">https://www.youtube.com/watch?v=b_T-eCToX1I</a></li> <li><a href="http://www.llvm.org/devmtg/2016-11/#talk10">http://www.llvm.org/devmtg/2016-11/#talk10</a></li> <li>llbuild: A low-level build system, used by Xcode and the Swift Package Manager <ul> <li><a href="https://github.com/apple/swift-llbuild">https://github.com/apple/swift-llbuild</a></li> </ul> </li> </ul> </li> </ul> <h2>2017</h2> <ul> <li>Building C++ Modules <ul> <li>CppCon 2017; Boris Kolpackov</li> <li><a href="https://www.youtube.com/watch?v=E8EbDcLQAoc">https://www.youtube.com/watch?v=E8EbDcLQAoc</a></li> </ul> </li> <li>There Will Be Build Systems: I Configure Your Milkshake <ul> <li>CppCon 2017; Isabella Muerte</li> <li><a href="https://www.youtube.com/watch?v=7THzO-D0ta4">https://www.youtube.com/watch?v=7THzO-D0ta4</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/calca/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/calca/index.html <p><a href="https://calca.io/">Website</a> | Commercial?</p> https://tedneward.github.io/Research/tools/checker/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/checker/index.html <p><a href="https://checkerframework.org/">Website</a> | <a href="https://github.com/typetools/checker-framework/">Source</a> | <a href="https://checkerframework.org/manual/checker-framework-manual.pdf">Doc (PDF)</a> | <a href="https://checkerframework.org/tutorial/">Tutorials</a></p> <ul> <li><a href="https://checkerframework.org/manual/checker-framework-dataflow-manual.pdf">Dataflow Manual</a></li> <li><a href="https://checkerframework.org/annotation-file-utilities/">Annotation File Utilities</a></li> </ul> https://tedneward.github.io/Research/tools/clang/static-analysis/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/clang/static-analysis/index.html <ul> <li> <p>Clang Static Analyzer</p> <ul> <li><a href="http://clang-analyzer.llvm.org/">http://clang-analyzer.llvm.org/</a></li> <li>Checker Developer Manual <ul> <li><a href="http://clang-analyzer.llvm.org/checker_dev_manual.html">http://clang-analyzer.llvm.org/checker_dev_manual.html</a></li> </ul> </li> <li>Clang Static Analysis <ul> <li>Meeting C++ 2016; Gabor Horvath</li> <li><a href="https://www.youtube.com/watch?v=UcxF6CVueDM">https://www.youtube.com/watch?v=UcxF6CVueDM</a></li> </ul> </li> <li>Clang Static Analyzer - A Checker Developer's Guide. <ul> <li><a href="https://github.com/haoNoQ/clang-analyzer-guide">https://github.com/haoNoQ/clang-analyzer-guide</a></li> </ul> </li> <li>Faster, Stronger C++ Analysis with the Clang Static Analyzer <ul> <li>2018 LLVM Developers’ Meeting; George Karpenkov, Artem Dergachev</li> <li><a href="https://www.youtube.com/watch?v=4n3l-ZcDJNY">https://www.youtube.com/watch?v=4n3l-ZcDJNY</a></li> </ul> </li> </ul> </li> <li> <p>Clang-Tidy: Clang-based C++ “linter” tool</p> <ul> <li><a href="http://clang.llvm.org/extra/clang-tidy/">http://clang.llvm.org/extra/clang-tidy/</a></li> <li>Clang-Tidy - KDAB <ul> <li>part 1: Modernize your source code using C++11/C++14 - <a href="https://www.kdab.com/clang-tidy-part-1-modernize-source-code-using-c11c14/">https://www.kdab.com/clang-tidy-part-1-modernize-source-code-using-c11c14/</a></li> <li>part 2: Integrate qmake and other build systems using Bear - <a href="https://www.kdab.com/clang-tidy-part-2-integrate-qmake-and-other-build-systems-using-bear/">https://www.kdab.com/clang-tidy-part-2-integrate-qmake-and-other-build-systems-using-bear/</a></li> </ul> </li> <li>How to use Clang Tidy to automatically correct code <ul> <li><a href="https://github.com/KratosMultiphysics/Kratos/wiki/How-to-use-Clang-Tidy-to-automatically-correct-code">https://github.com/KratosMultiphysics/Kratos/wiki/How-to-use-Clang-Tidy-to-automatically-correct-code</a></li> </ul> </li> <li>Intro to clang-tidy - C++ Weekly - Ep 3 <ul> <li><a href="https://www.youtube.com/watch?v=OchPaGEH4TE">https://www.youtube.com/watch?v=OchPaGEH4TE</a></li> </ul> </li> </ul> </li> <li> <p>Clang Thread Safety Analysis</p> <ul> <li><a href="https://clang.llvm.org/docs/ThreadSafetyAnalysis.html">https://clang.llvm.org/docs/ThreadSafetyAnalysis.html</a></li> <li>C/C++ Thread Safety Analysis <ul> <li>DeLesley Hutchins, Aaron Ballman, Dean Sutherland</li> <li>2014 IEEE 14th International Working Conference on Source Code Analysis and Manipulation</li> <li><a href="https://research.google/pubs/pub42958/">https://research.google/pubs/pub42958/</a></li> </ul> </li> <li>Thread Safety Analysis in C and C++ <ul> <li><a href="https://insights.sei.cmu.edu/sei_blog/2014/10/thread-safety-analysis-in-c-and-c.html">https://insights.sei.cmu.edu/sei_blog/2014/10/thread-safety-analysis-in-c-and-c.html</a></li> </ul> </li> <li>Thread Safety Annotations in Clang <ul> <li>DeLesley Hutchins</li> <li>2011 LLVM Developers' Meeting; <a href="https://llvm.org/devmtg/2011-11/Hutchins_ThreadSafety.pdf">https://llvm.org/devmtg/2011-11/Hutchins_ThreadSafety.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=5Xx0zktqSs4">https://www.youtube.com/watch?v=5Xx0zktqSs4</a></li> </ul> </li> </ul> </li> </ul> https://tedneward.github.io/Research/tools/cmdpixl/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cmdpixl/index.html <p><a href="https://github.com/knosmos/cmdpxl">Source</a></p> <p>Written in Python; install with <code>pip install cmdpxl</code>.</p> https://tedneward.github.io/Research/tools/codechecker/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/codechecker/index.html <p><a href="https://github.com/Ericsson/codechecker">Source</a></p> https://tedneward.github.io/Research/tools/compilerexplorer/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/compilerexplorer/index.html <p><a href="https://godbolt.org/">Website</a> | <a href="https://github.com/compiler-explorer/compiler-explorer">Source</a></p> <p>Edit code in C, C++, C#, F#, Rust, Go, D, Haskell, Swift, Pascal, ispc, Python, Java, or any of the other 30+ supported languages components, and see how that code looks after being compiled in real time.</p> <p>Articles:</p> <ul> <li><a href="https://android-developers.googleblog.com/2024/09/become-better-android-developer-compiler-explorer.html">"Tools not Rules"</a></li> </ul> https://tedneward.github.io/Research/tools/crab/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/crab/index.html <p><a href="https://github.com/seahorn/crab">Source</a></p> <p><a href="https://github.com/seahorn/crab-llvm">Crab-llvm</a>: Abstract Interpretation of LLVM bitcode</p> https://tedneward.github.io/Research/tools/ampt/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/ampt/index.html <p><a href="https://www.getampt.com/">Website</a> | <a href="https://github.com/getampt">Source</a></p> <p>Ampt automatically provisions, manages, and optimizes cloud environments on AWS. They are fully-managed, highly-secure, completely isolated, and come pre-configured with all the cloud resources you need to deploy and scale modern apps.</p> <pre><code>// Import your favorite Node.js libraries and the Ampt SDK import axios from 'axios'; import { task } from '@ampt/sdk'; // Write your business logic const fetchData = async () =&gt; { const response = await axios.get('https://api.my-saas.com'); // Do something with this data } // Create a 'task' handler const myTask = task('Fetch Sample Data', fetchData) // Schedule it to run every hour myTask.every("1 hour") </code></pre> https://tedneward.github.io/Research/tools/anythingllm/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/anythingllm/index.html <p><a href="https://anythingllm.com/">Website</a> | <a href="https://github.com/Mintplex-Labs/anything-llm">Source</a> | <a href="https://docs.anythingllm.com/">Docs</a> | <a href="https://anythingllm.com/download">Download</a></p> <p>A full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as a reference during chatting. This application allows you to pick and choose which LLM or Vector Database you want to use as well as supporting multi-user management and permissions.</p> <p><img src="https://github.com/Mintplex-Labs/anything-llm/assets/16845892/cfc5f47c-bd91-4067-986c-f3f49621a859" alt="Chatting"></p> <h3>Product Overview</h3> <p>AnythingLLM is a full-stack application where you can use commercial off-the-shelf LLMs or popular open source LLMs and vectorDB solutions to build a private ChatGPT with no compromises that you can run locally as well as host remotely and be able to chat intelligently with any documents you provide it.</p> <p>AnythingLLM divides your documents into objects called <code>workspaces</code>. A Workspace functions a lot like a thread, but with the addition of containerization of your documents. Workspaces can share documents, but they do not talk to each other so you can keep your context for each workspace clean.</p> <h2>Cool features of AnythingLLM</h2> <ul> <li>🆕 <a href="https://docs.anythingllm.com/mcp-compatibility/overview"><strong>Full MCP-compatibility</strong></a></li> <li>🆕 <a href="https://docs.anythingllm.com/agent-flows/overview"><strong>No-code AI Agent builder</strong></a></li> <li>🖼️ <strong>Multi-modal support (both closed and open-source LLMs!)</strong></li> <li><a href="https://docs.anythingllm.com/agent/custom/introduction"><strong>Custom AI Agents</strong></a></li> <li>👤 Multi-user instance support and permissioning <em>Docker version only</em></li> <li>🦾 Agents inside your workspace (browse the web, etc)</li> <li>💬 <a href="https://github.com/Mintplex-Labs/anythingllm-embed/blob/main/README.md">Custom Embeddable Chat widget for your website</a> <em>Docker version only</em></li> <li>📖 Multiple document type support (PDF, TXT, DOCX, etc)</li> <li>Simple chat UI with Drag-n-Drop functionality and clear citations.</li> <li>100% Cloud deployment ready.</li> <li>Works with all popular <a href="#supported-llms-embedder-models-speech-models-and-vector-databases">closed and open-source LLM providers</a>.</li> <li>Built-in cost &amp; time-saving measures for managing very large documents compared to any other chat UI.</li> <li>Full Developer API for custom integrations!</li> <li>Much more...install and find out!</li> </ul> <h3>Supported LLMs, Embedder Models, Speech models, and Vector Databases</h3> <p><strong>Large Language Models (LLMs):</strong></p> <ul> <li><a href="/server/storage/models/README.md#text-generation-llm-selection">Any open-source llama.cpp compatible model</a></li> <li><a href="https://openai.com">OpenAI</a></li> <li><a href="https://openai.com">OpenAI (Generic)</a></li> <li><a href="https://azure.microsoft.com/en-us/products/ai-services/openai-service">Azure OpenAI</a></li> <li><a href="https://aws.amazon.com/bedrock/">AWS Bedrock</a></li> <li><a href="https://www.anthropic.com/">Anthropic</a></li> <li><a href="https://build.nvidia.com/explore/discover">NVIDIA NIM (chat models)</a></li> <li><a href="https://ai.google.dev/">Google Gemini Pro</a></li> <li><a href="https://huggingface.co/">Hugging Face (chat models)</a></li> <li><a href="https://ollama.ai/">Ollama (chat models)</a></li> <li><a href="https://lmstudio.ai">LM Studio (all models)</a></li> <li><a href="https://localai.io/">LocalAI (all models)</a></li> <li><a href="https://www.together.ai/">Together AI (chat models)</a></li> <li><a href="https://fireworks.ai/">Fireworks AI (chat models)</a></li> <li><a href="https://www.perplexity.ai/">Perplexity (chat models)</a></li> <li><a href="https://openrouter.ai/">OpenRouter (chat models)</a></li> <li><a href="https://deepseek.com/">DeepSeek (chat models)</a></li> <li><a href="https://mistral.ai/">Mistral</a></li> <li><a href="https://groq.com/">Groq</a></li> <li><a href="https://cohere.com/">Cohere</a></li> <li><a href="https://github.com/LostRuins/koboldcpp">KoboldCPP</a></li> <li><a href="https://github.com/BerriAI/litellm">LiteLLM</a></li> <li><a href="https://github.com/oobabooga/text-generation-webui">Text Generation Web UI</a></li> <li><a href="https://apipie.ai/">Apipie</a></li> <li><a href="https://x.ai/">xAI</a></li> <li><a href="https://z.ai/model-api">Z.AI (chat models)</a></li> <li><a href="https://novita.ai/model-api/product/llm-api?utm_source=github_anything-llm&amp;utm_medium=github_readme&amp;utm_campaign=link">Novita AI (chat models)</a></li> <li><a href="https://ppinfra.com?utm_source=github_anything-llm">PPIO</a></li> <li><a href="https://ai.gitee.com/">Gitee AI</a></li> <li><a href="https://www.moonshot.ai/">Moonshot AI</a></li> <li><a href="https://github.com/microsoft/Foundry-Local">Microsoft Foundry Local</a></li> <li><a href="https://api.cometapi.com/">CometAPI (chat models)</a></li> <li><a href="https://docs.docker.com/ai/model-runner/">Docker Model Runner</a></li> <li><a href="https://privatemode.ai/">PrivateModeAI (chat models)</a></li> <li><a href="https://cloud.sambanova.ai/">SambaNova Cloud (chat models)</a></li> <li><a href="https://lemonade-server.ai">Lemonade by AMD</a></li> </ul> <p><strong>Embedder models:</strong></p> <ul> <li><a href="/server/storage/models/README.md">AnythingLLM Native Embedder</a> (default)</li> <li><a href="https://openai.com">OpenAI</a></li> <li><a href="https://azure.microsoft.com/en-us/products/ai-services/openai-service">Azure OpenAI</a></li> <li><a href="https://localai.io/">LocalAI (all)</a></li> <li><a href="https://ollama.ai/">Ollama (all)</a></li> <li><a href="https://lmstudio.ai">LM Studio (all)</a></li> <li><a href="https://cohere.com/">Cohere</a></li> </ul> <p><strong>Audio Transcription models:</strong></p> <ul> <li><a href="https://github.com/Mintplex-Labs/anything-llm/tree/master/server/storage/models#audiovideo-transcription">AnythingLLM Built-in</a> (default)</li> <li><a href="https://openai.com/">OpenAI</a></li> </ul> <p><strong>TTS (text-to-speech) support:</strong></p> <ul> <li>Native Browser Built-in (default)</li> <li><a href="https://github.com/rhasspy/piper">PiperTTSLocal - runs in browser</a></li> <li><a href="https://platform.openai.com/docs/guides/text-to-speech/voice-options">OpenAI TTS</a></li> <li><a href="https://elevenlabs.io/">ElevenLabs</a></li> <li>Any OpenAI Compatible TTS service.</li> </ul> <p><strong>STT (speech-to-text) support:</strong></p> <ul> <li>Native Browser Built-in (default)</li> </ul> <p><strong>Vector Databases:</strong></p> <ul> <li><a href="https://github.com/lancedb/lancedb">LanceDB</a> (default)</li> <li><a href="https://github.com/pgvector/pgvector">PGVector</a></li> <li><a href="https://www.datastax.com/products/datastax-astra">Astra DB</a></li> <li><a href="https://pinecone.io">Pinecone</a></li> <li><a href="https://trychroma.com">Chroma &amp; ChromaCloud</a></li> <li><a href="https://weaviate.io">Weaviate</a></li> <li><a href="https://qdrant.tech">Qdrant</a></li> <li><a href="https://milvus.io">Milvus</a></li> <li><a href="https://zilliz.com">Zilliz</a></li> </ul> <h3>Technical Overview</h3> <p>This monorepo consists of six main sections:</p> <ul> <li><code>frontend</code>: A viteJS + React frontend that you can run to easily create and manage all your content the LLM can use.</li> <li><code>server</code>: A NodeJS express server to handle all the interactions and do all the vectorDB management and LLM interactions.</li> <li><code>collector</code>: NodeJS express server that processes and parses documents from the UI.</li> <li><code>docker</code>: Docker instructions and build process + information for building from source.</li> <li><code>embed</code>: Submodule for generation &amp; creation of the <a href="https://github.com/Mintplex-Labs/anythingllm-embed">web embed widget</a>.</li> <li><code>browser-extension</code>: Submodule for the <a href="https://github.com/Mintplex-Labs/anythingllm-extension">chrome browser extension</a>.</li> </ul> <h2>🛳 Self-Hosting</h2> <p>Mintplex Labs &amp; the community maintain a number of deployment methods, scripts, and templates that you can use to run AnythingLLM locally. Refer to the table below to read how to deploy on your preferred environment or to automatically deploy.<br> | Docker | AWS | GCP | Digital Ocean | Render.com |<br> |----------------------------------------|----|-----|---------------|------------|<br> | <a href="./docker/HOW_TO_USE_DOCKER.md"><img src="https://tedneward.github.io/Research/tools/images/deployBtns/docker.png" alt="Deploy on Docker"></a> | <a href="./cloud-deployments/aws/cloudformation/DEPLOY.md"><img src="https://tedneward.github.io/Research/tools/images/deployBtns/aws.png" alt="Deploy on AWS"></a> | <a href="./cloud-deployments/gcp/deployment/DEPLOY.md"><img src="https://deploy.cloud.run/button.svg" alt="Deploy on GCP"></a> | <a href="./cloud-deployments/digitalocean/terraform/DEPLOY.md"><img src="https://www.deploytodo.com/do-btn-blue.svg" alt="Deploy on DigitalOcean"></a> | <a href="https://render.com/deploy?repo=https://github.com/Mintplex-Labs/anything-llm&amp;branch=render"><img src="https://render.com/images/deploy-to-render-button.svg" alt="Deploy on Render.com"></a> |</p> <table> <thead> <tr> <th> Railway </th> <th> RepoCloud </th> <th> Elestio </th> <th> Northflank </th> </tr> </thead> <tbody> <tr> <td> <a href="https://railway.app/template/HNSCS1?referralCode=WFgJkn"><img src="https://railway.app/button.svg" alt="Deploy on Railway"></a> </td> <td> <a href="https://repocloud.io/details/?app_id=276"><img src="https://d16t0pc4846x52.cloudfront.net/deploylobe.svg" alt="Deploy on RepoCloud"></a> </td> <td> <a href="https://elest.io/open-source/anythingllm"><img src="https://elest.io/images/logos/deploy-to-elestio-btn.png" alt="Deploy on Elestio"></a> </td> <td> <a href="https://northflank.com/stacks/deploy-anythingllm"><img src="https://assets.northflank.com/deploy_to_northflank_smm_36700fb050.svg" alt="Deploy on Northflank"></a> </td> </tr> </tbody> </table> <p><a href="./BARE_METAL.md">or set up a production AnythingLLM instance without Docker →</a></p> <h2>How to setup for development</h2> <ul> <li><code>yarn setup</code> To fill in the required <code>.env</code> files you'll need in each of the application sections (from root of repo).</li> <li>Go fill those out before proceeding. Ensure <code>server/.env.development</code> is filled or else things won't work right.</li> <li><code>yarn dev:server</code> To boot the server locally (from root of repo).</li> <li><code>yarn dev:frontend</code> To boot the frontend locally (from root of repo).</li> <li><code>yarn dev:collector</code> To then run the document collector (from root of repo).</li> </ul> <p><a href="./server/storage/documents/DOCUMENTS.md">Learn about documents</a></p> <p><a href="./server/storage/vector-cache/VECTOR_CACHE.md">Learn about vector caching</a></p> <h2>External Apps &amp; Integrations</h2> <p><em>These are apps that are not maintained by Mintplex Labs, but are compatible with AnythingLLM. A listing here is not an endorsement.</em></p> <ul> <li><a href="https://io.midori-ai.xyz/subsystem/anythingllm/">Midori AI Subsystem Manager</a> - A streamlined and efficient way to deploy AI systems using Docker container technology.</li> <li><a href="https://coolify.io/docs/services/anythingllm/">Coolify</a> - Deploy AnythingLLM with a single click.</li> <li><a href="https://gptlocalhost.com/demo/">GPTLocalhost for Microsoft Word</a> - A local Word Add-in for you to use AnythingLLM in Microsoft Word.</li> </ul> <h2>Telemetry &amp; Privacy</h2> <p>AnythingLLM by Mintplex Labs Inc contains a telemetry feature that collects anonymous usage information.</p> <details> <summary><kbd>More about Telemetry &amp; Privacy for AnythingLLM</kbd></summary> <h3>Why?</h3> <p>We use this information to help us understand how AnythingLLM is used, to help us prioritize work on new features and bug fixes, and to help us improve AnythingLLM's performance and stability.</p> <h3>Opting out</h3> <p>Set <code>DISABLE_TELEMETRY</code> in your server or docker .env settings to "true" to opt out of telemetry. You can also do this in-app by going to the sidebar &gt; <code>Privacy</code> and disabling telemetry.</p> <h3>What do you explicitly track?</h3> <p>We will only track usage details that help us make product and roadmap decisions, specifically:</p> <ul> <li> <p>Type of your installation (Docker or Desktop)</p> </li> <li> <p>When a document is added or removed. No information <em>about</em> the document. Just that the event occurred. This gives us an idea of use.</p> </li> <li> <p>Type of vector database in use. This helps us prioritize changes when updates arrive for that provider.</p> </li> <li> <p>Type of LLM provider &amp; model tag in use. This helps us prioritize changes when updates arrive for that provider or model, or combination thereof. eg: reasoning vs regular, multi-modal models, etc.</p> </li> <li> <p>When a chat is sent. This is the most regular "event" and gives us an idea of the daily-activity of this project across all installations. Again, only the <strong>event</strong> is sent - we have no information on the nature or content of the chat itself.</p> </li> </ul> <p>You can verify these claims by finding all locations <code>Telemetry.sendTelemetry</code> is called. Additionally these events are written to the output log so you can also see the specific data which was sent - if enabled. <strong>No IP or other identifying information is collected</strong>. The Telemetry provider is <a href="https://posthog.com/">PostHog</a> - an open-source telemetry collection service.</p> <p>We take privacy very seriously, and we hope you understand that we want to learn how our tool is used, without using annoying popup surveys, so we can build something worth using. The anonymous data is <em>never</em> shared with third parties, ever.</p> <p>[View all telemetry events in source code](<a href="https://github.com/search?q=repo%3AMintplex-Labs%2Fanything-llm%20.sendTelemetry(&amp;type=code)">https://github.com/search?q=repo%3AMintplex-Labs%2Fanything-llm%20.sendTelemetry(&amp;type=code)</a></p> </details> <h2>👋 Contributing</h2> <ul> <li><a href="./CONTRIBUTING.md">Contributing to AnythingLLM</a> - How to contribute to AnythingLLM.</li> </ul> <h2>💖 Sponsors</h2> <h3>Premium Sponsors</h3> <!-- premium-sponsors (reserved for $100/mth sponsors who request to be called out here and/or are non-private sponsors) --> <a href="https://www.dcsdigital.co.uk" target="_blank"> <img src="https://a8cforagenciesportfolio.wordpress.com/wp-content/uploads/2024/08/logo-image-232621379.png" height="100px" alt="User avatar: DCS DIGITAL"> </a> <!-- premium-sponsors --> <h3>All Sponsors</h3> <!-- all-sponsors --><a href="https://github.com/jaschadub"><img src="https://github.com/jaschadub.png" width="60px" alt="User avatar: Jascha"></a><a href="https://github.com/KickingAss2024"><img src="https://github.com/KickingAss2024.png" width="60px" alt="User avatar: KickAss"></a><a href="https://github.com/ShadowArcanist"><img src="https://github.com/ShadowArcanist.png" width="60px" alt="User avatar: ShadowArcanist"></a><a href="https://github.com/AtlasVIA"><img src="https://github.com/AtlasVIA.png" width="60px" alt="User avatar: Atlas"></a><a href="https://github.com/cope"><img src="https://github.com/cope.png" width="60px" alt="User avatar: Predrag Stojadinović"></a><a href="https://github.com/DiegoSpinola"><img src="https://github.com/DiegoSpinola.png" width="60px" alt="User avatar: Diego Spinola"></a><a href="https://github.com/PortlandKyGuy"><img src="https://github.com/PortlandKyGuy.png" width="60px" alt="User avatar: Kyle"></a><a href="https://github.com/peperunas"><img src="https://github.com/peperunas.png" width="60px" alt="User avatar: Giulio De Pasquale"></a><a href="https://github.com/jasoncdavis0"><img src="https://github.com/jasoncdavis0.png" width="60px" alt="User avatar: "></a><a href="https://github.com/macstadium"><img src="https://github.com/macstadium.png" width="60px" alt="User avatar: MacStadium"></a><a href="https://github.com/armlynobinguar"><img src="https://github.com/armlynobinguar.png" width="60px" alt="User avatar: "></a><a href="https://github.com/MikeHago"><img src="https://github.com/MikeHago.png" width="60px" alt="User avatar: "></a><a href="https://github.com/maaisde"><img src="https://github.com/maaisde.png" width="60px" alt="User avatar: "></a><a href="https://github.com/mhollier117"><img src="https://github.com/mhollier117.png" width="60px" alt="User avatar: "></a><a href="https://github.com/pleabargain"><img src="https://github.com/pleabargain.png" width="60px" alt="User avatar: Dennis"></a><a href="https://github.com/broichan"><img src="https://github.com/broichan.png" width="60px" alt="User avatar: Michael Hamilton, Ph.D."></a><a href="https://github.com/azim-charaniya"><img src="https://github.com/azim-charaniya.png" width="60px" alt="User avatar: "></a><a href="https://github.com/gabriellemon"><img src="https://github.com/gabriellemon.png" width="60px" alt="User avatar: TernaryLabs"></a><a href="https://github.com/CelaDaniel"><img src="https://github.com/CelaDaniel.png" width="60px" alt="User avatar: Daniel Cela"></a><a href="https://github.com/altrsadmin"><img src="https://github.com/altrsadmin.png" width="60px" alt="User avatar: Alesso"></a><a href="https://github.com/bitjungle"><img src="https://github.com/bitjungle.png" width="60px" alt="User avatar: Rune Mathisen"></a><a href="https://github.com/pcrossleyAC"><img src="https://github.com/pcrossleyAC.png" width="60px" alt="User avatar: "></a><a href="https://github.com/saroj-pattnaik"><img src="https://github.com/saroj-pattnaik.png" width="60px" alt="User avatar: "></a><a href="https://github.com/techmedic5"><img src="https://github.com/techmedic5.png" width="60px" alt="User avatar: Alan"></a><a href="https://github.com/ddocta"><img src="https://github.com/ddocta.png" width="60px" alt="User avatar: Damien Peters"></a><a href="https://github.com/dcsdigital"><img src="https://github.com/dcsdigital.png" width="60px" alt="User avatar: DCS Digital"></a><a href="https://github.com/pm7y"><img src="https://github.com/pm7y.png" width="60px" alt="User avatar: Paul Mcilreavy"></a><a href="https://github.com/tilwolf"><img src="https://github.com/tilwolf.png" width="60px" alt="User avatar: Til Wolf"></a><a href="https://github.com/ozzyoss77"><img src="https://github.com/ozzyoss77.png" width="60px" alt="User avatar: Leopoldo Crhistian Riverin Gomez"></a><a href="https://github.com/AlphaEcho11"><img src="https://github.com/AlphaEcho11.png" width="60px" alt="User avatar: AJEsau"></a><a href="https://github.com/svanomm"><img src="https://github.com/svanomm.png" width="60px" alt="User avatar: Steven VanOmmeren"></a><a href="https://github.com/socketbox"><img src="https://github.com/socketbox.png" width="60px" alt="User avatar: Casey Boettcher"></a><a href="https://github.com/zebbern"><img src="https://github.com/zebbern.png" width="60px" alt="User avatar: "></a><a href="https://github.com/avineetbespin"><img src="https://github.com/avineetbespin.png" width="60px" alt="User avatar: Avineet"></a><a href="https://github.com/invictus-1"><img src="https://github.com/invictus-1.png" width="60px" alt="User avatar: Chris"></a><a href="https://github.com/mirbyte"><img src="https://github.com/mirbyte.png" width="60px" alt="User avatar: mirko"></a><a href="https://github.com/bisonbet"><img src="https://github.com/bisonbet.png" width="60px" alt="User avatar: Tim Champ"></a><a href="https://github.com/Sinkingdev"><img src="https://github.com/Sinkingdev.png" width="60px" alt="User avatar: Peter Mathisen"></a><a href="https://github.com/Ed-STEM"><img src="https://github.com/Ed-STEM.png" width="60px" alt="User avatar: Ed di Girolamo"></a><a href="https://github.com/milkowski"><img src="https://github.com/milkowski.png" width="60px" alt="User avatar: Wojciech Miłkowski"></a><a href="https://github.com/ADS-Fund"><img src="https://github.com/ADS-Fund.png" width="60px" alt="User avatar: ADS Fund"></a><a href="https://github.com/arc46-io"><img src="https://github.com/arc46-io.png" width="60px" alt="User avatar: arc46 GmbH"></a><a href="https://github.com/liyin2015"><img src="https://github.com/liyin2015.png" width="60px" alt="User avatar: Li Yin"></a><a href="https://github.com/SylphAI-Inc"><img src="https://github.com/SylphAI-Inc.png" width="60px" alt="User avatar: SylphAI"></a><!-- all-sponsors --> <h2>🌟 Contributors</h2> <p><a href="https://github.com/mintplex-labs/anything-llm/graphs/contributors"><img src="https://contrib.rocks/image?repo=mintplex-labs/anything-llm" alt="anythingllm contributors"></a></p> <p><a href="https://star-history.com/#mintplex-labs/anything-llm&amp;Date"><img src="https://api.star-history.com/svg?repos=mintplex-labs/anything-llm&amp;type=Timeline" alt="Star History Chart"></a></p> <h2>🔗 More Products</h2> <ul> <li><strong><a href="https://github.com/mintplex-labs/vector-admin">VectorAdmin</a>:</strong> An all-in-one GUI &amp; tool-suite for managing vector databases.</li> <li><strong><a href="https://github.com/Mintplex-Labs/openai-assistant-swarm">OpenAI Assistant Swarm</a>:</strong> Turn your entire library of OpenAI assistants into one single army commanded from a single agent.</li> </ul> <div align="right"> <p><a href="#readme-top">![][back-to-top]</a></p> </div> <hr> <p>Copyright © 2026 <a href="https://github.com/mintplex-labs">Mintplex Labs</a>. <br><br> This project is <a href="./LICENSE">MIT</a> licensed.</p> <!-- LINK GROUP --> https://tedneward.github.io/Research/tools/augur/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/augur/index.html <p><a href="https://github.com/chaoss/augur">Source</a> | <a href="https://oss-augur.readthedocs.io/en/main/">Docs</a></p> <p>Features:</p> <ul> <li>focus on data from GitHub and GitLab platforms, which can scale to tens of thousands of repositories, and use Augur’s relational database as a data engineering tool to write custom queries that explore complex or unanticipated questions while performing in-depth research.</li> <li>explore data about compliance, security, dependencies, and related software topics to better understand potential risks associated with an open source project in addition to using visualizations to learn about community health.</li> <li>Data is collected incrementally and includes all messages and commits associated with issues, pull requests, and pull request reviews, including historical data. This high velocity collection has been tested for up to 100,000 repositories, enabling assessment of diverse open source software ecosystems.</li> <li>Collection and analysis goes beyond the counting of activities to include license coverage and license type information, COCOMO based software complexity and cost of replacement data by project and file, software dependency scanning, measurement of dependency LibYears, and time series persistent OpenSSF Scorecard information.</li> <li>Understanding of community health is extended through automatically detecting unusual activity by performing computational linguistics machine learning analysis and contribution anomaly detection.</li> <li>Users can explore complex or unanticipated questions while performing in-depth research using Augur’s relational database or API to write custom queries.</li> <li>Augur includes data visualizations through an extensible frontend built using tools familiar to data scientists (e.g., Dash and Plotly) upon which 8Knot is developed.</li> </ul> https://tedneward.github.io/Research/tools/bazel/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bazel/index.html <p><a href="https://bazel.build/">Website</a> | <a href="https://github.com/bazelbuild/bazel">Github</a></p> <p><a href="https://github.com/jin/awesome-bazel">Awesome Bazel</a>: A curated list of Bazel rules, tooling and resources.</p> <p><a href="https://github.com/spotify/bazel-tools">bazel-tools</a>: Tools for dealing with very large Bazel-managed repositories</p> <h4>Links/reading</h4> <p>Bazel How to build at Google scale?<br> - FOSDEM 2017; Klaus Aehlig<br> - <a href="https://archive.fosdem.org/2017/schedule/event/bazel/">https://archive.fosdem.org/2017/schedule/event/bazel/</a><br> - <a href="https://www.youtube.com/watch?v=lOUwu0myF8M">https://www.youtube.com/watch?v=lOUwu0myF8M</a><br> External C++ dependency management in Bazel<br> - <a href="https://medium.com/@htuch/external-c-dependency-management-in-bazel-dd37477422f5">https://medium.com/@htuch/external-c-dependency-management-in-bazel-dd37477422f5</a><br> Faster Bazel builds with remote caching: a benchmark<br> - <a href="https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html">https://nicolovaligi.com/faster-bazel-remote-caching-benchmark.html</a><br> Your Build in a Datacenter: Remote Caching and Execution in Bazel<br> - FOSDEM 2018; Jakob Buchgraber<br> - <a href="https://www.youtube.com/watch?v=PrTun05W5g4">https://www.youtube.com/watch?v=PrTun05W5g4</a><br> - <a href="https://fosdem.org/2018/schedule/event/datacenter_build/">https://fosdem.org/2018/schedule/event/datacenter_build/</a></p> https://tedneward.github.io/Research/tools/binwalk/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/binwalk/index.html <p><a href="https://github.com/ReFirmLabs/binwalk">Source</a></p> https://tedneward.github.io/Research/tools/bookstack/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bookstack/index.html <p><a href="https://www.bookstackapp.com/">Website</a> | <a href="https://github.com/BookStackApp/BookStack">Source</a></p> <p>Most who use BookStack enjoy its intuitive philosophy of a shelf being a generic subject or category, a book being a specific part of that shelf, and chapters within the book containing the most specific information.</p> <p>Pros</p> <ul> <li>Lots of people consider BookStack the "gold standard" of wikis</li> <li>Divide notes into logical structure, which may not be applicable to all notes.</li> <li>Example usage: Shelf for servers, book for hardware, and chapters for specific servers.</li> </ul> <p>Cons</p> <ul> <li>Could be spending a lot of time figuring out the best way for information to fit this structure.</li> <li>By default allowable upload file sizes are quite small. Instructions on how to modify this can be found in the documentation.</li> </ul> https://tedneward.github.io/Research/tools/bottles/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/bottles/index.html <p><a href="https://usebottles.com/">Website</a></p> <p>Windows prefixes are environments where it is possible to run Windows software using runners. Runners are compatibility layers capable of running Windows applications on Linux systems. In Bottles we call the these environments, bottles.</p> https://tedneward.github.io/Research/tools/buku/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/buku/index.html <p><a href="https://github.com/jarun/buku">Source</a></p> https://tedneward.github.io/Research/tools/canopy/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/canopy/index.html <p><a href="https://canopy.jcoglan.com/">Website</a> | <a href="https://github.com/jcoglan/canopy">Source</a></p> <p>Canopy can generate parsers in Java, JavaScript, Python, Ruby</p> https://tedneward.github.io/Research/tools/checkstyle/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/checkstyle/index.html <p><a href="https://checkstyle.sourceforge.io/">Website</a> | <a href="https://github.com/checkstyle/checkstyle">Source</a></p> https://tedneward.github.io/Research/tools/cloudfoundry/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/cloudfoundry/index.html <p><a href="https://www.cloudfoundry.org/">Website</a></p> https://tedneward.github.io/Research/tools/co/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/co/index.html <p><a href="https://abhinavsarkar.net/posts/implementing-co-1/">Part 1</a> | <a href="https://abhinavsarkar.net/posts/implementing-co-2/">Part 2</a> | <a href="https://abhinavsarkar.net/posts/implementing-co-3/">Part 3</a> | <a href="https://abhinavsarkar.net/posts/implementing-co-4/">Part 4</a></p> <p>Co has these basic features that are found in many programming languages:</p> <ul> <li>Dynamic and strong typing.</li> <li>Null, boolean, string and integer literals, and values.</li> <li>Addition, subtraction, multiplication and integer division arithmetic operations.</li> <li>String concatenation operation.</li> <li>Equality and inequality checks on booleans, strings and numbers.</li> <li>Less-than and greater-than comparison operations on numbers.</li> <li>Variable declarations, usage and assignments.</li> <li>if and while statements.</li> <li>Function declarations and calls, with support for recursion.</li> <li>First class functions and anonymous functions.</li> <li>Mutable closures.</li> </ul> <p>It also has these special features:</p> <ul> <li>yield statement to yield the current thread of computation (ToC).</li> <li>spawn statement to start a new ToC.</li> <li>First class channels with operators to send and receive values over them.</li> <li>sleep function to sleep the current ToC for a given number of milliseconds.</li> </ul> https://tedneward.github.io/Research/tools/codeql/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/codeql/index.html <p><a href="https://codeql.github.com/docs/">Docs</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://book.martiandefense.org/notes/product-security-engineering/sast-sca/codeql-for-beginners">CodeQL for Beginners</a></li> <li><a href="https://www.javaadvent.com/2021/12/finding-your-presents-using-codeql.html">Finding your presents using CodeQL</a>: "When I was a kid, I could not contain my excitement about Christmas. Meeting my extended family, having a great dinner with lots of laughter and joy. There was a tradition for the kids to search for Christmas presents in the living room. Be it behind the couch, between the Christmas tree branches, or way up on the shelf. Given how much I enjoyed the journey of finding the presents each year, my parents tried to make it a bit more interesting. Sometimes using wrapping paper that matched the wallpaper, sometimes by hiding it inside the chimney. This game reached a point that the stashes were so good that I could not find all the presents anymore and at one point, even my parents forgot where they put all the gifts. Long story short, I found the present eventually…3 months later, tucked away in the bookshelf, wrapped in the book cover. Fast forward to today; let’s get some help using technology to find our presents this year. This year, let’s turn the complexity up a notch and use a whole codebase as a searching ground for our presents. The good thing is that we can hide a lot more gifts in our codebase due to its inherent complexity. We have a lot of actors (our team) that can hide presents – either accidentally or even deliberately 🤫. For the sake of simplicity, let’s assume we’re hosting our code on GitHub, as this offers us an easy way to set up some of the other things we need in our journey to find all the presents."</li> </ul> <h3>Papers</h3> <ul> <li><a href="https://codeql.github.com/publications/algebraic-data-types.pdf">Algebraic Data Types for Object-oriented Datalog</a></li> </ul> <h3>Getting Started and Guides (along side the <a href="https://codeql.github.com/docs/">official docs</a>)</h3> <ul> <li><a href="https://codeql-learning-catalog.github.com/">CodeQL Learning Catalog</a> - The CodeQL Learning Catalog is a resource dedicated providing detailed CodeQL learning resources. The Catalog contains workshops, recordings, and learning paths for improving your knowledge and skill in using CodeQL.</li> <li><a href="https://securitylab.github.com/get-involved/">GitHub Security Lab</a> - From trying out CodeQL to secure your own code to collecting bug bounties by securing others', here are a few ways we can keep the world's software safe, together.</li> <li><a href="https://github.com/trailofbits/testing-handbook">testing-handbook</a> - The <a href="https://appsec.guide/docs/static-analysis/codeql/">Trail of Bits Testing Handbook</a> is a resource that guides developers and security professionals in configuring, optimizing, and automating many of the static and dynamic analysis tools used at Trail of Bits.</li> <li><a href="https://gist.github.com/bthomas2622/e520926b88ebb93e79b30f7f32ed4849">Custom Configuration File</a></li> </ul> <h3>Documentation</h3> <ul> <li><a href="https://codeql.github.com/docs/writing-codeql-queries">How to write CodeQL Queries</a></li> <li><a href="https://codeql.github.com/docs/codeql-language-guides">CodeQL Language Guide</a></li> <li><a href="https://codeql.github.com/docs/ql-language-reference">QL Language reference</a></li> <li><a href="https://codeql.github.com/codeql-standard-libraries">CodeQL Standard Libraries</a></li> <li><a href="https://codeql.github.com/codeql-query-help">CodeQL Query Help</a></li> <li><a href="https://codeql.github.com/docs/">Full CodeQL Documentation</a></li> </ul> <h3>Blogs</h3> <ul> <li><a href="https://github.blog/developer-skills/github/codeql-zero-to-hero-part-1-the-fundamentals-of-static-analysis-for-vulnerability-research/">GitHub - CodeQL zero to hero series</a></li> <li><a href="https://github.blog/engineering/how-github-uses-codeql-to-secure-github/">GitHub - How GitHub uses CodeQL to secure GitHub</a></li> </ul> <h3>YouTube learning</h3> <ul> <li><a href="https://www.youtube.com/live/y_-pIbsr7jc?&amp;t=310">Find bugs in your code with CodeQL</a></li> <li><a href="https://www.youtube.com/watch?v=pYzfGaLTqC0">Finding security vulnerabilities in JavaScript with CodeQL</a></li> <li><a href="https://www.youtube.com/watch?v=nvCd0Ee4FgE">Finding security vulnerabilities in Java with CodeQL</a></li> <li><a href="https://www.youtube.com/watch?v=eAjecQrfv3o&amp;t=98s">Finding security vulnerabilities in C/C++ with CodeQL</a></li> <li><a href="https://www.youtube.com/watch?v=-bJ2Ioi7Icg&amp;t=8s">CodeQL as an Audit Oracle</a></li> </ul> <h2>Software</h2> <h3>Installers</h3> <ul> <li><a href="https://github.com/github/gh-codeql">GH CodeQL</a> - GitHub CLI Extension for CodeQL to help manage installation</li> <li><a href="https://github.com/advanced-security/grab_ql">grab_ql</a> - Grab some/all of CodeQL CLI binary, QL library, VSCode starter workspace, VSCode and VSCode QL extension</li> <li><a href="https://github.com/david-wiggs/codeql-anywhere">codeql-anywhere</a> - Put the power of CodeQL in your pocket, take it with you to any CI 🚀</li> <li><a href="https://github.com/GitHubSecurityLab/codeql-jupyter-kernel">codeql-jupyter-kernel</a> - Jupyter Kernel for CodeQL</li> <li><a href="https://github.com/Homebrew/homebrew-cask/blob/master/Casks/c/codeql.rb">homebrew-cask</a> - Homebrew cask to install the CodeQL CLI <code>brew install --cask codeql</code></li> </ul> <h3>CLI Tooling</h3> <ul> <li><a href="https://github.com/github/gh-codeql">gh-codeql</a> - GitHub CLI extension for working with CodeQL</li> <li><a href="https://github.com/advanced-security/gh-codeql-scan">gh-codeql-scan</a> - GH CLI CodeQL Scan Extension</li> <li><a href="https://github.com/GitHubSecurityLab/gh-mrva">gh-mrva</a> - Multi-repo variant analysis CLI support</li> </ul> <h3>Customizations</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-summarize">codeql-summarize</a> - CodeQL Summary Generator to generate Models as Data (MaD) from CodeQL databases.</li> </ul> <h3><a href="https://docs.github.com/en/code-security/codeql-cli/using-the-codeql-cli/publishing-and-using-codeql-packs">Packs</a></h3> <ul> <li><a href="https://github.com/orgs/codeql/packages">GitHub-maintained packages</a></li> <li><a href="https://github.com/GitHubSecurityLab/CodeQL-Community-Packs">GitHub Security Lab community</a> - Collection of community-driven CodeQL query, library and extension <a href="https://github.com/orgs/githubsecuritylab/packages">packages</a>. Blog: <a href="https://github.blog/security/vulnerability-research/announcing-codeql-community-packs/">Announcing CodeQL Community Packs</a></li> <li>Trail of Bits - <a href="https://github.com/trailofbits/codeql-queries">codeql-queries</a> - CodeQL queries and <a href="https://github.com/orgs/trailofbits/packages?ecosystem=all&amp;q=repo%3Atrailofbits%2Fcodeql-queries">packs</a> developed by Trail of Bits</li> <li><a href="https://github.com/github/codeql-coding-standards">GitHub codeql-coding-standards</a> - This repository contains CodeQL queries and libraries which support various Coding Standards. (AUTOSAR C++, CERT-C++,CERT C, MISRA C)</li> </ul> <h3>Tooling (Bundles + Packs)</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-bundle-action">codeql-bundle-action</a> - Action to retrofit a CodeQL bundle with additional queries, libraries, and customizations</li> <li><a href="https://github.com/rvermeulen/codeql-bundle">codeql-bunldle</a> - CLI to build a custom CodeQL bundle</li> <li><a href="https://github.com/zbazztian/gh-tailor">gh-tailor</a> - A tool for customizing CodeQL packs.</li> </ul> <h3>Libraries</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-qtil">codeql-qtil</a> - A library with a wide variety of handy CodeQL utilities, from simple to complex.</li> </ul> <h3>Queries/Bundles</h3> <ul> <li><a href="https://www.microsoft.com/en-us/security/blog/2021/02/25/microsoft-open-sources-codeql-queries-used-to-hunt-for-solorigate-activity/">Microsoft solorigate queries</a></li> <li><a href="https://github.com/advanced-security/codeql-coding-standards-bundle-releases">GitHub codeql-coding-standards-bundle-releases</a> - CodeQL bundles containing the CodeQL Coding Standards queries</li> </ul> <h3>Query Suites</h3> <ul> <li><a href="https://github.com/zbazztian/only-critical-queries/blob/main/.github/critical-alternative.qls">Only Critical Queries sample .qls</a></li> <li><a href="https://github.com/securingdev/codeql-query-suites/blob/main/.github/configurations/owasp-top-10.qls">OWASP Top 10 CWE Only .qls</a></li> <li><a href="https://github.com/github/codeql/actions/workflows/query-list.yml?query=branch%3Acodeql-cli%2Flatest">CodeQL per Suite Query list</a> - download the attached <code>code-scanning-query-list.csv</code> artifact.</li> </ul> <h3>Troubleshooting</h3> <ul> <li><a href="https://github.com/advanced-security/advanced-security-material/tree/main/troubleshooting/codeql-builds">CodeQL Build Failure Troubleshooting</a></li> <li><a href="https://github.com/advanced-security/advanced-security-material/blob/main/troubleshooting/sarif-upload/troubleshooting.md">GitHub SARIF Upload Troubleshooting</a></li> <li><a href="https://github.com/github/codeql-coding-standards/blob/main/docs/user_manual.md#hazard-and-risk-analysis">CodeQL Coding Standards - Hazard and risk analysis</a></li> </ul> <h3>Monorepo Actions Samples</h3> <ul> <li><a href="https://github.com/dassencio/parallel-code-scanning">parallel-code-scanning</a> - An example of a GitHub Actions workflow showing how code scanning with CodeQL can be parallelized on monorepos.</li> <li><a href="https://github.com/thedave42/multi-lang-monorepo">multi-lang-monorepo</a> - A repo that demonstrates using an Actions workflow Job matrix to run parallel CodeQL scans on applications in a monorepo.</li> <li><a href="https://github.com/advanced-security/sample-javascript-monorepo">sample-javascript-monorepo</a> - Detached fork of babel/babel to use as a TypeScript monorepo sample with 150+ packages using the <a href="https://github.com/advanced-security/monorepo-code-scanning-action">monorepo-code-scanning-action</a></li> </ul> <h3>Actions Helpers</h3> <ul> <li><a href="https://github.com/advanced-security/set-codeql-language-matrix">set-codeql-language-matrix</a> - Automatically set the CodeQL matrix job using the languages in your repository.</li> <li><a href="https://github.com/advanced-security/filter-sarif">filter-sarif</a> - GitHub Action for filtering Code Scanning alerts by path and id</li> <li><a href="https://github.com/advanced-security/sarif-toolkit/blob/main/submodules/">sarif-toolkit</a> - Allows users to split up SARIF files that use submodules into multiple SARIF files that are then published to there appropriate repositories.</li> <li><a href="https://github.com/zbazztian/codeql-debug">codeql-debug</a> - Add this action to an existing CodeQL analysis workflow to generate an html report</li> <li><a href="https://github.com/advanced-security/dismiss-alerts">dismiss-alerts</a> - Dismisses GitHub Code Scanning alerts from <code>//codeql[supress reason]</code> style comments on the default branch</li> <li><a href="https://github.com/advanced-security/adjust-cvss">adjust-cvss</a> - Adjust the severity of the CVSS score assigned to a result in SARIF file</li> <li><a href="https://github.com/advanced-security/codeql-sarif-security-standard-annotator">codeql-sarif-security-standard-annotator</a> - Add an <code>owasp-top10-2021</code> tag to relevant results</li> <li><a href="https://github.com/advanced-security/delombok">delombok</a> - Delombok Java Code for analysis with Code Scanning (deprecated - now <a href="https://github.blog/changelog/2023-09-01-code-scanning-with-codeql-improves-support-for-java-codebases-that-use-project-lombok/">supported by CodeQL</a>)</li> <li><a href="https://github.com/MichaelCurrin/badge-generator">badge-generator</a> - <a href="https://github.com/MichaelCurrin/badge-generator/actions?query=workflow%3ACodeQL" title="Code quality workflow status"><img src="https://github.com/MichaelCurrin/badge-generator/workflows/CodeQL/badge.svg" alt="CodeQL"></a> Magically generate Markdown badges for your docs 🛡️ 🦡 🧙</li> <li><a href="https://github.com/advanced-security/monorepo-code-scanning-action">monorepo-code-scanning-action</a> - Focus SAST scans (with CodeQL) on just the changed parts of your monorepo, split up as you define</li> <li><a href="https://github.com/advanced-security/codeql-extractor-action">codeql-extractor-action</a> - An Action that allows you to specify a CodeQL extractor to be used in your workflows as an author of an Extractor.</li> </ul> <h3>SARIF</h3> <ul> <li><a href="https://marketplace.visualstudio.com/items?itemName=WDGIS.MicrosoftSarifViewer">Visual Studio SARIF Viewer</a> - Visual Studio Static Analysis Results Interchange Format (SARIF) log file viewer</li> <li><a href="https://marketplace.visualstudio.com/items?itemName=MS-SarifVSCode.sarif-viewer">VSCode SARIF Viewer</a> - Adds support for viewing SARIF logs in Visual Studio Code</li> <li><a href="https://plugins.jetbrains.com/plugin/23159-sarif-viewer">IntelliJ SARIF Viewer</a></li> <li><a href="https://microsoft.github.io/sarif-web-component/">SARIF Viewer Web Component</a></li> <li><a href="https://github.com/psastras/sarif-rs/tree/main/sarif-fmt">psastras/sarif-rs-sarif-fmt</a> - This crate provides a command line tool to pretty print SARIF files to easy human readable output.</li> </ul> <h3>Containers</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-docker">codeql-docker</a> - CodeQL Docker image</li> <li><a href="https://github.com/microsoft/codeql-container">codeql-container</a> - Prepackaged and precompiled github codeql container for rapid analysis, deployment and development.</li> <li><a href="https://github.com/advanced-security/codeql_container_example">codeql_container_example</a> - Example showing CodeQL to scan containerized applications in GitHub Actions.</li> <li><a href="https://some-natalie.dev/blog/codeql-container-builds/">codeql-container-builds</a> - Blog walking through the complexities of implementing containerized CodeQL workloads sprinkled with bits of Kubernetes wisdom.</li> </ul> <h3>Enforcement</h3> <ul> <li><a href="https://github.com/zkoppert/advanced-security-enforcer">advanced-security-enforcer</a> - A GitHub action for organizations that enables advanced security code scanning on all new repos</li> <li><a href="https://github.com/octodemo/codeql-selective-analysis">codeql-selective-analysis</a> - Make CodeQL a required status check for Pull Requests, but to skip the analysis in the case that only a certain subset of files are modified</li> </ul> <h3>Extractors</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-extractor-iac">codeql-extractor-iac</a> - CodeQL Extractors, Library, and Queries for Infrastructure as Code ( Terraform / HCL, JSON, YAML, Container files, Bicep )</li> <li><a href="https://github.com/GitHubSecurityLab/codeql-extractor-bicep">codeql-extractor-bicep</a> - CodeQL Extractor for Bicep Configurations</li> <li><a href="https://github.com/aibaars/codeql-kaleidoscope/">codeql-kaleidoscope</a> - CodeQL for LLVM Kaleidoscope (<a href="https://github.com/aibaars/codeql-kaleidoscope/commits/main/">AST/CFG/SSA/Dataflow in separate commits</a>)</li> <li><a href="https://github.com/microsoft/codeql/blob/main/powershell/README.md">Powershell Extractor</a> - CodeQL extractor, sample queries, and tools for Powershell</li> <li><a href="https://github.com/CoinFabrik/CyScout/tree/main/solidity/codeql">CyScout Solidity Extractor</a> - Run queries and detect vulnerabilities in your smart contracts using CodeQL-Solidity</li> <li><a href="https://github.com/krisds/cobol-codeql">cobol-codeql</a> - Archive of CodeQL support for COBOL (This is a one-off release of code for supporting analysis of COBOL programs using QL. The release of this code does not imply any intention to support it in the future.)</li> </ul> <h3>Extractor Helpers</h3> <ul> <li><a href="https://github.com/advanced-security/codeql-extractor-action">codeql-extractor-action</a> - specify a CodeQL extractor to be used in your workflows as an author of an Extractor.</li> </ul> <h3>Samples</h3> <ul> <li><a href="https://github.com/advanced-security/sample-codeql-pipeline-config">sample-pipeline-files</a> - This repository contains pipeline files for various CI/CD systems (AWS CodeBuild, Azure Devops, CircleCI, DroneCI, Jenkins, Tekton, Travis), illustrating how to integrate the CodeQL CLI Bundle for Automated Code Scanning</li> <li><a href="https://github.com/octodemo/vulnerable-pickle-app/blob/main/custom-queries/python/dangerous-functions.ql">Python Pickle</a> - mapping a custom framework in python</li> </ul> <h3>Query Writing</h3> <ul> <li><a href="https://github.com/readme/guides/custom-codeql-queries">ReadMe Project</a> - A beginner’s guide to running and managing custom CodeQL queries</li> </ul> https://tedneward.github.io/Research/tools/conan/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/conan/index.html <p><a href="https://conan.io/">Website</a> | <a href="https://github.com/conan-io/conan">Source</a></p> <p>macOS: <code>brew install conan</code></p> https://tedneward.github.io/Research/thinking/thinking-in-bets/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/thinking-in-bets/index.html <p><em>(by Annie Duke, Portfolio/Penguin, 2018, ISBN 9780735216365)</em></p> <h2>Chapter 1: Life is Poker, not Chess</h2> <p><strong>The hazards of resulting</strong>: we link results with decisions even though it is easy to point out indisputable examples where the relationship between decisions and results isn't so perfectly correlated.</p> <p><strong>Quick or dead: our brains weren't built for rationality</strong>: our brains evolved to create certainty and order. We are uncomfortable with the idea that luck plays a significant role in our lives. We recognize the existence of luck, but we resist the idea that, despite our best efforts, things might not work out the way we want. It feels better for us to imagine the world as an orderly place, where randomness does not wreak havoc and things are perfectly predictable. We evolved to see the world that way. Creating order out of chaos has been necessary for our survival. When we work backwards from results to figure out why those things happened, we are susceptible to a variety of cognitive traps, like assuming causation when there is only correlation, or cherry-picking data to confirm the narrative we prefer. We will pound a lot of square pegs into round holes to maintain the illusion of a tight relationship between our outcomes and our decisions. Kahneman's System 1" and "System 2" thinking; also called "reflexive mind" and "deliberative mind" by Gary Marcus (Kluge: The Haphazard Evolution of the Human Mind).</p> <p><strong>Game theory</strong>: "the study of mathematical models of conflict and cooperation between intelligent rational decision-makers."</p> <p><strong>Poker vs chess</strong>: chess contains no hidden information and very little luck. Poker is a game of incomplete information.</p> <p><strong>"I'm not sure": using uncertainty to our advantage</strong>: admitting that we don't know has an undeservedly bad reputation. The first step is understanding what we don't know. Ignorance: How It Drives Science (Stuart Firestein) points out that in science, "I don't know" is a necessary step toward enlightenment. What makes a decision great is not that it has a great outcome. A great decision is the result of good process, and that process must include an attempt to accurately represent our own state of knowledge. That state of knowledge, in turn, is some variation of "I'm not sure". Acknowledging uncertainty is the first step in executing on our goal to get closer to what is objectively true. "I'm not sure" is simply a more accurate representation of the world. When we accept that we can't be sure, we are less likely to fall into the trap of black-and-white thinking. The secret is to make peace with walking around in a world where we recognize that we are not sure and it's OK.</p> <p><strong>Redefining wrong</strong>: When we think in advance about the chances of alternative outcomes and make a decision based on those chances, it doesn't automatically make us wrong when things don't work out. It just means that one event in a set of possible futures occurred. Any prediction that isn't 0% or 100% can't be wrong solely because the most likely future doesn't unfold.</p> <h2>Chapter 2: Wanna Bet?</h2> <p><strong>We've all been to Des Moines</strong>: hiring an employee is not a riskless choice. By treating decisions as bets, poker players explicitly recognize that they are deciding on alternative futures, each with benefits and risks. They also recognize that there are no simple answers. If we follow the example of poker players by making explicit that our decisions are bets, we can make better decisions and anticipate (and take protective measures) when irrationality is likely to keep us from acting in our best interest.</p> <p><strong>All decisions are bets</strong>: "bet": "a choice made by thinking about what will probably happen"; "to risk losing (something) when you try to do or achieve something"; "to make decisions that are based on the belief that something will happen or is true". Every decision commits us to some course of action that, by definition, eliminates acting on other alternatives.</p> <p><strong>Most bets are against ourselves</strong>: One of the reasons we don't naturally think of decisions as bets is because we get hung up on the zero-sum nature of the betting that occurs in the gambling world. In most of our decisions, we are not betting against another person; we are betting against all the future versions of ourselves that we are not choosing. Whenever we make a choice, we are betting on a potential future; we are betting that the future version of us that results from the decisions we make will be better off. Ignoring the risk and uncertainty in every decision might make us feel better in the short run, but the cost to the quality of our decision-making can be immense. If we can find ways to become more comfortable with uncertainty, we can see the world more accurately and be better for it.</p> <p><strong>Our bets are only as good as our beliefs</strong>: we bet based on what we believe about the world. Part of the skill in life comes from learning to be a better calibrator, using experience and information to more objectively update our beliefs to more accurately represent the world. The more accurate our beliefs, the better the foundation of the bets we make. There is also skill in identifying when our thinking patterns might lead us astray, no matter what our beliefs are, and in developing strategies to work with (and sometimes around) those thinking patterns.</p> <p><strong>Hearing is believing</strong>: we form beliefs in a haphazard way, believing all sorts of things based just on what we hear out in the world but haven't researched for ourselves. (How to determine a man will go bald; how you calculate a dog's age in human years. Both are common misperceptions -- Google "common misconceptions".) How we think we form abstract beliefs: (1) We hear something; (2) We think about it and vet it, determining whether it is true or false; (3) We form our belief. How we actually form abstract beliefs: (1) We hear something; (2) We believe it to be true; (3) Only sometimes, later, if we have the time or inclination, we think about it and vet it, determining whether it is, in fact, true or false. Daniel Gilbert (Stumbling on Happiness) is responsible for pioneering work on belief formation: "People are credulous creatures who find it very easy to believe and very difficult to doubt." How we form beliefs was shaped by the evolutionary push towards efficiency rather than accuracy. Truthseeking, the desire to know truth regardless of whether the truth aligns with the beliefs we currently hold, is not naturally supported by the way we process information. We might think of ourselves as open-minded and capable of updating our beliefs based on new information, but the research conclusively shows otherwise. Instead of altering our beliefs to fit new information, we do the opposite, altering our interpretation of that information to fit our beliefs.</p> <p><strong>They saw a game</strong>: our pre-existing beliefs influence the way we experience the world. That those beliefs aren't formed in a particularly orderly way leads to all sorts of mischief in our decision-making.</p> <p><strong>The stubbornness of beliefs</strong>: once a belief is lodged, it becomes difficult to dislodge, leading us to notice and seek out evidence confirming our belief, rarely challenging the validity of confirming evidence and ignore or work hard to actively discredit information contradicting the belief. This is called motivated reasoning.</p> <p><strong>Being smart makes it worse</strong>: the smarter you are, the better you are at constructing a narrative that supports your beliefs, rationalizing and framing the data to fit your argument or point of view. The better you are with numbers, the better you are at spinning those numbers to conform to and support your beliefs.</p> <p><strong>Wanna bet?</strong>: when someone challenges us on a belief, signaling their confidence that our belief is inaccurate in some way, ideally it triggers us to vet the belief, taking an inventory of the evidence that informed us. Being asked if we are willing to bet money on it makes it much more likely that we will examine our information in a less-biased way, be more honest with ourselves about how sure we are of our beliefs, and be more open to updating and calibrating our beliefs. Offering a wager brings the risk out in the open, making explicit what is already implicit (and frequently overlooked).</p> <p><strong>Redefining confidence</strong>: The Half-Life of Facts is a great read about how practically every fact we've ever known has been subject to revision or reversal. We are in a perpetual state of learning, and that can make any prior fact obsolete. We would all be well-advised to take a good hard look at our beliefs; we would be better served as communicators and decision-makers if we thought less about whether we are confident in our beliefs and more about how confident (scale of zero to ten) we are; capture the shades of grey. When we work toward belief calibration, we become less judgmental of ourselves. Incorporating percentages or ranges of alternatives into the expression of our beliefs means that our personal narrative no longer hinges on whether we were right or wrong but on how well we incorporate new information to adjust the estimate of how accurate our beliefs are. Declaring our uncertainty in our beliefs to others makes us more credible communicators. Expressing our level of confidence also invites other people to be our collaborators. When we declare something as 100% fact, others might be reluctant to offer up new and relevant information that would inform our beliefs for two reasons: first, they might be afraid they are wrong and so won't speak up; second, even if they are very confident their information is high quality, they might be afraid of making us feel bad or judged. Expressing our beliefs this way also serves our listeners, by signaling that the belief needs further vetting.</p> <h2>Chapter 3: Bet to Learn: Fielding the Unfolding Future</h2> <p><strong>Outcomes are feedback</strong>: "Experience is not what happens to a man; it is what a man does with what happens to him." --Aldous Huxley. The difference between getting experience and becoming an expert lies in the ability to identify when the outcomes of our decisions have something to teach us and what that lesson might be. The challenge is that any single outcome can happen for multiple reasons; the unfolding future is a big data dump that we have to sort and interpret, and the world doesn't connect the dots for us between outcomes and causes. To reach our long-term goals, we have to improve at sorting out when the unfolding future has something to teach us, when to close the feedback loop. The first step to doing this well is in recognizing that things sometimes happen because of the other form of uncertainty: luck.</p> <p><strong>Luck vs skill: fielding outcomes</strong>: The way our lives turn out is the result of two things: the influence of skill and the influence of luck. Chalk up an outcome to skill, and we take credit for the result; chalk up an outcome to luck and it wasn't in our control. It is hard to get this right.</p> <p><strong>Working backward is hard: the SnackWell's Phenomenon</strong>: betting on low fat instead of sugar. Working backwards from the way things turn out isn't easy. We can get to the same outcome by different routes. Outcomes don't tell us what's our fault and what isn't, what we should take credit for and what we shouldn't. We can't simply work backward from the quality of the outcome to determine the quality of our beliefs or decisions. Rats get tripped up by uncertainty as well: when rats are trained on a fixed reward schedule, they learn pretty fast; when you remove the reward, the behave is quickly extinguished; when rats are trained on a variable/intermittent reinforcement schedule, that introduces uncertainty; when you remove the reward, the behavior extinguishes only after a very long time of fruitless behavior, sometimes thousands of tries. Worse, outcomes are rarely all skill or all luck.</p> <p><strong>"If it weren't for luck, I'd win every one"</strong>: the way we field outcomes is predictably patterned: we take credit for the good stuff, and blame the bad stuff on luck so it won't be our fault. Stanford law professor Robert MacCoun found that in 75% of accidents, the victims blamed someone else for their injuries. In single-vehicle accidents, 37% of drivers still found a way to pin the blame on someone else. Self-serving bias has immediate and obvious consequences for our ability to learn from experience.</p> <p><strong>People watching</strong>: watching is an established learning method. Unfortunately, learning from watching is just as fraught with bias. We field the outcomes of our peers predictably; where we blame the bad outcomes on bad luck, when it comes to our peers, bad outcomes are clearly their fault. While our own good outcomes are due to our awesome decision-making, when it comes to other people, good outcomes are because they got lucky.</p> <p><strong>Other people's outcomes reflect on us</strong>: Blaming others for their bad results and failing to give them credit for their good ones is under the influence of ego; knocking down a peer by finding them at fault for a loss lifts our personal narrative. Schadenfreude. We feel it's a zero-sum game. Our genes are competitive; natural selection proceeds by competition among the phenotypes of genes so we literally evolved to compete, a drive that allowed our species to survive. We think we know the ingredients for happiness; Sonja Lyubomirsky (psych prof at UCR) summarized several reviews of the literature on the elements we commonly consider: "a comfortable income, robust health, a supportive marriage, and lack of tragedy or trauma", however, "the general conclusion from almost a century of research on the determinants of well-being is that objective circumstances, demographic variables and life events are correlated with happiness less than intuition and everyday experience tells they out to be. By several estimates, all of these variables put together account for no more than 8% to 15% of the variance in happiness." What accounts for most of the variance in happiness is how we're doing comparatively. Would You Rather… earn $70k in 1900 or $70k now, most choose 1900, even though 1900 has no novocaine, air-conditioning, refrigeration or computers; we'd rather lap the field in 1900 with an average life expectancy of only forty-seven years than life in the middle of the pack with an average life expectancy of seventy-six years. A lot of the way we feel about ourselves comes from how we think we compare with others. We can learn better and be more open-minded if we work toward a positive narrative driven by engagement in truthseeking and striving toward accuracy and objectivity: giving others credit when it's due, admitting when our decisions could have been better, and acknowledging that almost nothing is black and white.</p> <p><strong>Reshaping habit</strong>: habits operate in a neurological loop consisting of three parts: the cue, the routine and the reward. To change a habit, you must keep the old cue, and deliver the old reward, but insert a new routine. We can work to change the habit of mind by substituting what makes us feel good. The golden rule of habit change says we don't have to give up the reward of a positive update to our narrative: we can work to get the reward of feeling good from being a good credit-giver, a good learner, and (as a result) a good decision-maker. Instead of feeling bad when we have to admit a mistake, what if the bad feeling came from the thought that we might be missing a learning opportunity just to avoid blame?</p> <p><strong>Wanna bet? Redux</strong>: The key is that in explicitly recognizing the way we field an outcome is a bet, we consider a greater number of alternative causes more seriously than we otherwise would have. The prospect of a bet makes us examine and refine our beliefs, in this case the belief about whether luck or skill was the main influence in the way things turned out. When we treat outcome fielding as a bet, it pushes us to field outcomes more objectively into the appropriate buckets because that is how bets are won. Thinking in bets triggers a more open-minded exploration of alternative hypotheses, of reasons supporting conclusions opposite to the routine of self-serving bias; we are more likely to explore the opposite side of an argument more often and more seriously--and that will move us closer to the truth of the matter. Thinking in bets also triggers perspective taking, leveraging the difference between how we field our own outcomes versus others' outcomes to get closer to the objective truth.</p> <h2>Chapter 4: The Buddy System</h2> <p><strong>"Maybe you're the problem, do you think?"</strong>: Not all situations are appropriate for truthseeking, nor are all people interested in the pursuit. Any of us who wants to get better at thinking in bets would benefit from having more David Lettermans in our lives; Lettermaning needs agreement by both parties to be effective.</p> <p><strong>The red pill or the blue pill?</strong>: Our brains have evolved make our version of the world more comfortable: our beliefs are nearly always correct; favorable outcomes are the result of our skill; there are plausible reasons why unfavorable outcomes are beyond our control; we compare favorably with our peers; we deny or at least dilute the most painful parts of the message. Giving that up is not the easiest choice; by choosing to exit the matrix, we are asserting that striving for a more objective representation of the world, even if it is uncomfortable at times, will make us happier and more successful in the long run.</p> <p>Thinking in bets is easier if you have other people to help you. A good decision group is a grown-up version of the buddy system. If we can find a few people to choose to form a truthseeking pod with us and help us to do the hard work connected with it, it will move the needle--just a little bit, but with improvements that accumulate and compound over time. We will be more successful in fighting bias, seeing the world more objectively, and, as a result, we will make better decisions. As long as there are three people in the group (two to disagree and one to referee), the truthseeking group can be stable and productive.</p> <p><strong>Not all groups are created equal</strong>: a well-chartered group can be particularly useful for habits that are difficult to break or change. But while a group can function to be better than the sum of the individuals, it doesn't automatically turn out that way. Being in a group can improve our decision quality by exploring alternatives and recognize where our thinking might be biased, but a group can also exacerbate our tendency to confirm what we already believe (echo chamber). Confirmatory thought amplifies bias; promotes a love and celebration of one's own beliefs. Exploratory thought encourages an open-minded and objective consideration of alternative hypotheses and a tolerance of dissent to combat bias. Exploratory thought helps the members of a group reason toward a more accurate representation of the world. Without an explicit charter for exploratory thought and accountability to that charter, our tendency when we interact with others follows our individual tendency. "Complex and open-minded thought is most likely to be activated when decision makers learn prior to forming any opinions that they will be accountable to an audience (a) whose views are unknown, (b) who is interested in accuracy, (c) who is reasonably well-informed, and (d) who has a legitimate reason for inquiring into the reasons behind participants' judgments/choices." (Lerner, Tetlock) Groups can improve the thinking of individual decision-makers when the individuals are accountable to a group whose interest is in accuracy; charter should also encourage and celebrate a diversity of perspectives to challenge biased thinking by individual members. Jonathan Haidt (The Righteous Mind: Why Good People Are Divided by Politics and Religion): "If you put individuals together in the right way, such that some individuals can use their reasoning powers to disconfirm the claims of others, and all individuals feel some common bond or shared fate that allows them to interact civilly, you can create a group that ends up producing good reasoning as an emergent property of the social system. This is why it's so important to have intellectual and ideological diversity within any group or institution whose goal is to find truth." In combination, the advice of these experts in group interaction adds up to a blueprint for a truthseeking charter: (1) A focus on accuracy (over confirmation), which includes rewarding truthseeking, objectivity, and open-mindedness within the group; (2) Accountability, for which members have advance notice, and (3) Openness to a diversity of ideas.</p> <p>The group rewards focus on accuracy -- a productive decision group can harness the desire for approval by rewarding accuracy and intellectual honesty with social approval.</p> <p><strong>"One Hundred White Castles... and a large chocolate shake"</strong>: how accountability improves decision-making -- accountability is a willingness or obligation to answer for our actions or beliefs to others. A bet is a form of accountability. After spending time in that (poker players) kind of environment, you become hypervigilant about your level of confidence in your beliefs; it is truly putting your money where your mouth is. Accountability also improves our decision-making and information processing when we are away from the group because we know in advance that we will have to answer to the group for our decisions: imagining how the discussion will go helps us to spot more errors on our own and catch them more quickly.</p> <p>The group ideally exposes us to a diversity of viewpoints -- diversity and dissent are not only checks on fallibility, but the only means of testing the ultimate truth of an opinion. If we take a bunch of people with that limitation [being a human and thus having only one point of view] and put them together into a group, we get exposed to diverse opinions, can test alternative hypotheses, and move toward accuracy. To get a more objective view of the world, we need an environment that exposes us to alternate hypotheses and different perspectives. To view ourselves in a more realistic way, we need other people to fill in our blind spots. A group with diverse viewpoints can help us by sharing the work to combat motivated reasoning and biased outcome fielding; by thinking in bets, we run through a series of questions to examine the accuracy of our beliefs, for example:<br> * Why might my belief not be true?<br> * What other evidence might be out there bearing on my belief?<br> * Are there similar areas I can look toward to gauge whether similar beliefs to mine are true?<br> * What sources of information could I have missed or minimized on the way to reaching my belief?<br> * What are the reasons someone else could have a different belief, what's their support, and why might they be right instead of me?<br> * What other perspectives are there as to why things turned out the way that they did?</p> <p>By asking these questions, we are taking a big step toward calibration, but there is only so much we can do to answer these questions on our own. It is a lot easier to have someone else offer their perspective than for you to imagine you're another person and think about what their perspective might be. A diverse group can do some of the heavy lifting of de-biasing for us. Dissent channels and red teams are a beautiful implementation of John Stuart Mill's bedrock principle that we can't know the truth of a matter without hearing the other side. Diversity is the foundation of productive group decision-making, but we can't underestimate how hard it is to maintain.</p> <p><strong>Federal judges: drift happens</strong>: Cass Sunstein (Harvard law professor) conducted massive study on ideological diversity in federal judicial panels; when there was political diversity on the panels, that diversity improved the panel's work--a single panelist from the other party had "a large disciplining effect". The more homogeneous we get, the more the group will promote and amplify confirmatory thought.</p> <p><strong>Social psychologists: confirmatory drift and Heterodox Academy</strong>: First, there is a natural drift toward homogeneity and confirmatory thought; we all experience this gravitation toward people who think like we do. Second, groups with diverse viewpoints are the best protection against confirmatory thought; the opinions of the group members aren't much help if it is a group of clones.</p> <p><strong>Wanna bet (on science)?</strong>: Experts engaging in traditional peer review, providing their opinion on whether an experimental result would replicate, were right 58% of the time. A betting market in which the traders were the exact same experts and those experts had money on the line predicted correctly 71% of the time.</p> <h2>Chapter 5: Dissent to Win</h2> <p><strong>CUDOS to a magician</strong>: CUDOS stands for Communism (data belonging to the group), Universalism (apply uniform standards to claims and evidence, regardless of where they came from), Disinterestedness (vigilance against personal conflicts that can influence the group's evaluation) and Organized Skepticism (discussion among the group to encourage engagement and dissent).</p> <p><strong>Mertonian communism: more is more</strong>: the communal ownership of data within groups. Any attempt at accuracy is bound to fall short if the truthseeking group has only limited access to potentially pertinent information; without all the facts, accuracy suffers. As a rule of thumb, if we have an urge to leave out a detail because it makes us uncomfortable or requires even more clarification to explain away, those are exactly the details we must share. The mere fact of our hesitation and discomfort is a signal that such information may be critical to providing a complete and balanced account. We are naturally reluctant to share information that could encourage others to find fault in our decision-making. Group(s) [can] make this easier by making [one] feel good about commiting [oneself] to improvement.</p> <p><strong>Universalism: don't shoot the message</strong>: "Truth-claims, whatever their source, are to be subjected to preestablished impersonal criteria." (Merton) Acceptance/rejection of an idea must not "depend on the personal or social attributes of their protagonist." Don't disparage or ignore an idea just because you don't like who or where it came from. The accuracy of the statement should be evaluated independent of its source. Nearly any group can create an exercise to develop and reinforce the open-mindedness universalism requires.</p> <p><strong>Disinterestedness: we all have a conflict of interest, and it's contagious</strong>: Conflicts of interest come in many flavors; our brains have built-in conflicts of interest, interpreting the world around us to confirm our beliefs (and more). We are not naturally disinterested. A group is less likely to succumb to ideological conflicts of interest when they don't know what the interest is. Another way a group can de-bias members is to reward them for skill in debating opposing points of view and finding merit in opposing positions. The group's reinforcement ought to discourage us from creating a straw-man argument when we're arguing against our beliefs, and encourage us to feel good about winning the debate. This is one of the reasons it's good for a group to have at least three members, two to disagree and one to referee.</p> <p><strong>Organized skepticism: real skeptics make arguments and friends</strong>: True skepticism is consistent with good manners, civil discourse, and friendly communications. Skepticism is about approaching the world by asking why things might not be true rather than why they are true. Thinking in bets embodies skepticism by encouraging us to examine what we do and don't know and what our level of confidence is in our beliefs and predictions. This moves us closer to what is objectively true. Without embracing uncertainty, we can't rationally bet on our beliefs, and we need to be particularly skeptical of information that agrees with us because we know that we are biased to just accept and applaud confirming evidence. When we implement the norm of skepticism, we naturally modulate expression of dissent with others. Organized skepticism invites people into a cooperative exploration. Skepticism should be encouraged and, where possible, operationalized.</p> <p><strong>Communicating with the world beyond our group</strong>: we have to take the most constructive, civil elements of truthseeking communication and introduce them carefully. First, express uncertainty. Second, lead with assent; listen for things you agree with, state those and be specific, and then follow with an "and" instead of "but". Third, ask for a temporary agreement to engage in truthseeking; if someone is offloading emotion to us, we can ask them if they are just looking to vent or if they are looking for advice. Finally, focus on the future; rather than rehashing what has already happened, try instead to engage about what the person might do so that things will turn out better going forward.</p> <h2>Chapter 6: Adventures in Mental Time Travel</h2> <p><strong>Let Marty McFly run into Marty McFly</strong>: in real-life decision-making, when we bring our past- or future-self into the equation, the space-time continuum doesn't unravel; a visit from past- or future-us helps present-us make better bets.</p> <p><strong>Night Jerry</strong>: "Night Guy always screws Morning Guy." When we make in-the-moment decisions (and don't ponder the past or future) we are more likely to be irrational and impulsive. This is called temporal discounting; we are willing to take an irrationally large discount to get a reward now instead of waiting for a bigger reward later.</p> <p><strong>Moving regret in front of our decisions</strong>: Regret is one of the most intense emotions we feel, but it's arguable about whether it's productive or useful. The problem isn't so much whether regret is an unproductive emotion; it's that regret occurs after the fact, instead of before. Suzy Welch developed 10-10-10: "What are the consequences of each of my options in ten minutes? Ten months? Ten years?" We can build on Welch's tool by asking the question through the frame of the past: "How would I feel today if I had made this decision ten minutes ago? Ten months ago? Ten years ago?" Moving regret in front of a decision has numerous benefits. First it can influence us to make a better decision. Second, it helps us treat ourselves (regardless of the actual decision) more compassionately after the fact. We can anticipate and prepare for negative outcomes; we can devise a plan to respond to a negative outcome instead of just reacting to it; we can also familiarize ourselves with the likelihood of a negative outcome and how it will feel.</p> <p><strong>A flat tire, the ticker, and a zoom lens</strong>: Flat tire in a downpour--how does it feel? Likely, like the worst moment of your life. But if the flat tire had happened a year ago, do you think it would have an effect on your happiness today, or your overall happiness over the past year? Not likely; it likely wouldn't cause your overall happiness to tick up or down. In our decision-making lives, we aren't that good at taking this kind of perspective; it just feels how it feels in the moment and we react to it.</p> <p><strong>"Yeah, but what have you done for me lately?"</strong>: the way we field outcomes is path dependent; it doesn't matter so much where we end up as how we got there. What has happened in the recent past drives our emotional response much more than how we are doing overall. Our in-the-moment emotions affect the quality of the decisions we make in those moments, and we are very willing to make decisions when we are not emotionally fit to do so.</p> <p><strong>Tilt</strong>: from pinball machines; when the emotional center of the brain starts pinging, the limbic system (specifically the amygdala) shuts down the prefrontal cortex. We light up, then we shut down our cognitive control center. By recognizing in advance these verbal and physiological signs that ticker watching is making us tilt, we can commit to develop certain habit routines at those moments. We can precommit to walk away from the situation when we feel the signs of tilt, take some space until we calm down and get some perspective, recognizing that when are on tilt we aren't decision-fit.</p> <p><strong>Ulysses contracts</strong>: time traveling to precommit -- past-us preventing present-us from doing something stupid has become known as a Ulysses contract (Ulysses/Odysseus filling his sailors' ears with wax and tying himself to the mast so he could hear the Sirens). It's the perfect interaction between past-you, present-you and future-you. Ulysses contracts can be barrier-raising (against irrational or undesirable behavior) or barrier-reducing (to increase the ease of making the desirable decision).</p> <p><strong>Decision swear jar</strong>: a simple kind of precommitment contract to implement accountability.</p> <p><strong>Reconnaissance</strong>: mapping the future -- for us to make better decisions, we need to perform reconnaissance on the future. If a decision is a bet on a particular future based on our beliefs, then before we place a bet we should consider in detail what those possible futures might look like. Figure out the possibilities, then take a stab at the probabilities. Scenario planning: consider a broad range of possibilities for how the future might unfold to help guide long-term planning and preparation. After identifying as many of the possible outcomes as we can, we want to make our best guess at the probability of each of those futures occurring. The reason we do reconnaissance is because we are uncertain; we don't (and likely can't) know how often things will turn out a certain way with exact precision. It's not about approaching our future predictions from a point of perfection; it's about acknowledging that we're already making a prediction about the future every time we make a decision, so we're better off if we make that explicit. If we're worried about guessing, we're already guessing. By at least trying to assign probabilities, we will naturally move away from the default of 0% or 100%, away from being sure it will turn out one way and not another. Scouting various futures has numerous additional benefits. First, scenario planning reminds us that the future is inherently uncertain; by making that explicit in our decision-making process, we have a more realistic view of the world. Second, we are better prepared for how we are going to respond to different outcomes that might result from our initial decision. We can anticipate positive or negative developments and plan our strategy, rather than being reactive. If our reconnaissance has identified situations where we are susceptible to irrationality, we can try to bind our hands with a Ulysses contract. Third, anticipating the range of outcomes also keeps us from unproductive regret (or undeserved euphoria) when a particular future happens. Finally, by mapping out the potential futures and probabilities, we are less likely to fall prey to resulting or hindsight bias.</p> <p><strong>Backcasting</strong>: working backward from a positive future -- If we were contemplating a thousand-mile walk, we'd be better off imagining ourselves looking back from the destination and figuring how we got there. When it comes to advance thinking, standing at the end and looking backward is much more effective than looking forward from the beginning. Imagining a successful future and backcasting from there is a useful time-travel exercise for identifying necessary steps for reaching our goals; working backward helps even more when we give ourselves the freedom to imagine an unfavorable future.</p> <p><strong>Premortems</strong>: working backward from a negative future -- A premortem is where we check our positive attitude at the door and imagine not achieving our goals. Despite the popular wisdom that we achieve success through positive visualization, it turns out that incorporating negative visualization makes us more likely to achieve our goals. Gabriele Oettingen (Rethinking Positive Thinking: Inside the New Science of Motivation) has conducted over twenty years of research, consistently finding that people who imagine obstacles in the way of reaching their goals are more likely to achieve success, a process she has called "mental contrasting". We need to have positive goals, but we are more likely to execute on those goals if we think about negative futures. We start a premortem by imagining why we failed to reach our goal; then we imagine why. All those reasons why we didn't achieve our goal help us anticipate potential obstacles and improve our likelihood of succeeding. The key to a successful premortem is that everyone feels free to look for the most creative, relevant and actionable reasons why things didn't work out, and they are motivated to scour everything--personal experience, company experience, historical precedent, sports analogies, etc--to come up with ways a decision or plan can go bad, so the team can anticipate and account for them.</p> <p><strong>Dendrology and hindsight bias</strong> (or, Give the chainsaw a rest): think about time as a tree. The trunk is the past. A tree has only one, growing trunk, just as we have only one, accumulating past. Branches are the potential futures. Thicker branches are the equivalent of more probable futures, thinner branches are less probable ones. As the future becomes the past, what happens to all those branches? The ever-advancing present acts like a chainsaw; when one of those many branches happens to be the way things turn out, present-us cuts off all those other branches that didn't materialize and obliterates them. When we look into the past and see the only thing that happened, it seems to have been inevitable. Why wouldn't it? That's hindsight bias, an enemy of probabilistic thinking. By keeping an accurate representation of what could have happened (and not a version edited by hindsight), memorializing the scenario plans and decision trees we create through good planning process, we can be better calibrators going forward.</p> https://tedneward.github.io/Research/thinking/thinking-in-systems/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/thinking-in-systems/index.html <p><em>(Donella H. Meadows; ISBN ...)</em></p> <h3>Part 1: System Structure and Behavior</h3> <h4>Ch 1: The Basics</h4> <h5>More Than the Sum of Its Parts</h5> <ul> <li>A system is an interconnected set of elements that is coherently organized in a way that achieves something. It consists of three kinds of things: elements, interconnections, and a function or purpose.</li> <li>There is an integrity or wholeness about a system and an active set of mechanisms to maintain that integrity.</li> <li>The system is more than the sum of its parts. It may exhibit adaptive, dynamic, goal-seeking, self-preserving, and sometimes evolutionary behavior.</li> </ul> <h5>Look Beyond the Players to the Rules of the Game</h5> <ul> <li>The elements of a system are often the easiest parts to notice, because many of them are visible, tangible things. But intangibles can also be elements.</li> <li>Many interconnections in a system are flows of information, or signals that go to decision or action points. Information holds systems together and greatly determines how they operate.</li> <li>The purpose of a system is deduced from its behavior, not from rhetoric or stated goals. One important purpose of almost every system is to ensure its own perpetuation.</li> <li>Keeping sub-purposes and overall system purposes in harmony is an essential function of successful systems.</li> <li>A system generally goes on being itself, changing only slowly if at all, even with complete substitutions of its elements –&nbsp;as long as its interconnections and purposes remain intact.</li> <li>The elements are often the least important in defining the unique characteristics of the system, unless changing an element also changes relationships or purpose.</li> </ul> <h5>Bathtubs 101 – Understanding System Behavior over Time</h5> <ul> <li>Stocks are the elements of a system that you can see, feel, count, or measure. They are a store, quantity, or accumulation of material or information that has built up over time.</li> <li>Stocks change over time through the actions flows. A stock is therefore a memory of the history of changing flows within the system.</li> <li>Understanding the dynamics of stocks and flows reveals much about a complex system's behavior. In a state of dynamic equilibrium, a stock does not change.</li> <li>We focus more on inflows than outflows, but a stock can be increased by decreasing its outflow rate as well as increasing its inflow rate.</li> <li>A stock takes time to change, even when the flows into our out of them change suddenly. Stocks therefore act as delays or buffers or shock absorbers in a system.</li> <li>The time lags that come from slowly changing stocks can cause problems, but also provide stability. They allow room to maneuver, to experiment, and to revise policies that aren't working.</li> <li>Stocks allow inflows and outflows to be decoupled and to be independent and temporarily out of balance with one another.</li> <li>Most individual and institutional decisions are designed to regulate the levels of stocks. System thinkers see the world as stocks with the mechanisms for regulating their levels by manipulating flows.</li> </ul> <h5>How the System Runs Itself – Feedback</h5> <ul> <li>Feedback loops permit behaviors that persist over time. Such a loop is formed when a change in stock affects the flows into our out of that same stock.</li> <li>Feedback loops can cause stocks to maintain their levels within a range or grow or decline.</li> </ul> <h5>Stabilizing Loops – Balancing Feedback</h5> <ul> <li>Balancing feedback loops are goal-seeking or stability-seeking. They keep a stock at a given value or range of values. They oppose whatever change is imposed on the system.</li> </ul> <h5>Runaway Loops – Reinforcing Feedback</h5> <ul> <li>A reinforcing feedback loop enhances whatever direction of change is imposed on it, thereby creating a virtuous or vicious cycle of healthy growth or runaway destruction.</li> <li>Reinforcing loops are found wherever a system element can reproduce itself or grow as a constant fraction of itself. It leads to exponential growth or to runaway collapses over time.</li> <li>Ask yourself: "If A causes B, is it possible that B also causes A?" Feedback opens up the idea that a system can cause its own behavior.</li> </ul> <h4>Ch 2: A Brief Visit to the Systems Zoo</h4> <h5>One-Stock Systems</h5> <p><em>A Stock with Two Competing Balancing Loops – a Thermostat</em></p> <ul> <li>A furnace cannot heat a room to a target temperature because heat leaks to the outside. This leak drains away some heat even as the furnace is getting the signal to put it back.</li> <li>The information delivered by a feedback loop can only affect future behavior; it can't deliver a signal fast enough to correct behavior that drove the current feedback.</li> <li>This means that a flow can't react instantly to a flow. It can only react to a change in stock, and only after a slight delay to register the incoming information.</li> <li>A stock-maintaining balancing feedback loop must have its goal set to compensate for draining or inflowing processes that affect the stock. Otherwise the process will fall short or exceed the target.</li> <li>Every feedback loop has its breakdown point, where other loops pull the stock away from its goal more strongly than it can pull back.</li> </ul> <p><em>A Stock with One Reinforcing Loop and One Balancing Loop – Population and Industrial Economy</em></p> <ul> <li>Because systems often have several competing feedback loops operating simultaneously, those loops that dominate the system will determine its behavior.</li> <li>Complex behaviors of systems often arise as the relative strengths of feedback loops shift, causing first one loop and then another to dominate the behavior.</li> <li>To test the value of a model, ask: Are the driving factors likely to unfold this way? If they did, would the system react this way? What is driving the driving factors?</li> <li>Model utility depends on not whether its driving scenarios are realistic (since no one can know that for sure), but on whether it responds with a realistic pattern of behavior.</li> <li>Population has a reinforcing loop of births and a balancing loop of deaths. Capital has a reinforcing loop of investment of output and a balancing loop of depreciation.</li> <li>A central question of economic development is how to keep the reinforcing loop of capital accumulation growing more slowly than the reinforcing loop of population growth, so that people are getting richer instead of poorer.</li> </ul> <p><em>A System with Delays – Business Inventory</em></p> <ul> <li>A delay in a balancing feedback loop makes a system likely to oscillate.</li> <li>Changes in stock while waiting for a flow to change can lead us to overreact. If the change in flow is more than expected then we can overreact again and change the flow too much in the other direction.</li> <li>Delays are strong determinants of behavior, and so we can't begin to understand the dynamic behavior of systems unless we know where and how long the delays are.</li> <li>Economies are extremely complex systems; they are full of balancing feedback loops with delays, and they are inherently oscillatory.</li> </ul> <h5>Two-Stock Systems</h5> <p><em>A Renewable Stock Constrained by a Nonrenewable Stock – an Oil Company</em></p> <ul> <li>Any real physical entity is always exchanging things with its environment, and so any physical, growing system is going to eventually run into some sort of constraint.</li> <li>This constraint takes the form of a balancing loop that shifts the dominance of the reinforcing loop driving the growth, either by strengthening the outflow or weakening the inflow.</li> <li>Growth in a constrained environment is called the "limits-to-growth" archetype.</li> <li>The more capital, the higher the extraction rate, the lower the resource stock, the lower the yield per unit of capital, the lower the profit, the lower the investment rate, and so the lower growth rate of capital.</li> <li>When you're building a capital stock dependent on a nonrenewable resource, the higher and faster you grow, the farther and faster you fall.</li> <li>The real choice in the management of a nonrenewable resource is whether to get rich very fast or to get less rich but stay that way longer.</li> <li>Raising the price of the resource gives the industry higher profits, so investment goes up, capital stock continues rising, and the more costly resources can be extracted.</li> </ul> <p><em>Renewable Stock Constrained by a Renewable Stock – a Fishing Economy</em></p> <ul> <li>A simplified model is affected by three non-linear relationships: price (scarcer fish cost more), regeneration rate (scarcer fish breed less), and yield per unit of capital (efficiency of fishing).</li> <li>Nonrenewable resources are stock-limited: The entire stock is available at once and can be extracted at any rate, and so the faster you extract, the shorter its lifetime.</li> <li>Renewable resources are flow-limited: The stock supports indefinite extraction at a rate equal to its regeneration rate, or else the resource may be driven below a threshold and become non-renewable.</li> <li>More and more, increases in technology and harvest efficiency have the ability to drive resource populations to extinction.</li> <li>If the balancing feedback loop is weak, so that capital grows even as the resource dips below its threshold ability to regenerate itself, the resource and industry both collapse.</li> </ul> <h3>Part 2: Systems and Us</h3> <h4>Ch 3: Why Systems Work So Well</h4> <ul> <li>When a system works well, chances are good that you may have observed one of three characteristics: resilience, self-organization, or hierarchy.</li> </ul> <h5>Resilience</h5> <ul> <li>Resilience is a measure of a system's ability to survive and persist within a variable environment. The opposite is brittleness or rigidity.</li> <li>Resilience arises from a rich structure of many feedback loops that can work in different ways to restore a system even after a large perturbation.</li> <li>It can be very hard to see, unless you exceed its limits, overwhelm and damage the balancing loops, and the system structure breaks down.</li> <li>Think of resilience as a plateau upon which the system can play, performing its normal functions in safety.</li> </ul> <h5>Self-Organization</h5> <ul> <li>The capacity of a system to make its own structure more complex is called self-organization.</li> <li>If we weren't so blind to the property of self-organization, we'd do a better job of encouraging rather than destroying the self-organizing capacities of systems of which we're a part.</li> <li>Self-organization produces heterogeneity and unpredictability, yielding whole new structures or ways of doing things. It requires freedom of experimentation and a certain amount of disorder.</li> <li>Out of simple rules of self-organization can grow enormous, diversifying crystals of technology, physical structures, organizations, and cultures.</li> </ul> <h5>Hierarchy</h5> <ul> <li>Complex systems can evolve from simple systems only if there are stable intermediate forms. The resulting complex forms will be naturally hierarchic.</li> <li>Hierarchies not only give a system stability and resilience, but they reduce the amount of information that any part of the system has to keep track of.</li> <li>Hierarchies evolve from the lowest level up, and the original purpose of a hierarchy is always to help its originating subsystems do their jobs better.</li> <li>When a subsystem's goals dominate at the expense of the total system's goals, the resulting behavior is called sub-optimization. But just as damaging is too much central control.</li> <li>To be a highly functional system, hierarchy must balance the welfare, freedoms, and responsibilities of the subsystems and the total system. It must balance autonomy and central control.</li> </ul> <h4>Ch 4: Why Systems Surprise Us</h4> <ul> <li>Everything we think we know about the world is a model. Our models have a strong congruence with the world, but fall far short of representing the real world fully.</li> </ul> <h5>Beguiling Events</h5> <ul> <li>It's engrossing to see the world as a series of events, and constantly surprising, because that way of seeing the world has almost no predictive or explanatory value.</li> <li>We are less likely to be surprised if we can see how events accumulate into dynamic patterns of behavior.</li> <li>System structure, or the stocks and flows and feedback loops, reveals itself as a series of events over time. It's the key to understanding not just what is happening, but why.</li> <li>All systems surprise us because we focus too little on their history, and we are insufficiently skilled at seeing in this history clues to the structures from which behavior and events flow.</li> </ul> <h5>Linear Minds in a Nonlinear World</h5> <ul> <li>We often are not very skilled in understanding the nature of relationships, especially nonlinear ones, where a cause does not produce a proportional effect.</li> <li>Nonlinearities foil the reasonable expectation that if a little of some cure did a little good, then a lot of it will do a lot of good. Similarly for destructive actions and harm.</li> <li>Nonlinearities change the relative strengths of feedback loops, flipping a system from one mode of behavior to another. They are the chief cause of shifting dominance.</li> </ul> <h5>Nonexistent Boundaries</h5> <ul> <li>The greatest complexities of a system exist at its boundaries, or sources or sinks as stocks that are ignored for the purposes of simplifying a system.</li> <li>Disorderly, mixed-up borders are sources of diversity and creativity.</li> <li>We have to invent boundaries for clarity and sanity, and boundaries can produce problems when we forget that we've artificially created them.</li> <li>There are no separate systems. The world is a continuum. Where to draw a boundary around a system depends on the purpose of the discussion –&nbsp;the questions we want to ask.</li> <li>We are attached to the boundaries that our minds happen to be accustomed to, but boundaries can and should be reconsidered for each new discussion, problem, or purpose.</li> </ul> <h5>Layers of Limits</h5> <ul> <li>We live in a world where multiple inputs come together to produce multiple outputs, and virtually all of the inputs, and therefore all of the outputs, are limited.</li> <li>At any given time, the input that is the most important to a system is the one that is the most limiting.</li> <li>Whenever one factor ceases to be limiting, growth occurs, which itself changes the relative scarcity of factors until another becomes limiting.</li> <li>To shift attention from the abundant factors to the next potential limiting factor is to gain real understanding of, and control over, the growth process.</li> <li>Any physical entity with multiple inputs and outputs is surrounded by layers of limits. The choice is not to grow forever, but to decide what limits to live within.</li> </ul> <h5>Ubiquitous Delays</h5> <ul> <li>Delays are often sensitive leverage points for policy, if they can be made shorter or longer.</li> <li>Delays determine how fast systems can react, how accurately they hit their targets, and how timely is the information that is passed around. Overshoots, oscillations, and collapses are all caused by delays.</li> <li>When there are long delays in feedback loops, some sort of foresight is essential. To act only when a problem becomes obvious is to miss an important opportunity to solve a problem.</li> </ul> <h5>Bounded Rationality</h5> <ul> <li>Bounded rationality means people make reasonable decisions based on the information they have, but they don't have perfect information, especially about distant parts of the system.</li> <li>Instead of finding a long-term optimum, we are blundering "satisficers," discovering within our limited purview a choice we can live with for now, and sticking to it until we are forced to change.</li> <li>We don't make decisions that optimize our individual good, much less the good of the system as a whole. We imperfectly interpret our imperfect information.</li> <li>Seeing how individual decisions are rational within the bounds of the information available does not excuse narrow-minded behavior, but it illuminates why that behavior arises.</li> <li>If the bounded rationality of each actor does not lead to decisions that furthers the welfare of a system, we must redesign the system to improve the information, incentives, disincentives, goals, stresses, and constraints that affect each actor.</li> </ul> <h4>Ch 5: System Traps... and Opportunities</h4> <ul> <li>We call the system structures that produce common patterns of problematic behavior archetypes.</li> </ul> <h5>Policy Resistance –&nbsp;Fixes that Fail</h5> <ul> <li>Policy resistance is when balancing loops persist undesirable behaviors despite efforts to invent technological or policy fixes.</li> <li>Policy resistance comes from the bounded rationalities of the actors in a system, where each actor has goals that are inconsistent with the goals of others.</li> <li>In such a system, each actor has to put great effort into keeping a system where no one wants to be, because if any actor gives up then the others will drag the system closer to their goals.</li> <li>The way out: Find a way of aligning the various goals of the subsystems, usually by providing an overarching goal that allows all actors to break out of their bounded rationality.</li> </ul> <h5>The Tragedy of the Commons</h5> <ul> <li>For a system to be subject to tragedy, the resource must not only be limited, but erodable. Beyond some threshold it cannot regenerate itself and is more likely to be destroyed.</li> <li>A commons system also needs users of the resource, which have good reasons to increase, and which increase at a rate that is not influenced by the conditions of the commons.</li> <li>The tragedy of the commons arises from missing (or too long delayed) feedback from the resource to the growth of the users of that resource.</li> <li>The structure of a commons system makes selfish behavior much more convenient and responsible than behavior that is responsible to the whole community and to the future.</li> <li>The way out: Educate and exhort, privatize the commons (divide it up so that each person reaps the consequences of his or her own actions), or regulate the commons (enforced by policing and penalties).</li> </ul> <h5>Drift to Low Performance</h5> <ul> <li>Drift to low performance happens when a system not only resists policy and stays in a normal bad state, but it keeps getting worse.</li> <li>The actor believes bad news more than good news, and so the actor thinks things are worse than they are. Standards aren't absolute, and so as perceived performance slips, the goal is allowed to slip.</li> <li>The lower the perceived system state, the lower the desired state, and so the less the discrepancy, and so the less the corrective action, and so the lower the system state.</li> <li>The way out: Keep standards absolute, regardless of performance. And make goals sensitive to the best performances of the past, instead of the worst.</li> </ul> <h5>Escalation</h5> <ul> <li>Escalation comes from a reinforcing loop set up by competing actors trying to get ahead of each other.</li> <li>If nothing is done to break the loop, the process usually ends with one or both of the competitors breaking down.</li> <li>The way out: Unilateral disarmament, thereby refusing to compete and interrupting the reinforcing loop. Or negotiate a new system with balancing loops to control the escalation.</li> </ul> <h5>Success to the Successful –&nbsp;Competitive Exclusion</h5> <ul> <li>This is a reinforcing feedback loop where winners in a competition receive, in the reward, a means to compete even more effectively in the future. And so winners go on winning, while losers go on losing.</li> <li>If everything the winner wins is extracted from the losers, then the losers are gradually bankrupted, or forced out, or starved.</li> <li>The "competitive exclusion principle" says given two species competing for the same resource, one will reproduce faster or use the resource more efficiently, and so it reproduces faster and drives the other to extinction.</li> <li>One way out is diversification (or exploiting new resources), but this is not a strategy for the poor when the monopolizing firm can crush or buy up all offshoots.</li> <li>Other ways out: Add a feedback loop to stop any competitor from taking over entirely, or by periodically "leveling the playing field."</li> </ul> <h5>Shifting the Burden to the Intervenor –&nbsp;Addiction</h5> <ul> <li>An intervenor may step in to bring a system to a desirable state, but the original problem will reappear since no one has fixed whatever feedback process is not maintaining the state of the system.</li> <li>The trap is formed if an intervention undermines the capacity of the system to maintain itself. Increased intervention only weakens the capacity of the original system, creating a vicious cycle.</li> <li>Addiction is finding a quick and dirty solution to the symptom of the problem, which distracts one from the harder and longer-term task of solving the real problem.</li> <li>The way out: Intervene in a way that strengthens the ability to shoulder its own burdens, which can be cheaper and easier than taking over and running the system. Then remove yourself.</li> </ul> <h5>Rule Beating</h5> <ul> <li>Rule beating is evasive action to get around the intent of a system's rules&nbsp;– abiding by the letter, and not the spirit, of the law.</li> <li>Rule beating becomes a problem when it leads a system into large distortions, unnatural behaviors that would make no sense at all in the absence of the rules.</li> <li>Rule beating is usually a response of the lower levels in a hierarchy to over-rigid, unworkable, or ill-defined rules from above.</li> <li>The way out: Better explain the rules, or re-design the rules to release creativity not in the direction of beating the rules, but in the direction of achieving their purpose.</li> </ul> <h5>Seeking the Wrong Goal</h5> <ul> <li>If the goal of a system is designed badly, if it doesn't measure what it's supposed to measure, if it doesn't reflect the real welfare of the system, then the system can't produce a desirable result.</li> <li>Confusing effort with results is one of the most common mistakes in designing systems around a wrong goal.</li> <li>You have the problem of wrong goals when something stupid happening "because it's the rule." You have the problem of rule beating when something stupid happens as a way around the rule.</li> <li>The way out: Specify indicators and goals that reflect the real welfare of the system.</li> </ul> <h3>Part 3: Creating Change –&nbsp;in Systems and in Our Philosophy</h3> <h4>Ch 6: Leverage Points –&nbsp;Places to Intervene in a System</h4> <ul> <li>Leverage points are frequently not intuitive because as systems become more complex, their behavior becomes more surprising.</li> <li>In the end, it seems that mastery has less to do with pushing leverage points than it does with strategically, profoundly, madly, letting go and dancing with the system.</li> </ul> <h5>12. Numbers –&nbsp;Constants and parameters such as subsidies, taxes, standards</h5> <ul> <li>Different hands on the faucets may change the rate at which they turn, but they're still the same faucets plumbed into the same system, turned according to the same information and goals and rules.</li> <li>Parameters become leverage points when they go into ranges that kick off one of the higher items on this list.</li> </ul> <h5>11. Buffers -&nbsp;The sizes of stabilizing stocks relative to their flows</h5> <ul> <li>You can often stabilize a system by increasing a buffer. But if a buffer is too big, the system gets inflexible, and it reacts too slowly.</li> </ul> <h5>10. Stock-and-Flow Structures –&nbsp;Physical systems and their nodes of intersection</h5> <ul> <li>After a structures is built, the leverage is in understanding its limitations and bottlenecks, using it with maximum efficiency, and refraining from fluctuations or expansions that strain its capacity.</li> </ul> <h5>9. Delays –&nbsp;The lengths of time relative to the rates of system changes</h5> <ul> <li>Delays that are too short cause over-reaction, oscillations amplified by the jumpiness of the response. Delays that are too long cause damped, sustained, or exploding oscillations, depending on how much too long.</li> <li>Delay length would be a high leverage point, except for the fact that delays are not often easily changeable.</li> </ul> <h5>8. Balancing Feedback Loops –&nbsp;The strength of the feedbacks relative to the impacts they are trying to correct</h5> <ul> <li>The strength of a balancing feedback loop is important relative to the impact it is trying to correct. If the impact increases in strength, so must the feedbacks.</li> </ul> <h5>7. Reinforcing Feedback Loops –&nbsp;The strength of the gain of driving loops</h5> <ul> <li>A system with an unchecked reinforcing loop will ultimately destroy itself.</li> <li>Slowing down the growth of the reinforcing loop is usually a more powerful leverage point than strengthening balancing loops.</li> </ul> <h5>6. Information Flows –&nbsp;The structure of who does and does not have access to information</h5> <ul> <li>Given missing information flows, adding or restoring information is usually much easier and cheaper than rebuilding physical infrastructure.</li> <li>There are so many missing feedback loops because we tend to avoid accountability for our own decisions.</li> </ul> <h5>5. Rules – Incentives, punishments, constraints</h5> <ul> <li>The rules of a system determine its scope, its boundaries, its degrees of freedom.</li> <li>Power over the rules is real power. If you want to understand the deepest malfunctions of systems, pay attention to the rules and who has power over them.</li> </ul> <h5>4. Self-Organization – The power to add, change, or evolve system structure</h5> <ul> <li>Self-organization means changing any aspect of a system lower on this list –&nbsp;adding new balancing or reinforcing loops, or new rules. It is the strongest form of system resilience.</li> <li>It is basically a matter of an evolutionary raw material –&nbsp;a highly variable stock of information – and a means for experimentation, for selecting and testing new patterns.</li> <li>This intervention is unpopular, as encouraging variability and experimentation and diversity means "losing control."</li> </ul> <h5>3. Goals&nbsp;– The purpose or function of the system</h5> <ul> <li>Even people within systems don't often recognize what whole-goal system they are serving.</li> <li>Changing the person at the top of a system can change its goals and therefore radically change its behavior.</li> </ul> <h5>2. Paradigms –&nbsp;The mind-set out of which the system (its goals, structures, rules, delays, parameters) arises</h5> <ul> <li>The shared idea in the minds of society, the great big unstated assumptions, constitute that society's paradigm, or deepest set of beliefs about how the world works.</li> <li>Paradigms are the sources of systems. From them, from shared social agreements about the nature or reality, come system goals and information flows, feedbacks, stocks, flows, and everything else.</li> <li>There's nothing physical or expensive or slow in the process of paradigm change. In a single individual it can happen in a millisecond.</li> </ul> <h5>1. Transcending Paradigms</h5> <ul> <li>Every paradigm, including the ones you hold dear, is a tremendously limited understanding of an immense and amazing universe that is far beyond human comprehension.</li> <li>If you think no paradigm is right, you can choose whatever one will help to achieve your purpose.</li> </ul> <h4>Ch 7: Living in a World of Systems</h4> <ul> <li>Seeing the future exactly and preparing for it perfectly is unrealizable. Making a complex system do just what you want it to do can be achieved only temporarily, at best.</li> <li>For any objective other than the most trivial, we can't optimize. We don't even know what to optimize.</li> <li>But we can listen to what the system tells us, and discover how its properties and our values can work together to create something much better than could ever be produced by our will alone.</li> <li>Living successfully in a world of systems requires our full humanity – our rationality, our ability to sort out truth from falsehood, our intuition, our compassion, our vision, and our morality.</li> </ul> <h6>Get the Beat of the System</h6> <ul> <li>Starting with the behavior of a system forces you to focus on facts, not theories, and helps you avoid any beliefs or misconceptions.</li> <li>Moreover starting with the history of a system discourages us from defining a problem not by its actual behavior, but by the lack of our favorite solution.</li> </ul> <h6>Expose Your Mental Models to the Light of Day</h6> <ul> <li>Everything you know, and everything everyone knows, is only a mental model.</li> <li>Exposing your mental models, making them rigorous, testing them against the evidence, and scuttling them if they are no longer supported is simply practicing the scientific method.</li> </ul> <h6>Honor, Respect, and Distribute Information</h6> <ul> <li>Most of what goes wrong in a system is from biased, late, or missing information. So do not distort, delay, or withhold information.</li> </ul> <h6>Use Language with Care and Enrich It with Systems Concepts</h6> <ul> <li>The language and information systems of an organization are not an objective means of describing an outside reality&nbsp;–&nbsp;they structure the perceptions and actions of its members.</li> <li>Keep language as concrete, meaningful, and truthful as possible. And then enlarge language to make it consistent with our enlarged understanding of systems.</li> </ul> <h6>Pay Attention to What Is Important, Not Just What Is Quantifiable</h6> <ul> <li>Our culture is obsessed with numbers, and has therefore given us the idea that what we can measure is more important than what we can't measure.</li> <li>We have been endowed not only with the ability to count, but also with the ability to assess quality. Be a quality detector.</li> </ul> <h6>Make Feedback Policies for Feedback Systems</h6> <ul> <li>The best policies contain meta-feedback loops –&nbsp;or loops that alter, correct, and expand loops. They design learning into the management process.</li> </ul> <h6>Go for the Good of the Whole</h6> <ul> <li>Don't maximize parts of the system while ignoring the whole. Don't go to great trouble to optimize something that never should be done at all.</li> </ul> <h6>Listen to the Wisdom of the System</h6> <ul> <li>Before you charge in to make things better, pay attention to the value of what's already there. Don't destroy the system's own self-maintenance policies.</li> </ul> <h6>Locate Responsibility in the System</h6> <ul> <li>This means looking for the ways the system creates its own behavior.</li> <li>"Intrinsic responsibility" means that the system is designed to send feedback about the consequences of decision making directly, quickly, and compellingly to decision makers.</li> </ul> <h6>Stay Humble – Stay a Learner</h6> <ul> <li>When you're learning take small steps, constantly monitor, and change course as you find out more about where it's leading.</li> <li>Error-embracing is the condition for learning. It means seeking and using –&nbsp;and sharing – information about what went wrong with what you expected or hoped would go right.</li> </ul> <h6>Expand Time Horizons</h6> <ul> <li>In a systems sense, there is no long-term or short-term distinction. Phenomena at different time scales are nested within each other.</li> </ul> <h5>Defy the Discipline</h5> <ul> <li>Interdisciplinary communication works only if there is a real problem to be solved, and if its representatives are more committed to solving the problem than being academically correct.</li> </ul> <h5>Don't Erode the Goal of Goodness</h5> <ul> <li>The most damaging example of the systems archetype "drift to low performance" is the process by which modern industrial culture has eroded the goal of morality.</li> <li>Don't weigh the bad news more heavily than the good. And keep standards absolute.</li> </ul> https://tedneward.github.io/Research/thinking/untools/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/untools/index.html <p><a href="https://untools.co/">Website</a></p> https://tedneward.github.io/Research/tools/4coder/index.html Mon, 13 Apr 2026 14:04:56 +0000 tools/4coder/index.html <p><a href="http://4coder.net/">Website</a> | <a href="https://mr-4th.itch.io/4coder">Website (Itch)</a> |</p> https://tedneward.github.io/Research/thinking/simplexity/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/simplexity/index.html <p><em>(by Jeffrey Kluger, Hyperion, ISBN 978-1-4013-0301-3)</em></p> <h1>Prologue</h1> <p>Cholera outbreak in London; John Snow (physician/investigator) knew there was a source to the outbreak, but needed the data to prove it; canvassed the neighborhood, tracked it to a pump on Broad Street. Knew two things: Plagues were fantastically complex things--with the illness working myriad horrors in the body and spreading across the landscape in myriad ways. But diseases moved through simple choke points too--one person's handkerchief, one person's handshake, one handle on one foiled pump--all of them bottlenecks in the pathogen's flow. Seal off one, stop the disease. The complex illness could collide hard with the simple fix.</p> <p>M.Coy bookshop (Michael Coy and Michael Brasky) in Seattle vs Amazon in the complex art/science of book recommendations. The machines aren't parsing data as much as steam-shoveling it, digging up vast masses of information; the likes of Brasky and Coy do the very same thing, but finely/instantly using human clues. It's easy to say that one of the two approaches is more complex than the other; it's a lot harder to say which one.</p> <p>Complexity is a slipper idea, one that defies almost any effort to hold it down and pin it in place. Things that seem complicated can be preposterously simple; things that seem simple can be dizzyingly cpmlex. Manufacturing plant vs houseplant. Colony of garden ants vs community of people. Sentence vs book, couplet vs song, hobby shop vs corporation. "Human beings are not wired to look at matters that way; we're suckers for scale. Things that last for a long time impress us more than things that don't, things that scare us by their sheer size strike us more than things we dwarf. Star vs guppy. Consider a pencil: cedar (wood for the dowel), bauxite (aluminum sleeve), coal (graphite), polymers (eraser); who feeds Paris?</p> <p>Psych is waking up to complexity too. Six-child family vs one-child family, complex in different ways. Political science plays simplicty-complexity, too; equally complex the stock market, crowds, geology, biology, even politics.</p> <p>Trying to distill this down to a working definition of complexity and simplicity is hard.</p> <ul> <li>Simple things are those things that create order in their environments, the way, say, modern design strips the clutter from a building; but complex things create order too, the way cogs and springs and flywheels of a watch yield a single, simple piece of data--the time--which in turn runs the world.</li> <li>Simple things create disorder, then, the way an underregulated economy can let markets and monopolies run wild; but complex things overrgulate human affairs can produce a Byzantine mess like the tax code.</li> <li>We call some software simple because it's easy to use, never mind that it relies on elaborate complex ancient code; we call older computers complex because they were hard to use, never mind the fact that by today's standards they're crude.</li> <li>Is free-form dance comparatively simple thanks to its lack of rigid rules? Or does that liquidity make it much harder to learn?</li> <li>Does a committee struggling over a problem lack the straightforward insight of a single wise mind, or is the wise mind blinkered by its biases, while the committee--with its clarifying tensions--is a better, ultimately less coplex arbiter?</li> </ul> <p>Brain is a real-time machine, constantly scanning for input and assembling into impressions and actions; that was useful for survival, but it can mislead us now, causing us to overfocus on the most conspicuous features of a thing and be struck--or confused--by that quality. Thus we are confused by beauty, by speed, by big numbers, by small numbers, by our small fear, ....</p> <h1>1: Why is the stock market so hard to predict?</h1> <p><em>(Confused by Everyone Else)</em></p> <h3>Markets react to Baker's comments on Germany</h3> <p>Treasury Secretary James Baker III, trip to Bonn to discuss Germany lower their interest rates and juice the mark, German finance minister did not agree, and Baker was unhappy: "not particularly pleased". This spooked the stock market, which triggered the crash of 87. Nothing smart or sensible had taken place that day--but smart and sensible forces had not ben at work, market forces had been. For every market analyst who sees traders as the informed and educated people they surely can be, there are scientists who see them another way: as unthinking actors who obey not so much the laws of economics as the laws of physics. Investors react not so much to variables that are in their interests, but to those that are in everyone else's interests. When the tide of the market shifts, most of us shift with it; when it flows back the other way, we do the same. We like to think we're informed by trends, but often as not, we're simply fooled by them--snookered by what everyone else is doing into concluding that we ought to do the same. "Economic models begin with the assumption of perfect rationality, of a universe of logical people all doing what they can to master their utility; physicists studying economics begin with the assumption that people can't think." (J. Doyne Farmer, Santa Fe Institute in New Mexico). "The term we use is zero-intelligence investors" (John Miller, economist at Carnegie Mellon University)</p> <h3>How do we tell if something is simple or complex?</h3> <p>One of Murray Gell-Mann's favorite ways of deciding whether something is simple or complex is to ask a decidedly unscientific question: How hard is it to describe the thing you're trying to understand? "Start with the minimum description length as your first inquiry. The shorter it is, the simpler the thing is likely to be. In most cases, you don't even have to take your language from scientific discourse. It can come from ordinary speech." But description depends on context, and context changes everything. "Imagine an anthropologist approaching a civilization with which he shares a common language, but which is naive of any culture outside of its own. Now imagine trying to explain to that community a tax-managed mutual fund. What do you think the preamble to that explanation would be?" Description length has its limits; the problem is that it begins with an assumption of a clean and consistent line, with simplicity and short descriptions at one end and complexity and long descriptions at the other. A clean line, however, doesn't really capture things as much as a somewhat messier arc does.</p> <p>Complexity scientists like to talk about hte ideas of pure chaos and pure robustness--and both are exceedingly simple things. Either extreme is uninteresting. Where you'd find rea complexity would be somewhere between those two states, the point at which the molecules begin to climb from disorder, sorting themselves into something interesting and organized, but catching themselves before they descend down the other side of the complexity hill. The more precisely the object can balanc at the pinnacle of that arc, the more complex it is. "It's the region between order and disorder that gives you complexity, not the order and disorder at the ends."</p> <p>Things are not even that straightforward as even that explanation suggests, since any one system is not necessarily composed of just one point on the arc. Many different points come into play and the question of complexity turns on which one you choose. A foot-long copper pipe might be nearly as static as frozen carbon, but take in the vast array of skyscraper plumbing (of which it is just a part) and things look a lot more complicated.</p> <h3>Handshakes</h3> <p>Same holds true for human behavior. Consider all that goes into a handshake.</p> <p>Any system (chemical, physical, cultural, fiscal) must be seen at all levels before you can begin to make a real determination about whether it can truly be called complex. "Ask me why I forgot my keys this morning and the answer might be simply that my mind was on something else. Ask me about the calcium channels in my brain that drive remembering or forgetting and you're asking a much harder question."</p> <h3>Markets</h3> <p>Blake LeBaron (Brandeis University) has an entire (virtual) stock exchange of his own. Over the years he's developed algorithms that allow him to simulate any kind of market--bull, bear, static, active, mixed--and then release simulated investors into that environment to see how they behave. On the whole, artificial traders behave precisely like the real ones, which is to say that they never show much imagination. "At the beginning of a run, all the traders just wobble around a bit, looking for guidance. Then someone will try a new strategy and do quite well at it and the others will notice. Pretty soon, a few more are trying it and it starts to get popular, like a clothing fad. Ultimately everyone starts to converge on that strategy and it dominates the market, precisely the way real markets behave." But it only goes so long--if everyone is chasing the same dollar in the same way, it takes only a few players to cash out before all of the other shares start to lose value. This is the classic bubble-popping phenomenon, one that's familiar to all investors. "The market first becomes too stable, too robust, then it collapses into instability.</p> <h3>Fish schools</h3> <p>Fishkeepers (aquarist) can learn a lot about chemistry, botany, animal behavior by watching what takes place on the other side. Study how schools of fish manage their fluid movements without any evident alpha fish leading. (Bird flocking, too?) In order for a traveling fish school to remain cohesive, all of the individuals must maintain a position no more or no less than about a single body length from every individual around them. Get too close, and you collide and compete. Drift too far apart, and you begin to stand out--never a good idea when predators lurk. The most important thing the fish thus keep in mind is not where they're going or who's leading them there, but making sure they don't fall out of ranks along the way.</p> <p>But who's in charge of the motion? Schools do require at least a few leaders ("informed individuals"), but not many. In an average-sized school, it takes only about 5% of the members to know the proper route and set out in that direction. What's more, as the school grows, that leadership share actually shrinks, so that as few as one in a hundred need have any idea what the goal is and yet they still lead all the others.</p> <h3>Power of combined brainpower</h3> <p>The jellybean contest: The average of everybody's guess will come signficantly closer than any one person's guess. "It happens virtually every time; it's freakish and it still amazes me." (Brooke Harrington, professor of public policy, Brown University) More than just laws of probability here--we could all guess high or guess low. Instead, we all guess in such a precise distribution around the target that together we practically hit it. The greater the number of people participating, the closer the collaborative guess becomes.</p> <p>Harrington wanted to pursue this more-is-more finding further; examined investment clubs (bringing largely inexperienced traders together in the hopes that they will collaboratively choose better). Confirmation: In general, the average group did better than the average individual; and larger, more diverse groups did better than smaller, more homogeneous ones.</p> <h3>The fallacy of the "jumpy market"</h3> <p>One of the great bits of stock market wisdom is that investors of all kinds are jumpy and esily frightened by bad headlines. In 1989, economists selected the 49 most newsmaking events from 1941 to 1987 (including Pearl Harbor, JFK's assassination, Reagan's assassination attempt, Chernobyl). Did the news shake the market? Using the Standard &amp; Poor's Index as their yardstick, what they found was that even on the S&amp;P, bad news in the papers did not have to mean bad news on Wall Street. Biggest drop of all 49 days was a 6.62% drop in 1955 when President Eisenhower had a hard attack. (Compare: 23% loss in 1929 and 1987.) Next biggest drops were 5.38% when North Korea invaded in 1950, and 4.37% loss after Pearl Harbor. Study next looked the other way, examining biggest drops and looking for the news of that day. In only nine of the cases did a nonfinancial news story like international tensions or a political shake-up sem to account for the market movement.</p> <h3>Fairness</h3> <p>One other exceedingly complex and uniquely emotional variable individual traders bring to the table: fairness; what it looks like or feels like when something looks like a square deal or when the bad guys are about to get away with omething. Researchers long wondered just how powerful the human fairness impulse is. Samuel Bowles (professor economics, University of Siena (Italy)) conducted a fairness test: Two volunteers, V1 and V2, are given a quantity of money to share which they get to keep if they can agree on how to divide it. V1 proposes the split, V2 accepts it. Perfectly rational actors will accept any split, since it's by definition more than what they had when they came in, but the average accepted bid is 57%/43%. "When you ask them why they reject a 20% share, they answer, 'Because that son of a bitch was getting 80%.' So people are willing to pay 20% just to punish a son of a bitch."</p> <h3>Conclusion</h3> <p>The key in all of these situations is to recognize that there is indeed great wisdom in what everyone else knows. But there is sometimes greater wisdom in knowing what only you know. There may be no such thing as mastering the perfect balance of those skills, but there is such a thing as becoming powerfully--and in the case of the markets, profitably--good at it.</p> <h1>2: Why is it so hard to leave a burning building or an endangered city</h1> <p><em>(Confused by Instincts)</em></p> <p>(9/11, Ed Schmitt, and his decision to leave the building (Tower One) when the first plane struck Tower Two, when so many others stayed.) Ultimately, we are misled by our most basic instincts--the belief that we know where the danger is and how best to respond to keep ourselves alive, whenin fact we sometimes have no idea at all. It's the job of the people who think about such matters to tease all these things apart and put them back together in buildings and vehicles that keep their occupants alive; it's the job of those occupants to learn enough about the systems so that they have the sense to use them.</p> <h3>Buildings</h3> <p>20,000 people in a building or 1/2 million in a coastal city occupy the same spot on the complexity spectrum as air molecules filling a room--moving randomly and chaotically in all directions, filling all the available space more or less uniformly. Very active, but also very simple and disordered. Send the same people on a stampede down stairways or highways and things quickly grow overloaded and grind to a halt, jumping to the other end of the complexity arc--robust, unchanging, frozen in place, but every bit as simple as the ever-shifting air molecules. It's in the middle of the arc, where the molecules just begin to take some shape (or the people just begin to move to the exits) that true complexity begins to emerge.</p> <h3>Water</h3> <p>Best way to understand the manner in which people move <em>en masse</em> may be to understand the way water does the same, particularly how it navigates around obstacles or breakwaters. A foundered boat or tumbled boulder creates turbulence; a non-sensical post in the middle of the floor does the same. That turbulence staggers their arrival slightly, allowing them to stream through the opening in a reasonably controlled flow, rather than colliding there at once and causing a pileup. The obstacle keeps you at the top of the complexity arc, preventing you from reaching the frozen end. "By adding a little noise to the system you produce coherence in the flow."</p> <p>This, of course, assumes people behave sensibly. For one thing, people have different levels of decision-making skills, with some behaving more rationally than others. For another, all of us have a tendency to believe that the rest of the group knows what it's doing, and thus will gravitate toward a popular exit simply because other people have chosen it, even if the alternative is perfectly safe and much less congested. Finally, information tends to get distributed unevenly, with some people learning about an emergency first and acting before the others.</p> <h3>Stairs</h3> <p>Simulations and models can't simulate emotion/fear; for example, the only thing that ought to count in stairways is pure speed; anything short of a stampede should get the building emptied in a hurry. But pure speed is hard to maintain. It doesn't take much for a smooth stream of downward-flowing evacuees to turn turbulent. For one thing, people from middle floors who enter the stairwell somewhere in mid-current can cause things to slow or stop. The delay flows back along the queue like a ripple through water or cars entering a highway. Sequential evacuation is the best way to handle this problem, with people on lower floors leaving first and each high floor following successively; but nobody pretends that people fearing for their lives would wait patiently at their desks until their floor is called. The design of the stairways themselves must provide the extra margin of speed. (Guiderails; lighting; fluorescent tape.) But stairways were too narrow, not taking into account that two-thirds of 2001 Americans are overweight or obese.</p> <h3>Social parts of an evacuation</h3> <p>Social norms about who's making decisions change during the day-to-day, and all these norms are broken in an emergency. What appears in a time of emergency is something complexity researchers call the emergent norm--that is a whole new set of rules.</p> <h1>3: How does a single bullet start a world war?</h1> <p><em>(Confused by Social Structure)</em></p> <h1>4: Why do the jobs that require the greatest skills pay the least? Why do companies with the least to sell often earn the most?</h1> <p><em>(Confused by Payoffs)</em></p> <h1>5: Why do people, mice, and worlds die when they do?</h1> <p><em>(Confused by Scale)</em></p> <h1>6: Why do bad teams win so many games, and good teams lose so many?</h1> <p><em>(Confused by Objective)</em></p> <h1>7: Why do we always worry about the wrong things?</h1> <p><em>(Confused by Fear)</em></p> <h1>8: Why is a baby the best linguist in the room?</h1> <p><em>(Confused by Silence)</em></p> <h1>9: Why are your cellphone and camera so absurdly complicated?</h1> <p><em>(Confused by Flexibility)</em></p> <h1>10: Why are only 10% of the world's medical resources used to treat 90% of its ills?</h1> <p><em>(Confused by False Targets)</em></p> <h1>11: Why does complexity science fall into the arts?</h1> <p><em>(Confused by Loveliness)</em></p> https://tedneward.github.io/Research/thinking/systems-thinking/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/systems-thinking/index.html <h2>Articles/Blog posts</h2> <p><a href="https://alexdanco.com/2021/08/21/dancoland/">Dancoland</a>: A <em>great</em> webcomic about how to learn systems thinking analysis. Stocks, flows, etc.</p> <p><a href="https://apenwarr.ca/log/20201227">Systems design explains the world: volume 1</a> | <a href="https://apenwarr.ca/log/?m=202304">Systems design 2: What we hope we know</a></p> <p><a href="https://apenwarr.ca/log/20180724">Books that explain (parts of) how the world really works</a></p> https://tedneward.github.io/Research/thinking/quit/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/quit/index.html <p><em>(by Annie Duke, Penguin Random House, ISBN 9780593544020)</em></p> <h1>Prologue: The Gaffed Scale</h1> <p>Muhammad Ali: GOAT, and yet didn't know when to quit</p> <p><strong>Grid vs Quit</strong> We view grit and quit as opposing forces. While grit is a virtue, quit is a vice. Some goals are just not reachable, no matter how hard you continue. We ought not to confuse hindsight with foresight, which is what these aphorisms do. <em>(Survivor bias)</em> Success does not lie in sticking to things; it lies in picking the right thing to stick to and quitting the rest.</p> <p><strong>Wrapped in Euphemism.</strong> Lindsey Vonn: quit, but then said "I'm not giving up! I'm just starting a new chapter." The idea of quitting is such a bitter pill to swallow that we have to take it with a spoonful of sugar, or in this case, a spoonful of euphemism, the most famous of which is "pivot." We ought to stop thinking that we need to wrap the idea of quitting in bubble wrap. When it comes to quitting, the scale is gaffed (butcher's thumb on the scale to cheat customers).</p> <p><strong>Science Says.</strong> The book <em>Grit</em> is interpreted to mean "Never quit" but that's misinterpretation. She has herself written about the importance of trying lots of things (which requires that you quit lots of other things) to find the thing you want to stick with. There is a rich universe of science studying the human tendency to persevere <em>too long</em>, particularly in the face of bad news. Sunk cost, status quo bias, loss aversion, escalation of commitment, and so on. Kahneman's and Thaler's work. Science is telling us every day, we tend to stick to things too long in the face of signals that we ought to quit.</p> <h1>Section I: The Case for Quitting</h1> <h2>Ch 1: The Opposite of a Great Virtue is Also a Great Virtue</h2> <p><strong>The Invisible Men at the Top of the World.</strong> Mt Everest: a place where people really should learn to quit sooner. Three climbers: Dr Stuart Hutchison, Dr John Taske, Lou Kasischke. Part of a commercial guided expedition of Everest. Expedition leader stressed importance of having strictly-observed turnaround times (return to camp if you've not yet reached your destination) for each day's climb up the mountain. (Eight times more people die on Everest on the way down than on the way up.) Turnaround times are there to prevent people from making poor decisions to keep going when they're in the shadow of the summit, building into a climbing plan three critical concepts:</p> <ol> <li>persistence is not always a virtue</li> <li>making a plan for when to quit should be done long before you are facing the quitting decision</li> <li>the turnaround time is a reminder that the real goal in climbing Everst is not to reach the summit, but to return safely to the base of the mountain</li> </ol> <p><em>(Simple Rules!)</em></p> <p>Hutchison, Taske, Kasischke were part of one of three expeditions, making the climb too crowded. Trio got stuck, with expedition leader, behind a group of incompetent climbers. At 11:30am, Hutchison called a meeting, realized that turnaround time was 1pm, and they wouldn't make it. Turned around. Made it safely back to the camp. Rest of the group was well-documented (book <em>Into Thin Air</em>, documentary <em>Everest</em>, 2015 film <em>Everest</em>), expedition leader Hall (who set the turnaround time!) and four others who reached the summit died on various parts of their descent back. But we don't know the Hutchison/Taske/Kasischke trio, because we tend to think about only one side of the human response to adversity (the ones who go for it). Even in this life-or-death situation, where quitting is the right decision, we don't remember the quitters at all.</p> <p><strong>Quitting is a Decision-Making Tool.</strong> Grit and quit are two sides of the same exact decision; you can't decide one thing without deciding the other. Grit is what gets you up the mountain, but quit is what tells you when to come down. <strong><em>"If I had to skill somebody up to get them to be a better decision-maker, quitting is the primary skill I would choose, because the option to quit is what allows you to react to a changing landscape."</em></strong> Any decision is made under some degree of uncertainty, stemming from two sources:</p> <ol> <li>The world is stochastic: luck makes it difficult to predict exactly how things will turn out, at least in the short run. We operate not with certainties but with probabilities.</li> <li>When we make decisions, we don't have all the facts.</li> </ol> <p>When you take all these aspects of uncertainty together, it makes decision-making hard. But, after you've set out on a particular course of action, <strong><em>new information will reveal itself to you</em></strong>. Feedback. Quitting is the tool that allows you to keep from being paralyzed by uncertainty or stuck forever in every decision you make. Quitting is what allows companies to maximize speed, experimentation, and effectiveness in highly uncertain environments.</p> <p>Richard Pryor--greatest comedian ever--would routinely do shows at the Comedy Store on Sunset Strip with nothing prepared beyond "a couple of ideas" or "one or two jokes at the most" and bomb. The next night, he'd drop everything that didn't work--nearly the whole set--and expand on whatever got a titter. At the end of 30 days, he had 40 minutes of incredible material. <em>(OODA! He had a freakin 24-hour OODA loop!)</em></p> <p><strong>The Siren Song of Certainty.</strong> Uncertainty is an impediment to making good decisions about quitting. Only one choice--the choice to persevere--lets you (eventually) find out the answer. The desire for certainty is the siren song calling us to persevere, because perseverance is the only path to avoiding the ambiguity of not knowing how it all turned out.</p> <p><strong>The Super Bowl is a Corporate Graveyard.</strong> The list of Super Bowl advertisers in Tom Brady's football career who've failed is long and impressive: AOL, Blockbuster, Circuit City, CompUSA, Gateway, Radio Shack, Sears. Super Bowl ads are big and expensive--if you could afford one, you were a big, successful company. Each couldn't maintain their success--the scale must be gaffed against quitting not just for individuals but for businesses as well. <strong><em>The road to sustained profitability for a business is not only about sticking to a strategy or business model, it is also about surveying and reacting to the changing landscape.</em></strong></p> <p><strong>"Know When to Hold 'Em, Know When to Fold 'Em": But Mostly, Fold 'Em.</strong> Three of the four things from Kenny Rogers' lyric are about quitting. When it comes to cutting your losses at the poker table, he "got it". Optimal quitting might be the most important skill separating great poker players from amateurs. Top poker players are know when to fold 'em: Pros play a mere 15% to 25% of the two-card starting combinations they are dealt in Texas Hold 'Em. Amateurs stick with over half. <strong><em>To be good at the game you ahve to learn to live with the idea that you folded some hands that might've won.</em></strong> Quitting a game is a decision fraught with uncertainty because it is never clear exactly why you are losing--you could be playing really well but losing because of an unlucky run of cards. In other words, if you want to glame your losses on luck and keep playing, you can always find a way to do that. Quitting a game when you're losing is the only way to guarantee you won't get those chips back in that game, which makes quitting when you are losing difficult. <em>(Again, here's the power of a stopping rule: Take the emotion and ego out of the decision by creating the stopping rule before it becomes an issue.)</em></p> <p><strong>Summary:</strong></p> <ul> <li>We tend to celebrate people who respond to adversity by soldiering on. The quitters, in comparison, are invisible.</li> <li>If we don't notice the decision-making of the quitters, it's hard to learn from them.</li> <li>Quitting a course of actio is sometimes the best way to win in the long run, whether you're cutting your losses at the poker table or getting to climb another day.</li> <li>Quit and grit are two sides of the exact same decision.</li> <li>Decision-making in the real world requires action without complete information. Quitting is the tool that allows us to react to new information that is revealed <em>after</em> we make a decision.</li> <li>Sticking with a course of action is the only way to find out for sure how it will turn out. Quitting requires being okday with not knowing what might have been.</li> <li>Having the option to quit helps you to explore more, learn more, and ultimately find the right things to stick with.</li> </ul> <h2>Ch 2: Quitting on Time Usually Feels Like Quitting Too Early</h2> <p>In 2002 Stuart Butterfield built <em>Game Neverending</em>, a cooperative game, and in 2004 it failed due to lack of funding in post-dot-com-crash environment. They salvaged one feature, an inventory of objects the players accumulated, represented by a shoebox of photos, which became Flickr, which within a year they sold to Yahoo for $25m. Butterfield left Yahoo in 2008 and created a new game, <em>Glitch</em>, secured a bunch of funding (due to Flickr's rep) and released it in 2011; it looked amazing, and by Nov 2012, it had a devoted following of 5,000 users playing 20+ hrs/week. Problem: paid users were less than 5% of the user base, the rest having done the free sign-up; over 95% of new users played <em>Glitch</em> for less than seven minutes and never returned. Made a major PR push, yet after a particularly good push, Nov 12, Butterfield sent an email to investors: "I woke up this morning with the dead certainty that Glitch was over." He was able to peek into the future, and saw that in the range of possible outcomes, the probability was too high that the game would end up being a money pit. The math of the conversion rate just didn't work. "I think I knew this six weeks ago and I mistook denial for prudence (in the sense of making sure that we didn't give up too early). But there are just too many things in the 'against' column." This freed him and the company to work on a different product: the "Searchable Log of All Conversation and Knowledge", aka Slack.</p> <p><strong>Quit While You Still Have a Choice.</strong> <strong><em>Quitting on time wil usually feel like quitting too early.</em></strong> Making good decisions about quitting requires mental time travel since the worst time to make a decision is when you're in it. That's when you are in the present, facing down the decision whether or not to cut your losses, unable to see past what is happening right now. When we do think about the future, we are often considering our hopes, our goals, our ambitions, and that optimism means that too often we allow a disastrous future to hurtle toward us, noticing it only as it's arriving on our doorstep. There is a well-known heuristic in management consulting that "the right time to fire someone is the first time it crosses your mind". But since this is a decision made under uncertainty, we tend to persist too long.</p> <p><strong>Thinking in Expected Value.</strong> To get the stick-or-quit decision right, you need to make an educated guess at the probability that things will go your way and the probability that things will go against you in order to figure out if the good stuff will occur enough of the time to warrant continuing on the same path. You need to think in Expected Value (EV). EV helps you answer two questions:</p> <ol> <li>it tells you whether any option you are considering is going to be, on balance, positive or negative for you in the long run.</li> <li>it allows you to compare different options to figure out which is the better choice, where "better choice" is the one that carries the highest expected value.</li> </ol> <p>To determine the EV for any course of action, you start with identifying the range of reasonably possible outcomes, each of which will have some probability of occurring. If you multiply the probability of the outcome (p) with the outcome's good-or-bad (o), and add that all together, you get the EV. Example: fair coin flip (.5 heads, .5 tails) where heads = +$100, tails = -$50. Heads <code>p * o</code> is +$100 * .5 = +$50, Tails <code>p * o</code> is -$50 * .5 = -$25, EV is (+$50 + -$25), or +$25, or a positive EV. Example 2: Unfair coin flip, heads +$50 at p 0.9 (+$45) vs tails -$100 at p 0.1 (-$10), EV is +$35. Example 3: heads +$100,000 at p 0.01 (+$1,000), tails -$100 at p 0.99 (-$99), EV = +$901. <em>(All of this assumes we get 100 coin flips, though--if it's only one flip, then the decision-making changes here somewhat, I would think? I think Duke makes a subtle allowance for this: "Of course, that's a much riskier bet than the other two bets. Managing risk is the subject of many other books, but not this one.")</em></p> <p>Next, compare the EV for each option you might be considering. "If I were to switch and do something else, would that have a higher EV than the thing I'm currently doing?" Winning poker players aren't thinking about trying to win a single hand, come what may--they know while any two cards can win, only some hands can win enough of the time to make them worth pursuing. They are making decisions based on whether playing or folding will have the greater EV.</p> <p><strong>Quitting Decisions are Expected-Value Decisions.</strong> 2001: Duke gets an email from a reader who wanted help working through a decision about whether to quit her job. Duke asks, "A year from now, if you stay, what's the probability that you're going to be unhappy at the end of that year?" 100%. "A year from now, if you take the new job, what's the probability that you're going to be unhappy?" "Not sure" "Is it 100%?" "Definitely not." Enlightenment. Duke reframed the question as an EV problem.</p> <p><strong>Time Travelers from the Past.</strong> Thinking in expected value is a kind of mental time travel, propelling yourself into the future to glimpse the range of possible outcomes and take some reasonable guess at how likely each of them is. As a means to becoming a better quitter, this works in both directions: You can get the benefit of listening to a message <em>from the past</em>. Adm William McRaven, one of the world's most respected figures on military strategy, US foreign policy, and counterterrorism ops, is a longtime student of military history: "Probably three quarters of the books behind me (on his wall of bookshelves) are history books about battles that went well and battles that went wrong." When you are making a decision about whether to quit, you need to listen to those people from the past who are giving you important advice, whether its others or yourself.</p> <p><strong>Flipping Coins.</strong> 2013: Steven Levitt (author, <em>Freakonomics</em>) put up a website inviting visotrs to flip a virtual coin to help them decide quit-or-stick. They would register what they were stuggling with, among a variety of types of decisions, like "Should I quit my job or stay?" or "Should I leave my relationship?" or "Should I stay in college?". The site would assign one side of the decision to Heads, the other to Tails, and then randomly flip the coin. 20,000 people over the course of a year did this. Obviously, these people must have felt that the choice of quit-or-stick was so close, so 50-50, that flipping a coin to help them decide seemed like a reasonable option. Theoretically, they would be equally likely to be happier with either heads or tails, stick or quit. But this isn't what Levitt found. When he followed up with the coin flippers at 2 months and 6 months later, he discovered that for the big life decisions, people who quit were happier on average than people who stuck, whether they quit on their own or after the coin flip suggested quitting. <strong><em>While the decisions may have felt close to the people making them, they were actually not close at all. As judged by the participants' happiness, quitting was the clear winner.</em></strong> Because people were much happier when they quit what they considered a close decision, that shows that people are generally quitting too late. Levitt concluded, "The results of this paper suggest that people may be excessively caution when facing life-changing choices." Corollary: <strong><em>When people quit on time, it will usually feel like they are quitting too early, because it will be long before they experience the choice as a close call.</em></strong></p> <p><strong>Jumping the Shark.</strong> (<em>Happy Days</em> pop-culture reference.) We have an expectation that people ought to have seen in foresight what we can so easily see in hindsight. And when they don't, we can't believe how obtuse they are. That's the point of jumping the shark: It's mocking someone who doesn't quit on time, no matter that it's much harder to see the shark in foresight. But the sad thing is that as much as we make fun of people who quit too late, when someone does manage to quit on time, we mock them for quitting too early. That's the "quitting bind".</p> <p><strong>The Quitting Bind.</strong> Dave Chappelle quits <em>Chappelle's Show</em> in season 3, at the top of his game. Rumors swirled why. He quit because he could see two things: (1) he was unhappy in a future where he continued the show, and (2) he could see the shark: he was close to crossing the line between his audience laughing with him, and his audience laughing at him. Similar disappointment around Phoebe Waller-Bridge (2019) ending <em>Fleabag</em>, which in two seasons (2016, 2019) earned massive worldwide acclaim. Fans beg for more, despite Waller-Bridge's explanation that ending the series was consistent with the arc of the title character.</p> <p><strong>Sunmmary:</strong></p> <ul> <li>Quitting on time usually feels like quitting too early.</li> <li>The hardest time to make a quitting decision is when you're in it.</li> <li>Our intuition is that quitting will slow down our progress. The reverse is actually true. If you walk away from something that is no longer worthwhile, that frees you up to switch to something that is more likely to help you achieve your goals--and you'll get there faster.</li> <li>When the time is objectively right to quit, nothing particularly dire will be happening right at that moment. Getting the timing right means looking into the future and seeing that the chances things will go your way are too slim.</li> <li>Thinking in expected value (EV) helps you figure out if the path you are on is worth sticking to. EV is not just about money. It can be measured in health, well-being, happiness, time, self-fulfillment, satisfaction in relationships, or anything else that affects you.</li> <li>If you feel like the choice between persevering and walking away is a close call, it's likely that quitting is the better choice.</li> <li>In hindsight, we can see when someone has waited too long to quit, and we tend ot be harsh in our judgment of those people. But when smoeone quits before it seems obvious to others, we mock them for quitting too early. That's the quitting bind.</li> </ul> <h2>Ch 3: Should I Stay or Should I Go?</h2> <p><strong>Summary:</strong></p> <ul> <li>A key finding of prospect theory is <em>loss aversion</em>, the phenomenon whereby the emotional impact of a loss is greater than the corresponding impact of an equivalent gain.</li> <li>Loss aversion creates a preference for options associated with a lower chance of incurring a loss. It makes us risk averse.</li> <li>When we are in the gains, we have a tendency to quit too early in order to avoid the risk of giving those gains back. In other words, we like to quit while we're ahead.</li> <li>When we are in the losses, we become risk seekers. We want to keep going, hoping we can avoid ever having to realize the loss. Kahneman characterizes this as <em>sure-loss aversion</em>. In other words, we like to stick when we're behind.</li> <li>Quitting on time usually feels like quitting too early, and the <em>usually</em> part is <em>specifically when you're in the losses</em>.</li> <li>Retail investors show this pattern of quitting when they're ahead and sticking when they're behind.</li> <li>Even expert investors don't get their quitting decisions just right. They outperform on their buying decisions but underperform on their selling decisions.</li> <li>We naturally track and get feedback on the things we are doing. But once we quit something, we also quit keeping track of that course of action. This creates a problem with getting high-quality feedback, which in turn makes it hard to hone our quitting skills.</li> </ul> <h1>Interlude I: Quitting When the World is Watching</h1> <p>2016: Alex Honnold prepares to free-solo-climb El Cap and a group of friends document the climb (which is them risking their lives, too). He quits. Comes back next June, crew reassembles, he successfully summits, free solo, and the documentary <em>Free Solo</em> is released in 2018, winning the Oscar for Best Documentary Feature. Had he not quit, he could easily have died, and the pressure was to stick.</p> <h1>Section II: In the Losses</h1> <h2>Ch 4: Escalating Commitment</h2> <p>Harold Staw, his rise, his refusal to sell off his babies (his stores), and how he lost most of his money doing so. The mystery of it all is why: What blinded such a nimble, flexible decision-maker to the clear signals right in front of him? How could some of the same behavior that helped him thrive (through grit, determination, and stick-to-itiveness) end up causing his failure (through inflexibility, intractability, and maybe even some hubris)?</p> <p><strong>Knee-Deep in the Big Muddy.</strong> The very first sentence of one of the earliest and most influential academic papers identifying our tendency to persist in losing endeavors, even in the face of strong signals that we ought to quit, states very simply why such behavior is so confounding: “Intuitively, one would expect individuals to reverse decisions or to change behaviors which result in negative consequences.” The author of that seminal 1976 paper, “Knee-Deep in the Big Muddy: A Study of Escalating Commitment to a Chosen Course of Action,” is Harold and Shirley’s son, Barry Staw. He saw the US involvement in the Vietnam War as a living, breathing, high-stakes, slow-motion train wreck of an example of our inability to quit. Staw pointed to the revelation from the Pentagon Papers, a secret Department of Defense history of that war published over government objection by The New York Times and The Washington Post, that Undersecretary of State George Ball warned LBJ in 1965 of the inevitable entrapment in the conflict, “Once we suffer large casualties, we will have started a well-nigh irreversible process. Our involvement will be so great that we cannot—without national humiliation—stop short of achieving our complete objectives. Of the two possibilities I think humiliation would be more likely than the achievement of our objectives—even after we have paid terrible costs.” Staw’s central insight about escalation of commitment is that the phenomenon is not confined to matters like the Vietnam War, a complex geopolitical conflict with national pride wrapped up in it. His laboratory and field experiments show that whether it is on the level of an individual, an organization, or a governmental entity, when we’re getting bad news, when we are getting strong signals that we’re losing—signals that others plainly see—we don’t merely refuse to quit. We will double and triple down, making additional decisions to commit more time and money (and other resources) toward the losing cause, and we will strengthen our belief that we are on the right path.</p> <p><strong>Waiting Until It Hurts.</strong> Psychologists Jeffrey Rubin and Joel Brockner conducted an amusing experiment to answer two questions: How long will people wait for something that never arrives, and what price will they pay to continue waiting? It turns out people will wait a surprisingly long amount of time, and they will pay an amount that clearly exceeds the value of what they were waiting for. Escalation of commitment is costly. If the participants had walked away sooner, they would have made more. It may feel like quitting slows us down, but Rubin and Brockner show that it is persistence that is often the culprit. The work on escalation of commitment over the last forty-five years--in different laboratory experiments, field experiments, and explanations of commonly observed behavior--has shown that this type of entrapment in losing causes occurs across a variety of settings and circumstances.</p> <p><strong>Summary:</strong><br> * When we are in the losses, we are not only more likely to stick to a losing course of action, but also to double down. This tendency is called escalation of commitment.<br> * Escalation of commitment is robust and universal, occurring in individuals, organizations, and governmental entities. All of us tend to get stuck in courses of action once started, especially in the face of bad news.<br> * Escalation of commitment doesn’t just occur in high-stakes situations. It also happens when the stakes are low, demonstrating the pervasiveness of the error.</p> <h2>Ch 5: Sunk Cost and the Fear of Waste</h2> <p>2008 approval of bond measure for a high-speed rail system connecting LA and SF. Estimated completion in 2020, at a cost of $33b, generating yearly revenue of $1.3b by 2020, with an operating surplus of $370m, making it self-supporting and profitable thereafter. As of now, no part of that line is operational. Given the accuracy of the Authority’s past projections, there is no basis for believing that the revised estimates of 2029 (for initial service) or 2033 (for completion) are reasonably achievable. This shouldn’t be too surprising since they approved building the first segment in 2010 (twenty-five miles between Madera and Fresno), but they didn’t break ground for another five years. Why is this all taking so long? It turns out there are two titanic engineering obstacles to connecting the central, interior part of the planned route with the key metropolitan areas at either end of the state. First, they have to figure out a way to build track over or blast through the Tehachapi Mountains, which is necessary to connect Bakersfield with Los Angeles to its south. That issue pales in comparison to a second bottleneck, a portion of the Diablo Range known as the Pacheco Pass that stands between the Central Valley and the Bay Area to its north. Given where the project stands now, it’s a good bet that if the decision-makers had known then what they know now about how much the bullet train would cost and how long it would take, it wouldn’t have been approved in the first place. But having started the project, the Authority seems unwilling to quit and cut their losses.</p> <p><strong>The Sunk Cost Effect.</strong> Richard Thaler, in 1980, was the first to point to the sunk cost effect as a general phenomenon, describing it as a systematic cognitive error in which people take into account money, time, effort, or any other resources they have previously sunk into an endeavor when making decisions about whether to continue and spend more. A perfectly rational decision-maker would consider only the future costs and benefits in deciding whether to continue with a course of action. In other words, if continuing on has a positive expected value, a rational actor would persevere. If it has a negative expected value, they would quit. Forty years of experiments and fieldwork across a variety of domains show that people behave as Thaler hypothesized regarding sunk costs. In decisions about whether to move forward, they do take into account what they’ve already spent. They do this because they irrationally think that the only way to recover or justify the costs is if they continue on. <strong><em>Put simply, the sunk cost effect causes people to stick in situations that they ought to be quitting.</em></strong></p> <p>There’s a simple elegance to Kahneman and Tversky’s proposition. You can see the decision error, free of complicating factors like how the participants got into the losing position or whether they realized their expected value was negative. In these experiments and those that followed, the math is made clear and transparent to the participants. <strong><em>This is not an error of calculation. This is an error of cognition.</em></strong> In a simple hypothetical like the concert, you can also see the error pretty clearly. It likely makes sense that a choice of how much you want to see the band versus how much you don’t want to spend hours in the cold and freezing rain shouldn’t depend on whether or how much you paid for the ticket. But this cognitive illusion is very strong. Just because you know it’s an error in theory doesn’t mean that you won’t fall for it when you are facing down these kinds of decisions.</p> <p><strong>When "Public Works" Is an Oxymoron.</strong> The sunk cost error’s fingerprints are all over the California bullet train. If most anyone were asked to start the project today as a new endeavor—knowing about the exploding costs, currently as much as $105 billion but likely to increase significantly, and how hard it will be to blast through those two mountain ranges—it seems obvious that the answer would be a hard no. In addition to the direct costs of the project, there is the issue of opportunity costs. Every dollar that California sinks into the project is a dollar that could otherwise be allocated to something that would create more value and a greater public good for the taxpayers whose money is funding the endeavor. But imagine how gutsy a politician would have to be to abandon the project, knowing they’re going to have to defend themselves against charges of having “wasted” more than $8 billion on a train that was never completed. The pressure to keep going to “recover” those costs is enormous. When it comes to these types of public works projects, sunk cost is a familiar refrain.</p> <p><strong>Katamari.</strong> There was a popular video game that came out in 2004 named <em>Katamari Damacy</em>. It was a silly game, strangely addictive, with a grandiose but simple plot. You control the actions of a tiny prince, whose father, the King of All Cosmos, gives him a katamari (Japanese for “clump”), a sticky ball you roll around different locations, picking up trash and debris off the floor, the ball growing bigger and bigger as it accumulates more stuff. Why are you on this mission? The king had gotten drunk and accidentally destroyed a bunch of stars and constellations. You have to grow the ball until it’s big enough to become a star to replace the ones the king destroyed. It’s a goofy plot, but no goofier than whatever is supposed to be motivating you to eat dots and fruit in Pac-Man or fit together block formations in Tetris. The katamari can’t roll over anything bigger than itself. If it does, the impact knocks some things off, making your ball even smaller. At the beginning, your katamari is only big enough to pick up things like ants, thumbtacks, and buttons. Running into a mouse can be a catastrophe. But as you successfully pick up debris, the ball gets bigger. Then you’re terrorizing the mouse. You’re rolling over batteries, plates of food, radios, shoes, pets. Cows, bears, sumo wrestlers, cars, monsters, buildings, islands, mountains. As one reviewer put it after listing some of the mundane little items, “25 minutes later the bloody thing is ripping rainbows out of the ground.” <strong><em>Like the katamari, rolling around collecting debris, which makes it grow in mass and collect more and even bigger debris, there is a self-reinforcing aspect to the sunk cost fallacy that we really need to watch out for.</em></strong></p> <p>A relationship that’s not working out turns into a game of Katamari. Your friend complains about being in a bad relationship. If you ask, “Why don’t you just break up?” they’ll frequently say, “Because I’ve put so much time into trying to make this relationship work.” Sometimes, they’ll even say, “I put my heart and soul into it.” The more time they put in, the less likely they are to break it off, which leads to them investing more time to get it to work. That makes them even less likely to break up. And so on. No wonder that once you have this talk with a friend, you end up having it over and over again. Their dysfunctional relationship keeps rolling up mass—living arrangements, friends, pets, consumer purchases, property—until they’re ripping rainbows out of the ground.</p> <p><strong>How Big Does the Katamari Grow?</strong> In Barry Staw’s classic 1976 experiment, “Knee-Deep in the Big Muddy: A Study of Escalating Commitment to a Chosen Course of Action,” he set out to ask how much prior commitment to a course of action influences future decisions about whether to stick or quit. He discovered that the answer is a lot. Staw recruited groups of business school students who were tasked with individually deciding how a corporation should allocate certain R&amp;D funds between two of its divisions. To help with the choice, the students were given ten years of historical financial performance about the company and the two divisions under consideration. The participants each had to make an all-or-nothing decision about which department should receive the $10 million in R&amp;D funds, meaning they only had two options: allocate the entire $10 million to one division and give no funds to the other or vice versa. Given the data provided to the students, there were reasonable arguments for allocating the funds to either of the two divisions. While one division was more profitable, the other was growing more quickly. Indeed, the participants, who were all coming into this decision fresh, split about 50-50 on which division they chose. What Staw wanted to find out was whether this initial decision about which division got the funds would affect future allocation decisions, particularly once the participants learned their first choice turned out poorly. In other words, Staw was exploring whether people who carry a loss into a new allocation decision would be more likely to persist, continuing to direct funds to the same division they had previously sunk resources into. To answer that question, he gave all the students a simulation of the next five years of financial results for the company. Regardless of which option they directed the funds to, the additional data showed that the division they chose suffered from half a decade of stagnant sales and deepening losses, significantly underperforming the division they passed over. After being shown this performance data, the participants were then given a new budget of $20 million, which they could now allocate <em>proportionally</em> across the two divisions. Staw hypothesized that students receiving negative feedback on their first decision would <em>increase</em> their commitment to their original cause, favoring that same division in the second allocation. Indeed, that is what he found. The participants allocated, on average, over $13 million of the $20 million to the division that they originally directed the R&amp;D funds to, and just under $7 million to the one they did not initially choose. To really hit the point home, he also had a separate group of participants come to the $20 million allocation fresh, as a new decision. They were shown the same financial results and informed of a $10 million R&amp;D allocation five years earlier (by a since-departed financial officer) to the division that subsequently underperformed. In this case, they were not the ones who initially made the decision to sink the R&amp;D funds into one division or the other. When these fresh participants allocated the $20 million proportionally across the two divisions, they gave an average of just $9 million to the division that received the prior funds, much less than the $13 million allocation by the participants carrying with them the sunk cost debris of having personally incurred the prior losses. Putting it into sharp focus, the participants responsible for the first, money-losing decision <em>directed nearly 50%</em> more of the $20 million to that same division, compared with others with the identical information and corporate history but no personal responsibility for prior policy.</p> <p><strong>Mental Accounting.</strong> There’s a saying among top poker players that poker is one long game. It’s a reminder that the particular hand they’re playing is not the last hand they’ll ever play or that any particular day that they’re playing is not the last day they’ll ever play. A poker player will play thousands upon thousands of hands over their lifetime, so in the grand scheme of things whether or not they lose one single hand of poker matters very little. What matters is that they’re maximizing their expected value over all those days and all those hands. That’s what they mean by one long game. This mantra is meant to help expert players overcome the sunk cost fallacy, expressed in poker as wanting to protect the money you’ve already invested in a single hand by not folding, or not wanting to quit a game when you’re in the losses. Of course, what applies to poker applies to life as well. <strong><em>We all need this kind of reminder because of a quirk in our mental accounting.</em></strong></p> <p>When we start something, whether it’s putting money into the pot in a hand of poker, or starting a relationship or a job, or buying a stock, we open up a mental account. When we exit that thing, whether it’s folding a hand, or leaving a relationship or job, or selling the stock, we close that mental account. It turns out that we just don’t like to close mental accounts in the losses. If we’re losing in a hand of poker, we don’t want to fold because that means we have to realize the loss of the money we put in the pot. If we’re losing in a poker game, we don’t want to quit because it means that we have to leave with less money than we started with. If we’re in a relationship or a job, we don’t want to walk away because we’ll feel like we will have wasted or lost all the time and effort that we put in. Of course, that’s irrational. What really matters is maximizing your expected value across all the things you start, across all of your mental accounts. If you’re investing in a number of stocks, some are going to win and some are going to lose. What matters is whether you’re winning across your whole portfolio not whether any one investment is up or down. But that’s not how we naturally think. We don’t think about the whole portfolio of stocks we own. Each is associated with its own mental account that we don’t want to close out unless we are in the gains.</p> <p><strong>The Hardest Cost to Bear.</strong> <strong><em>The greater the sunk costs, the harder it becomes to quit.</em></strong> And the greatest cost is, of course, the loss of human life. That makes decisions about whether or when to exit a war heartbreakingly difficult. Retired four-star general Tony Thomas, commander of U.S. Special Operations Command (USSOCOM), served in Afghanistan on missions between 2001 and 2013 (except for a year when he served in the Iraq War). He attended many military funerals and gave an American flag to many gold star families. He described to me those humbling experiences and how those tragic losses amplify the types of sunk cost problems we all face, making it particularly difficult for a country to extricate itself from a war once it has started to incur those losses. On one occasion, a gold star mother, having just lost her son, gripped his hand and said, “Stay on this and finish it.” The general’s knees almost buckled. At that moment, he wanted to run through a wall for her. The unspoken message, never expressed at these funerals but which he felt was on the minds of all those grieving parents, was, “Tell me my child didn’t die in vain.”</p> <p><strong>The Difference between Knowing and Doing.</strong> <em>There are lots of intuitions people have about cognitive biases, including the sunk cost fallacy. One of the most common is that if you are educated on the topic and know about it, that will stop you from committing the error.</em> Jebbrey Rubin climbed 99 of the 100 highest peaks in New England, and was climbing the last one, Fort Mountain, in Maine. When the weather turned bad and a fog came in, his climbing partner decided to turn around. Jeffrey R. disagreed and continued climbing alone. His body was found several days later. He apparently fell to his death. Why am I telling you this story, so similar to some of the others? One person turns around. The other continues to go on, with tragic consequences. The Jeffrey R. in this story is Jeffrey Rubin, the same Jeffrey Rubin who, with Joel Brockner, studied the behavior of people waiting for a crossword puzzle dictionary that never came, and followed it up with an impressive and influential body of work on escalation of commitment right up to his death in 1995. If anybody understood the problem of being entrapped in a course of action, unable to cut your losses even in the face of clear signals that you ought to be quitting, it would have been him. Yet he became entrapped that day. This should be a warning to all of us. Don’t think that, just because you’ve read up to this point in the book or understand the sunk cost fallacy, this knowledge alone is going to help you overcome it. If Rubin was unable to quit, that should open our eyes to how hard it is for the rest of us. <strong><em>Knowing is not the same as doing.</em></strong></p> <p><strong>You Can't Jedi Mind Trick Being Fresh to a Decision.</strong> A lot of people who know about the sunk cost fallacy tell me they’ve come up with a solution. Essentially, regardless of the history they have with the decision, they ask themselves, “If I were approaching this decision fresh, would I want to enter into this course of action?” Does this Jedi mind trick actually work? We can, once again, look to Barry Staw for the answer. In one of the follow-ups to the “Big Muddy” experiment, Itamar Simonson and Staw asked participants to make a corporate decision about allocating marketing funds to two products, a nonalcoholic beer and a lite beer. The first decision was, again, an all-or-nothing choice of which product should receive an additional $3 million in marketing support. After making their choice and receiving a simulation of three years of results based on that choice, the participants made a second decision as to how to split another $10 million marketing budget between the two products. The investigators tested several possible ways of mitigating escalation of commitment to the product that received the initial $3 million in marketing funds. One of those ways was the Jedi mind trick, where they asked some of the participants to approach the decision fresh, specifically instructing them to do an analysis listing the pros and cons of allocating funds to each product going forward. Despite the instruction to look forward rather than backward when making this new decision, the participants made a similar allocation ($5.1 million) to the product they earmarked the original funds to as compared to those who made the same prior, losing decision but were not given instruction to just look forward. In contrast, participants who actually came to the second allocation fresh gave just $3.7 million to the product that lost money after receiving the earlier addition of marketing funds. The instruction to treat it as a new decision did practically nothing to reduce escalation of commitment. Just knowing about the sunk cost effect doesn’t help. <strong><em>The Jedi mind trick doesn’t help.</em></strong></p> <p><strong>Summary:</strong></p> <ul> <li>The sunk cost effect is a cognitive illusion where people take into account resources they have previously sunk into an endeavor when making decisions about whether to continue and spend more.</li> <li>The sunk cost effect causes people to stick in situations that they ought to be quitting.</li> <li>When deciding whether to stick or quit, we are worried that if we walk away, we will have wasted the resources we have spent in the trying.</li> <li>You might be experiencing the sunk cost fallacy if you hear yourself thinking “If I don’t make this work I will have wasted years of my life!” or “We can’t fire her now, she’s been here for decades!”</li> <li>Sunk costs snowball, like a katamari. The resources you have already spent make it less likely you will quit, which makes it more likely you will accumulate additional sunk costs, which makes it again less likely you will quit, and so on. The growing debris of your prior commitment makes it increasingly harder to walk away.</li> <li>We don’t like to close mental accounts in the losses.</li> <li>Knowing about the sunk cost effect doesn’t keep you from falling prey to it.</li> <li>You can’t trick yourself into not taking sunk costs into account by trying to view the situation as a new choice. Asking whether or not you would continue if the decision were a fresh one doesn’t mitigate the sunk cost effect the way you might intuitively think it would.</li> </ul> <h2>Ch 6: Monkeys and Pedestals</h2> <p>Eric Teller is CEO of "X", an innovation labs subsidiary of Alphabet, though his actual job title is "Captain of Moonshots." X has become a famous incubator and developer of ideas in the nontraditional tradition of Bell Labs, Xerox PARC, and Thomas Edison’s laboratories. X’s mission is to build and launch technologies to “improve the lives of millions, even billions of people.” They’re specifically in the business of identifying and accelerating world-changing ideas. That means they reject plenty of good ideas because the change those ideas create would be too incremental for their mission. One of X’s slogans is “10x impact on the world’s most intractable problems, not just 10% improvement.” X takes a lot of big swings, knowing that most will be whiffs. Teller looks at each project as buying an option on the future. Like most options, you have to keep paying to hold it, in increasing amounts. To help X-ers become better quitters, Astro Teller has come up with a unique mental model that has been woven into the fabric of X: monkeys and pedestals.</p> <p>Imagine that you’re trying to train a monkey to juggle flaming torches while standing on a pedestal in a public park. If you can achieve such an impressive spectacle, you’ve got a moneymaking act on your hands. Teller recognizes that there are two pieces to becoming successful at this endeavor: training the monkey and building the pedestal. One piece of the puzzle presents a possibly intractable obstacle in the way of success. And the other is building the pedestal. People have been building pedestals since ancient Greece and probably before. Over two-plus millennia, pedestals have been thoroughly figured out. You can buy one at a furniture store or a hardware store, or turn a milk crate upside down. The bottleneck, the hard thing, is training a monkey to juggle flaming torches. The point of this mental model is to remind you that there is no point building the pedestal if you can’t train the monkey. <strong><em>In other words, you ought to tackle the hardest part of the problem first.</em></strong></p> <p><strong>Getting the Monkey Off Your Back.</strong> Project Foghorn, X’s initiative to develop technology to convert seawater into fuel, offers an example of how the monkeys-and-pedestals mental model works. The first monkey would be proof of concept, but they already had that from the scientists they were partnering with whose recent work attracted their attention to the innovation. The second monkey was commercial viability. They would have to produce the fuel at a cost that was significantly lower than the current price per gallon for gasoline to get broad adoption in the market.</p> <p>One of the beautiful things about the monkeys-and-pedestals mental model is that sometimes it helps you quit before you start. Years ago, X looked into developing what’s now known as a hyperloop, an experimental high-speed rail system. The concept was fine. Building the physical infrastructure wouldn’t be very hard from an engineering standpoint. The monkeys for the hyperloop to be viable were things like whether you could safely load and unload passengers or cargo, and whether you could get the system up to speed and get it to brake without incident. A couple hundred yards of track wouldn’t tell you anything about whether you could conquer those challenges. In fact, Teller and the team at X figured out that you would have to build practically the whole thing before you knew whether it worked. You would have to build a bunch of pedestals before you could find out if the monkeys were intractable. They quickly decided not to pursue it.</p> <p><strong><em>One of Teller’s valuable insights is that pedestal-building creates the illusion of progress rather than actual progress itself.</em></strong> When you are doing something that you already know you can accomplish, you’re not learning anything important about whether the endeavor is worth pursuing. You already know you can build the pedestal. The problem is whether you can train the monkey.</p> <p>On top of that, Teller realizes that <strong><em>when you’re building pedestals, you are also accumulating sunk costs that make it hard to quit even as you find out that you may not be able to train the monkey to juggle those torches</em></strong>. By focusing on the monkey first, you naturally reduce the debris you accumulate solving for something that’s, in reality, already solved.</p> <p>Teller also understands a subtler but no less important point, that <strong><em>we have a tendency, when we butt up against a monkey that is proving difficult to solve, to turn our attention to building pedestals rather than giving up</em></strong>. We prefer that illusion of progress to having to quit and admit defeat.</p> <blockquote> <p>Figure out the hard thing first. Try to solve that as quickly as possible. Beware of false progress.</p> </blockquote> <p><strong>Kill Criteria.</strong> If we can identify in advance what the signals are that we should pay attention to and make a plan for how we will react to them, we can increase the chances that we’ll cut our losses when we ought to. Ask yourself, “What are the signs that, if I see them in the future, will cause me to exit the road I’m on? What could I learn about the state of the world or the state of myself that would change my commitment to this decision?” That list offers you a set of <em>kill criteria</em>, literally criteria for killing a project or changing your mind or cutting your losses. It’s one of the best tools for helping you figure out when to quit closer to on time. <em>(Stopping rules!)</em> Essentially, kill criteria create a precommitment contract to quit.</p> <p><strong>Funnel Vision.</strong> You can likely imagine lots of applications of kill criteria in your personal life. When you start dating someone, think ahead. What could be happening that would make you think that it was time to end the relationship? Or, in the case of a single date, what would make you want to end the date? You could do that with going to a particular college, picking a major, starting a career, or taking a job. An obvious and high-value application of kill criteria has to do with funnel management for a business’s sales function. A big problem for sellers is managing all the opportunities at the top of the funnel: Which do you pursue? And, once you’ve started pursuing a lead, when do you give up on it? It’s in a company’s interest to make sure its sellers are spending their time pursuing the highest-value opportunities, based on a combination of the probability of closing and the potential size of the contract.</p> <p>(Story about creating kill criteria to help avoid spending sales time/resources on low-expected-value leads.) We tend to associate the idea of funnel management with sellers or investors. But every one of us has a funnel we are managing: the interests we can pursue, the classes we can take, the projects we can do at work, the jobs we can apply for, the people we can date. We all have to make these choices about which opportunities to pursue and which to skip or quit. As we’re making those choices, we want to spend as little time as possible on the things that aren’t worthwhile and as much time as possible on the things that are.</p> <p>The good news about kill criteria is that you haven’t missed your chance to set them once you have already started an endeavor. At any point, no matter whether it comes to someone you are dating or a house you already own or an investment you are in or a college you are attending, you can think about some time frame in the future, imagine you are unhappy with your situation, and identify the benchmarks you will have missed or the signals you will be seeing that will tell you that you ought to walk away. You may not have set a stop-loss or take-gain when you bought a stock but you can set one now. After all, the present is always in advance of something.</p> <p><strong>States and Dates.</strong> The best quitting criteria combine two things: a state and a date. A state is just what it sounds like, an objective, measurable condition you or your project is in, a benchmark that you have hit or missed. A date is the when. Kill criteria, generally, include both states and dates, in the form of “If I am (or am not) in a particular state at a particular date or at a particular time, then I have to quit.” Or “If I haven’t done X by Y (time), I’ll quit.” Or “If I haven’t achieved X by the time I’ve spent Y (amount in money, effort, time, or other resources), I should quit.”</p> <p>Admiral McRaven offered a unique, high-stakes application of this concept of states and dates when describing the planning for Operation Neptune Spear, the raid on Osama bin Laden. The operation was broken down into 162 phases. Each phase told you what state you would have to achieve to continue, and what state you might be in that would cause you to quit during that phase. Because this was all planned out in advance, it left McRaven, as he told me, with only about five command decisions he might have to make on the fly once the mission had commenced and they were already in it. He gave two examples of the criteria that would cause them to kill the mission. If at any point they fell an hour behind schedule, they would abort. Or, if they discovered that, at any time up to 50% of the way to bin Laden’s compound, they had been detected and compromised by the Pakistani government, they would turn around. If they were compromised beyond the 50% mark, that would be a command decision McRaven would have to make on the fly.</p> <p>You can apply states and dates to relationships. If your goal is marriage (or an equivalent long-term commitment), then if your relationship partner hasn’t proposed (or accepted your proposal or otherwise demonstrated a long-term commitment) by a certain date, you should move on and find someone who is as excited about committing to you as you are to them.</p> <p>You can do the same for career advancement. If you’re working at an entry-level position that has some prospect for advancement, figure out as early as you can the interim milestones for those who succeed, whether it’s raises, or initial promotions, or additional responsibilities, or whatever is specific in that company or practice. Get information about when others who’ve succeeded got those signals on the way up and include those states and dates in your kill criteria.</p> <p><strong>Better, Not Perfect.</strong> When I was playing poker, I applied a bunch of kill criteria that helped me be a better quitter (of hands and of games). One example was a stop-loss. If I lost a certain amount, I would quit. This was especially important at the start of my career, because novice players are particularly poor at judging whether they’re losing due to their poor play or because of bad luck. (Take-gains don’t make sense in poker, so I didn’t employ that tool.) After turning pro, I still maintained a stop-loss. Elite poker players are still going to be worse at making quitting decisions when they’re in it, especially if they’re in it and losing. So, even after I gained experience and got a better understanding of the quality of my play and the short-term swings of luck, I still set loss limits. I also realized that I played better in sessions of six to eight hours or less, so I committed to quitting after I played that long. Because I was more aware of the importance of game conditions, I also committed to quit if the quality of the players in the game drastically changed in an unfavorable way as some players cashed out and new ones took their seats. Those kill criteria helped me to become better at quitting games. But was I perfect? Not even close.</p> <p>Taken together, the monkeys-and-pedestals mental model and kill criteria help us overcome our aversion to closing accounts in the losses. First, they both get you to no faster, which naturally limits the losses that you have to absorb when you quit. And the less you are down, the easier it is to walk away. Second, when you set out clear kill criteria in advance and make a precommitment to walk away when you see those signals, you are just more likely to follow through, even when you are losing. Anytime you can make a decision about cutting your losses in advance, you’ll do better at closing those mental accounts.</p> <p><strong>Summary:</strong></p> <ul> <li>Monkeys and pedestals is a mental model that helps you quit sooner.</li> <li>Pedestals are the part of the problem you know you can already solve, like designing the perfect business card or logo. The hardest thing is training the monkey.</li> <li>When faced with a complex, ambitious goal, (a) identify the hard thing first; (b) try to solve for that as quickly as possible; and (c) beware of false progress.</li> <li>Building pedestals creates the illusion that you are making progress toward your goal, but doing the easy stuff is a waste of time if the hard stuff is actually impossible.</li> <li>Tackling the monkey first gets you to no faster, limiting the time, effort, and money you sink into a project, making it easier to walk away.</li> <li>When we butt up against a hard problem we can’t solve, we have a tendency to turn to pedestal-building rather than choosing to quit.</li> <li>Advance planning and precommitment contracts increase the chances you will quit sooner.</li> <li>When you enter into a course of action, create a set of kill criteria. This is a list of signals you might see in the future that would tell you it’s time to quit.</li> <li>Kill criteria will help inoculate you against bad decision-making when you’re “in it” by limiting the number of decisions you’ll have to make once you’re already in the gains or in the losses.</li> <li>In organizations, kill criteria allow people a different way to get rewarded beyond dogged and blind pursuit of a project until the bitter end.</li> <li>A common, simple way to develop kill criteria is with “states and dates:” “If by (date), I have/haven’t (reached a particular state), I’ll quit.”</li> </ul> <h1>Interlude II: Gold or Nothing</h1> <p>Sasha Cohen's figure-skating career and the idea that quitting on something is quitting on ourselves, as in, our identity of ourselves.</p> <h1>Section III: Identity and Other Impediments</h1> <h2>Ch 7: You Own What You've Bought and What You've Thought: Endowment and Status Quo Bias</h2> <p><strong>Summary:</strong></p> <ul> <li>The endowment effect is a cognitive bias where we value something we own more than we would if we didn’t own it.</li> <li>We can be endowed to objects but also to our own ideas and beliefs.</li> <li>Endowment is an obstacle to quitting because when we irrationally value things we own, we miscalculate their expected value. We might think the company we started or the project we devised or the belief we have is worth more than it actually is.</li> <li>We prefer to stick with the status quo.</li> <li>We are more tolerant of bad outcomes that come from sticking with what we are already doing than bad outcomes that come from switching to something new. This phenomenon is part of omission-commission bias.</li> <li>When you say, “I’m just not ready to decide yet,” what you are really saying is, “For now, I am choosing the status quo.”</li> <li>Even in highly data-rich environments like professional sports, sunk cost, endowment, and status quo bias distort decision-making.</li> </ul> <h2>Ch 8: The Hardest Thing to Quit is Who You Are: Identity and Dissonance</h2> <p><strong>Summary:</strong></p> <ul> <li>When it comes to quitting, the most painful thing to quit is who you are. Our ideas, beliefs, and actions are part of our identity.</li> <li>When new information conflicts with a belief, we experience cognitive dissonance.</li> <li>To resolve the conflict, we can either change the belief or rationalize away the new information. Too often, we choose the latter.</li> <li>Dissonance can also result from new information coming into conflict with our past actions.</li> <li>We have a desire to maintain internal consistency, where our past beliefs and actions line up with our present beliefs and actions.</li> <li>We also want others to view us as consistent. We worry that if others see inconsistency between our present and past decisions, beliefs, or actions, they will judge us as being wrong, irrational, capricious, and prone to mistakes.</li> <li>When we know or believe our decisions are being evaluated by others, our intuition is that we will be more rational, but the opposite is true. External validity <em>increases</em> escalation of commitment.</li> <li>The more extreme a position is, the more cognitive gymnastics we’ll do to defend it. The facts are more likely to persuade you away from the consensus opinion than a fringe view.</li> <li>Fears about how others will view us if we quit are usually overblown.</li> </ul> <h2>Ch 9: Find Someone Who Loves You but Doesn't Care About Hurt Feelings</h2> <p><strong>Summary:</strong></p> <ul> <li>Optimism makes you less likely to walk away while not actually increasing your chances of success. That means that being overly optimistic will make you stick to things longer that aren’t worthwhile. Better to be well calibrated.</li> <li>Life’s too short to spend your time on opportunities that are no longer worthwhile.</li> <li>When someone is on the outside looking in, they can usually see your situation more rationally than you can.</li> <li>The best quitting coach is a person who loves you enough to look out for your long-term well-being. They are willing to tell you the hard truth even if it means risking hurt feelings in the short term.</li> <li>Decisions about when to quit improve when the people who make the decisions to start things are different from the people who make the decisions to stop those things.</li> <li>Getting the most out of a quitting coach requires permission to speak the truth.</li> </ul> <h1>Interlude III: The Ants Go Marching... Mostly</h1> <h1>Section IV: Opportunity Cost</h1> <h2>Ch 10: Lessons from Forced Quitting</h2> <p><strong>Summary:</strong></p> <ul> <li>Being forced to quit forces you to start exploring new options and opportunities. But you should start exploring before you’re forced to.</li> <li>Even after you have found a path that you want to stick to, keep doing some exploration. Things change, and whatever you are doing now may not be the best path for you to pursue in the future. Having more options gives you something to switch to when the time is right.</li> <li>Exploration helps you to diversify your portfolio of skills, interests, and opportunities.</li> <li>A diversified portfolio helps to protect you against uncertainty.</li> <li>Backup plans are good to have especially because some backup plans can turn out to be better than what we’re already pursuing.</li> </ul> <h2>Ch 11: The Myopia of Goals</h2> <p><strong>Summary:</strong></p> <ul> <li>Goals can make it possible to achieve worthwhile things, but goals can also increase the chances that we will escalate commitment when we should quit.</li> <li>Goals are pass-fail in nature. You either reach the finish line or you don’t, and progress along the way matters very little.</li> <li>Don’t just measure whether you hit the goal, ask what you have achieved and learned along the way.</li> <li>Set intermediate goals and prioritize goals that allow you to recognize progress along the way or acquire something valuable even if you don’t reach the goal.</li> <li>Goals, when set, are a proxy for an expected-value equation, balancing the benefits that you’re trying to gain against the costs you’re willing to bear.</li> <li>Inflexible goals aren’t a good fit for a flexible world.</li> <li>With better advance planning (like identifying monkeys and pedestals and kill criteria) and the help of a good quitting coach, you can make goals more flexible, setting at least one “unless” and planning regular check-ins on the analysis that initially led to setting the goal.</li> <li>In general, when we quit, we fear two things: that we’ve failed and that we’ve wasted our time, effort, or money.</li> <li>Waste is a forward-looking problem, not a backward-looking one.</li> </ul> https://tedneward.github.io/Research/thinking/subtle-art-not-giving-a-fck/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/subtle-art-not-giving-a-fck/index.html <p><em>(by Mark Manson)</em></p> <h3>Chapter 1: Don't Try</h3> <ul> <li>Conventional life advice, or all the positive stuff and happy self-help stuff we hear all the time, fixates on what you lack.</li> <li>If you're dreaming of something all the time, then you're reinforcing the same unconscious reality over and over: that you are <em>not that</em>.</li> </ul> <h4>The Feedback Loop from Hell</h4> <ul> <li>Consumer culture and social media has bred a whole generation who believe that having negative experiences like anxiety, fear, guilt, etc is not okay.</li> <li>The desire for more positive experience is itself a negative experience, while the accepting of one's negative experience is itself a positive experience.</li> <li>Everything worthwhile in life is won through surmounting the associated negative experience, so don't try to escape it.</li> <li>You must learn how to focus and prioritize your thoughts effectively, or how to pick and choose what matters to you and what doesn't based on finely honed personal values.</li> </ul> <h4>The Subtle Art of Not Giving a Fuck</h4> <ul> <li>Subtlety #1: Not giving a fuck doesn't mean being indifferent; it means being comfortable with being different.</li> <li>Indifferent people are afraid of the world and the repercussions of their own choices, which is why they don't make any meaningful choices.</li> <li>You must say "fuck it" not to everything in life, but rather to everything unimportant in life. Save your fucks for what truly matters.</li> <li>Subtlety #2: To not give a fuck about adversity, you must first give a fuck about something more important than adversity.</li> <li>If you don't find that meaningful something, your fucks will be given to meaningless and frivolous causes.</li> <li>Subtlety #3: Whether you realize it or not, you are always choosing what to give a fuck about.</li> <li>Maturity is what happens when we learn to give fucks about only what's truly fuckworthy.</li> </ul> <h4>So Mark, What the Fuck Is the Point of This Book Anyway?</h4> <ul> <li>Practical enlightenment is becoming comfortable with the idea that some suffering is inevitable, and that no matter what you do, life is comprised of failures, loss, and even death.</li> <li>The only way to overcome pain is to first learn how to bear it.</li> </ul> <h3>Chapter 2: Happiness Is a Problem</h3> <ul> <li>Pain and loss are inevitable and we should let go of trying to resist them.</li> <li>Happiness is not a solvable equation. Dissatisfaction and unease are inherent parts of human nature, and necessary components to creating constant happiness.</li> </ul> <h4>The Misadventures of Disappointment Panda</h4> <ul> <li>We suffer because it is biologically useful: It is nature's preferred agent for inspiring change.</li> <li>A constant dissatisfaction has kept our species fighting and striving, building and conquering.</li> <li>A society that coddles itself more and more from the discomforts of life is dangerous: We lose the benefits of experiencing healthy doses of pain, which disconnects us from the reality of the world around us.</li> </ul> <h4>Happiness Comes from Solving Problems</h4> <ul> <li>True happiness occurs when you find the problems you enjoy having and enjoy solving.</li> <li>Instead of solving, people either a) deny they exist in the first place, or b) choose to believe that there is nothing they can do to solve their problems, even when there is.</li> <li>Forms of blame and denial allow us to temporarily escape our problems, and that escape can provide us a quick rush to feel better.</li> </ul> <h4>Emotions are Overrated</h4> <ul> <li>Negative emotions are a call to action. When you feel them, it's because you're supposed to <em>do something</em>.</li> <li>To deny one's negative emotions is to deny many of the feedback mechanisms that help a person solve problems.</li> </ul> <h4>Choose Your Struggle</h4> <ul> <li>An interesting question that most people will never consider is "What <em>pain</em> do you want in your life? What are you willing to struggle for?"</li> <li>Real, serious, lifelong fulfillment and meaning have to be earned through the choosing and managing of our struggles.</li> <li>What determines your success isn't "What do you want to enjoy?" but "What pain do you want to sustain?" Our struggles determine our success.</li> </ul> <h3>Chapter 3: You Are Not Special</h3> <ul> <li>A true and accurate measurement of one's self-worth is how you feel about the negative aspects of yourself.</li> <li>Entitled people have a delusional degree of self-confidence.</li> <li>Because entitled people are incapable of acknowledging their problems openly and honestly, they are incapable of improving their lives in any lasting or meaningful way.</li> </ul> <h4>Things Fall Apart</h4> <ul> <li>If we have problems that are unsolvable, our unconscious figures that we're uniquely special or uniquely defective in some way. Put simply: we become entitled.</li> <li>Entitled people flip between being on top of the world or having the world on top of them, demanding special treatment in each case.</li> <li>It takes just as much energy and delusional self-aggrandizement to maintain the belief that one has insurmountable problems as that one has no problems at all.</li> <li>In an age when we're more connected than ever, entitlement seems to be at an all-time high.</li> </ul> <h4>The Tyranny of Exceptionalism</h4> <ul> <li>Our lives today are filled with information from the extremes of the bell curve of human experience, because in the media business that what sells.</li> <li>This flood of extreme information has conditioned us to believe that exceptionalism is the new normal. Because we are mostly average, this drives us to feel insecure and desperate.</li> </ul> <h4>B-b-b-but If I'm Not Going to Be Special or Extraordinary, What's the Point?</h4> <ul> <li>When a culture's standard of success is "be extraordinary," it's better to be at the low end of the bell curve than in the middle, because there you're still special and deserve attention.</li> <li>People who become great at something become great because they understand that they're only mediocre, and that they could be so much better. It's anti-entitlement.</li> </ul> <h3>Chapter 4: The Value of Suffering</h3> <ul> <li>If suffering is inevitable and our problems are unavoidable, then we should not ask "How do I stop suffering?" but instead "Why am I suffering –&nbsp;for what purpose?"</li> </ul> <h4>The Self-Awareness Onion</h4> <ul> <li>The first layer of self-awareness is a simple understanding of one's emotions.</li> <li>The second layer is the ability to ask why we feel some way; these questions are important because they illuminate what we consider success or failure.</li> <li>The third layer integrates our personal values; this is important because our values determine the nature of our problems, which in turn determines the quality of our lives.</li> <li>We must ask these questions accurately to achieve a deeper knowledge of our own values.</li> <li>What is objectively true about your situation is not as important as how you come to see the situation, and how you choose to measure it and value it.</li> <li>Problems may be inevitable, but their <em>meaning</em> is not. We can control our problems mean based on how we choose to think about them, the standard by which we measure them.</li> </ul> <h4>Rock Star Problems</h4> <ul> <li>If you want to change how you see your problems, you have to change what you value and how you measure failure or success.</li> </ul> <h4>Shitty Values</h4> <ul> <li>Shitty values include:</li> <li>Pleasure: People who focus their energy on superficial pleasures end up more anxious, more emotionally unstable, and more depressed.</li> <li>Material Success: Once one is able to provide for basic physical needs (food, shelter, etc), the correlation between happiness and worldly success approaches zero.</li> <li>Always Being Right: Assuming you're ignorant keeps you unattached to superstitions or poorly informed beliefs and promotes a constant state of learning and growth.</li> <li>Staying Positive: Constant positivity is an avoidance of life's problems&nbsp;– but upon choosing the right values and metrics, these problems should motivate you.</li> <li>The trick with negative emotions is to 1) express them in a socially acceptable and healthy manner, and 2) express them in a way that aligns with your values.</li> </ul> <h4>Defining Good and Bad Values</h4> <ul> <li>Good values are reality-based, socially-constructive, and immediate and controllable; while bad values are superstitious, socially destructive, and not immediate or controllable.</li> <li>Good values are achieved internally, and thereby controllable so that you engage the world as it is rather than by how you wish it were.</li> <li>Bad values are generally reliant on external events, and thereby outside your control so that you must rely on socially destructive or superstitious means to achieve them.</li> <li>Values are about prioritization: What values do you prioritize above all else, and therefore influence your decision-making above all else?</li> <li>By choosing better values, you divert your fucks to something better –&nbsp;to things that matter, improve our state of well-being and generate happiness, pleasure, and success as side-effects.</li> <li>When you give better fucks, you get better problems. And when you get better problems, you get a better life.</li> </ul> <h3>Chapter 5: You Are Always Choosing</h3> <ul> <li>When we feel like we are choosing problems, we feel empowered. When we feel like our problems are being forced upon us, we feel victimized and miserable.</li> </ul> <h4>The Choice</h4> <ul> <li>We don't always control what happens to us. But we <em>always</em> control how we interpret what happens to us, as well as how we respond.</li> <li>Whether we like it or not, we are <em>always</em> taking an active role in what's occurring around or within us. We are <em>always</em> choosing, whether we recognize it or not.</li> </ul> <h4>The Responsibility/Fault Fallacy</h4> <ul> <li>The more we choose to accept responsibility into our lives, the more power we will exercise over our lives. Accepting responsibility for our problems is the first step to solving them.</li> <li>We hesitate to take responsibility for our problems because we believe that to be <em>responsible</em> for our problems is to also be at <em>fault</em> for our problems.</li> <li>But we are responsible for experiences that aren't our fault all the time.</li> <li>Fault results from choices that have already been made. Responsibility results from the choices you're currently making, every second of every day.</li> <li>Responsibility means you get to choose how you see things, how you react to things, and how you value things. You choose the metric by which to measure your experiences.</li> <li>Taking responsibility for our problems is far more important than taking responsibility for success and happiness, because that's where the real learning comes from.</li> </ul> <h4>Responding to Tragedy</h4> <ul> <li>Pain of one sort or another is inevitable for all of us, but again, we get to choose what it means to and for us.</li> </ul> <h4>Genetics and the Hand We're Dealt</h4> <ul> <li>People who consistently make the best choices in the situations they're given are the ones who eventually come out ahead in life.</li> </ul> <h4>Victimhood Chic</h4> <ul> <li>Public sharing of "injustices" garners attention and emotional outpouring, rewarding people who are able to perpetually feel victimized with ever-growing amounts of attention and sympathy.</li> <li>The biggest problem with victimhood chic is that it diverts attention away from <em>actual</em> victims.</li> <li>But part of living in a democracy and a free society is that we all have to deal with views and people we don't necessarily like.</li> </ul> <h4>There Is No "How"</h4> <ul> <li>You are <em>already choosing</em>, in every moment of every day, what to give a fuck about; so change is as simple as choosing to give a fuck about something else.</li> </ul> <h3>Chapter 6: You're Wrong About Everything (But So Am I)</h3> <ul> <li>Growth is endlessly iterative. We shouldn't seek to find the ultimate right answer for ourselves, but instead slowly chip away at the ways that we are wrong.</li> <li>There is no correct dogma or perfect ideology, but only what your experience has shown to be right for you.</li> <li>Instead of looking how we're right all the time, we should be looking for how we are wrong. Being wrong opens us up to change, and in turn brings us opportunity for growth.</li> <li>Don't trust your conception of positive/negative experiences. All we know for certain is what hurts in the moment and what doesn't.</li> </ul> <h4>Architects of Our Own Beliefs</h4> <ul> <li>What we understand as "meaning" is generated by the associations our brain makes between two or more experiences.</li> <li>But there are two problems: The brain is imperfect, and once we create meaning for ourselves, our brains are designed to hold onto that meaning.</li> <li>Many of our values are products of events that are not representative of the world at large, or of a totally misconceived past. So most of our beliefs are wrong.</li> </ul> <h4>The Dangers of Pure Certainty</h4> <ul> <li>For individuals to feel justified in doing horrible things to others, they must feel an unwavering uncertainty in their own righteousness, beliefs, and deservedness.</li> <li>The more you embrace being uncertain and not knowing, the more comfortable you will feel in knowing that you don't know.</li> <li>Uncertainty is the root of all progress and growth, as the person who believes he knows everything learns nothing.</li> <li>Before we can look at our values and priorities and change them into better, healthier ones, we must first become <em>uncertain</em> of our current values.</li> </ul> <h4>Manson's Law of Avoidance</h4> <ul> <li>Manson's Law of avoidance states the more something threatens your identity, the more you will avoid it.</li> <li>Until we change how we view ourselves, what we believe we are and are not, we cannot overcome our avoidance and anxiety, and so we cannot change.</li> </ul> <h4>Kill Yourself</h4> <ul> <li>There is little that is unique or special about your problems, and it's pure narcissism to believe otherwise.</li> <li>Redefine your metrics in mundane and broad ways. The narrower and rarer the identity you choose for yourself, the more everything will seem to threaten your identity.</li> </ul> <h4>How to Be a Little Less Certain of Yourself</h4> <ul> <li>To create uncertainty, ask yourselves three questions:</li> <li><em>What if I'm wrong?</em> We are the worst observers of ourselves, and so chip away at your certainty by consistently questioning how wrong we might be about ourselves.</li> <li><em>What would it mean if I'm wrong?</em> Being able to evaluate different values without necessarily adopting them is perhaps <em>the</em> central skill in changing one's own life meaningfully.</li> <li><em>Would being wrong create a better or worse problem than my current problem, for both myself and others?</em> Our problems are endless, so we must look at which problem is better.</li> <li>If it feels like you versus the world, chances are it's really just you versus yourself.</li> </ul> <h3>Chapter 7: Failure Is the Way Forward</h3> <h4>The Failure/Success Paradox</h4> <ul> <li>Improvement at anything is based on thousands of tiny failures, and the magnitude of your success is based on how many times you've failed at something.</li> <li>At some point, most of us reach a place where we're afraid to fail, and stick only to what is placed in front of us or only what we're really good at.</li> <li>Shitty values involve tangible external goals outside our control, and once they're achieved you feel empty because there are no more problems to solve.</li> <li>Better values are process-oriented, and their problems must continuously be re-engaged. Such a value is an ongoing, lifelong process that defies completion.</li> <li>Consequently goals, as they are traditionally defined, are limited in the amount of happiness they can produce in our lives.</li> </ul> <h4>Pain Is Part of the Process</h4> <ul> <li>We must suffer emotional pain to develop greater emotional resilience, a stronger sense of self, increased compassion, and a generally happier life.</li> <li>If you avoid pain by chasing highs or indulging in entitlement or overindulging in substances, then you'll never generate the requisite motivation to actually change.</li> <li>When you choose a new value, you choose to introduce a new form of pain into your life. Learn to sustain it, and act despite it.</li> </ul> <h4>The "Do Something" Principle</h4> <ul> <li>The "do something" principle states: Action isn't just the effect of motivation; it's also the cause of it.</li> <li>We assume a chain reaction of <em>emotional inspiration ⇒ motivation ⇒ desirable action</em>, but it's actually an endless loop: <em>inspiration ⇒ motivation ⇒ action ⇒ inspiration</em>.</li> <li>We can therefore reorient our mindset around the chain <em>action ⇒ inspiration ⇒ motivation</em>.</li> <li>So if you lack the motivation to make an important change in your life, <em>do something</em> and then harness the reaction to that action as a way to begin motivating yourself.</li> <li>With <em>doing something</em> as your only metric for success, then even failure pushes you forward.</li> </ul> <h3>Chapter 8: The Importance of Saying No</h3> <ul> <li>The only way to achieve meaning and a sense of importance is one's life is through a rejection of alternatives, or a narrowing of freedom&nbsp;–&nbsp;a choice of commitment to one place, one belief, or one person.</li> <li>Exposure to different cultural values and metrics forces you to reexamine what seems obvious in your own life and whether it's actually the best way to live.</li> <li>There is such pressure in the West to be likable that people often reconfigure their entire personally depending on whom they're dealing with.</li> </ul> <h4>Rejection Makes Your Life Better</h4> <ul> <li>We need to reject something. Or else we stand for nothing, and are without values and live our lives without any purpose.</li> <li>To truly appreciate something, you must confine your life to it. There is a certain level of joy and meaning you reach only after focusing significant time on a single relationship, craft, or career.</li> <li>Entitled people, because they feel as though they <em>deserve</em> to feel great all the time, avoid rejecting anything because doing so might make themselves or others feel bad.</li> </ul> <h4>Boundaries</h4> <ul> <li>Healthy relationships are defined by 1) each person accepting responsibility, and 2) each person being willing to both reject and be rejected by their partner.</li> <li>People in healthy relationships with strong boundaries take responsibility for their own values and problems, and do not take responsibility for those of their partner.</li> <li>Entitled people either expect take others to take responsibility for their problems, or take on too much responsibility for other people's problems.</li> <li>Entitled people adopt these strategies in their relationships, as with everything, to help avoid accepting responsibility for their own problems.</li> <li>In an unhealthy relationship two people solve each other's problems to feel good about themselves. In a healthy relationship they solve their own problems to feel good about each other.</li> <li>"Victims" and "savers" end up in relationships because they use each other to achieve emotional highs.</li> <li>Acts of love are valid only if they're performed without conditions or expectations.</li> <li>It's not about giving a fuck about everything your partner gives a fuck about; it's about giving a fuck about your partner regardless of the fucks he or she gives.</li> </ul> <h4>How to Build Trust</h4> <ul> <li>Conflict exists to show us who is there for us unconditionally and who is just there for the benefits.</li> <li>A healthy relationship requires both sides to be willing and able to say no and hear no, or else boundaries break down and one person's problems and values dominate the other's.</li> <li>When trust is destroyed, it can be rebuilt only if 1) the trust-breaker admits and owns up to the true values that caused the breach, and 2) the trust-breaker builds a solid track record of improved behavior over time.</li> </ul> <h4>Freedom Through Commitment</h4> <ul> <li>The more options we're given, the less satisfied we become with whatever we choose, because we're aware of all the other options we're potentially forfeiting.</li> <li>There are some experiences that you can have only when living in the same place, being with the same person, or working on the same craft for significant time.</li> <li>You will find increased opportunity and upside in rejecting alternatives and distractions in favor of what you've chosen to let truly matter to you.</li> <li>The rejection of alternatives liberates us from things that do not align with our most important values or with our chosen metrics.</li> </ul> <h3>Chapter 9: ... And Then You Die</h3> <ul> <li>In a backwards way, death is the light by which the shadow of all life's meaning is measured.</li> </ul> <h4>Something Beyond Our Selves</h4> <ul> <li>We are the only animals that are aware of the inevitability of our own death; this "death terror" is a deep existential anxiety that underlies everything we think or do.</li> <li>To compensate for our mortality, we try to construct a conceptual self that will live forever. All human civilization is a result of such "immortality projects."</li> <li>Our immortality projects are our values – the barometers of meaning and worth in our life. And when our values fail, so do we, psychologically speaking.</li> <li>Rather than attempting to implement our conceptual self across the world, we should question our conceptual self and become more comfortable with the reality of our own death.</li> <li>Becoming comfortable with our mortality allows us to choose values more freely, unrestrained by the quest for immortality, and freed from dangerous dogmatic views.</li> </ul> <h4>The Sunny Side of Death</h4> <ul> <li>Death confronts us all with a painful and important question: What is your legacy?</li> <li>To be comfortable with death, see yourself as something bigger than yourself, choose values that go beyond serving yourself, and that are simple and immediate and controllable and tolerant of our chaotic world.</li> <li>Our culture today confuses great attention with great success, but they are not the same.</li> <li>There is nothing to be afraid of. Ever.</li> </ul> https://tedneward.github.io/Research/thinking/tempo/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/tempo/index.html <p><em>(by Venkatesh Rao, (publisher))</em></p> https://tedneward.github.io/Research/thinking/red-team-handbook/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/red-team-handbook/index.html <p><em><a href="Red_Team_Handbook.pdf">PDF</a></em></p> <h2>Ch 1: Introduction</h2> <p>As human beings, we develop patterns of behavior and thought that help us achieve our goals with the least amount of effort possible. For example, we learn early in life that we can have greater success and more friends if we cooperate and agree with other people – go along to get along. To save time and energy, we develop shortcuts and apply solutions that work in one area to problems in another, even if the responses don’t fit perfectly. We assume we know more than we really do, and we don’t question our assumptions. The introverts among us, despite having valuable ideas, cede control in meetings to the extroverts and remain mute. These actions and this learned behavior combine to deceive us. We assume we are applying the best solutions without reflecting on our actions and asking if there is a better way, or if we are really applying the correct thought and behavior to get the outcomes we want. When we join together in groups, these human characteristics amplify, and our tendencies and learned patterns of behavior lead us to situations like the planning meeting described above.</p> <p>Expanding on the words of psychologist Dietrich Dörner, people court failure in predictable ways, by degrees, almost imperceptibly, and according to their own culture and context. In other words, we routinely take shortcuts because of limitations on time, personnel, or other resources, and we accept that as a normal way of doing business. We assume we understand situations because we have been in similar ones before, and we turn a blind eye to ambiguity or don’t fully appreciate asymmetries. We discount potential threats because we don’t fully appreciate the likelihood of occurrence or the complexity of influencing factors. We make many small decisions that are individually “close enough,” but when joined together, become the seeds of failure. We take comfort in the familiar, and assume others, even on the other side of the world, share our views, beliefs, and tendencies. These reasons and more are why we Red Team.</p> <blockquote> <p>“Every assumption we hold, every claim, every assertion, every single one of them must be challenged.” --CSA Mark A. Milley</p> </blockquote> <p><strong>What is Red Teaming</strong> Red Teaming is a flexible cognitive approach to thinking and planning that is specifically tailored to each organization and each situation. It is conducted by skilled practitioners normally working under charter from organizational leadership. It uses structured tools and techniques to help us ask better questions, challenge explicit and implicit assumptions, expose information we might otherwise have missed, and develop alternatives we might not have realized exist. It cultivates mental agility to allow Red Teamers to rapidly shift between multiple perspectives to develop a fuller appreciation of complex situations and environments. This leads to improved understanding, more options generated by everyone (regardless of rank or position), better decisions, and a level of protection from the unseen biases and tendencies inherent in all of us.</p> <p>This is built on four main principles:</p> <ul> <li>Self-awareness and Reflection</li> <li>Groupthink Mitigation and Decision Support</li> <li>Fostering Cultural Empathy</li> <li>Applied Critical Thinking</li> </ul> <h2>Ch 2: Self-Awareness and Recognition</h2> https://tedneward.github.io/Research/thinking/superforecasters/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/superforecasters/index.html <p><em>(Tetlock, Gardner; Crown Publishers, 2015)</em></p> <h1>Chapter 1: An Optimistic Skeptic</h1> <p>Tom Friedman (globally-recognized predictor) vs Bill Flack (virtually unknown predictor who is right quite often).</p> <h2>The One About the Chimp</h2> <p>The average expert was roughly as accurate as a dart-throwing chimpanzee. Author conducted a twenty-year (1984 - 2004) study: gathered up a big group of experts (academics, pundits, etc) to make thousands of predictions about the economy, stocks, elections, wars, and other issues of the day. Time passed, and when the researcher checked the accuracy of the predictions, he found that the average expert did as well as random guessing.</p> <p>People are in front of the cameras for reasons other than their skill at predictions. Old forecasts are like old news--soon forgotten--and pundits are almost never asked to reconcile what they said with what actually happened. <em>The one undeniable talent that talking heads have is their skill at telling a compelling story with conviction.</em></p> <p>It was easiest to beat chance on the shortest-range questions that only required looking one year out, and accuracy fell off the further out experts tried to forecast, approaching random-chance levels at 3-5 years out.</p> <p>Author: "Call me an optimistic skeptic."</p> <h2>The Skeptic</h2> <p>(The story of Mohamed Bouazizi, 17 Dec 2010, who self-immolated in Tunisia after a police shakedown and started the Arab Spring.) Could Tom Friedman have predicted the Arab Spring? Of course not. <em>(Black swan event.)</em></p> <p>1972: Meteorologist Edward Lorenz publishes "Predictability: Does the Flap of a Butterfly's Wings in Brazil Set Off a Tornado in Texas?" about tiny data entry variations in computer simulations (0.506127 with 0.506) could product dramatically different long-term forecasts. Inspired "chaos theory": in nonlinear systems (like the atmosphere) even small changes in initial conditions can mushroom to enormous proportions. This shifted scientific opinion toward the view that there are hard limits on predictability, a deeply philosophical question.</p> <h2>The Optimist</h2> <p>It is one thing to recognize the limits on predictability, and quite another to dismiss <em>all prediction</em> as an exercise in futility. We make mundane predictions routinely (when to leave the house to avoid traffic, how to anticipate other drivers' behavior, expecting other people to follow through on their commitments to meetings, etc). So much of our reality is this predictable, or more so.</p> <p>Of course each of these pockets of predictability can be abruptly punctured: a restaurant very likely will open its doors when it says it will, but it may not for a number of reasons (manager sleeping late, fire, bankruptcy, pandemic, nuclear war, ...). There are no certainties in life--not even death or taxes, if we assign a nonzero probability to the invention of technologies that let us upload the contents of our brains into a cloud-computing network and the emergence of a future society so public-spirited and prosperous that the state can be funded with charitable donations.</p> <p><strong><em>How predictable something is depends on what we are trying to predict, how far into the future, and under what circumstances.</em></strong></p> <p>Forecasters are often groping in the dark--they have no idea how good their forecasts are in the short, medium, or long term--and no idea how good their forecasts could become. At best, they have vague hunches. That's because the forecast-measure-revise procedure operates only within the rarified confines of high-tech forecasting. It's a demand-side problem: the consumers of forecasting don't demand evidence of accuracy, so there is no measurement, which means no revision, and without revision, they can be no improvement.</p> <p>(History of the Good Judgment Project (GJP) and its efforts.)</p> <p>Two conclusions emerge from this research: One, foresight is real. Two, what makes these superforecasters so good isn't who they are, it's what they do: <strong><em>Foresight isn't a mysterious gift bestowed at birth. It is the product of particular ways of thinking, of gathering information, of updating beliefs.</em></strong> Broadly speaking, <strong><em>superforecasting demands thinking that is open-minded, careful, curious and above all, self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly--only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance.</em></strong></p> <h2>A Forecast About Forecasting</h2> <p>(Discussions about human judgment vs computer machine learning/AI.) "I think it's going to get stranger and stranger" for people to listen to the advice of experts whose views are informed only by their subjective judgment--human thought is beset by psychological pitfalls. "So what I want is that human expert paired with a computer to overcome the human cognitive limitations and biases." (David Ferrucci, IBM Watson chief engineer)</p> <p>We will need to blend computer-based forecasting and subjective judgment in the future.</p> <h1>Chapter 2: Illusions of Knowledge</h1> <p>(Discussions of cognitive biases and people loudly proclaiming incorrect conclusions with no proof.)</p> <h1>Chapter 3: Keeping Score</h1> <h1>Chapter 4: Superforecasters</h1> <h1>Chapter 5: Supersmart?</h1> <h1>Chapter 6: Superquants?</h1> <h1>Chapter 7: Supernewsjunkies?</h1> <h1>Chapter 8: Perpetual Beta</h1> <h1>Chapter 9: Superteams</h1> <h1>Chapter 10: The Leader's Dilemma</h1> <h1>Chapter 11: Are They Really So Super?</h1> <h1>Chapter 12: What's Next?</h1> <h1>Appendix: 10 Commandments for Aspiring Superforecasters</h1> <p>For more details visit <a href="http://www.goodjudgment.com">www.goodjudgment.com</a></p> <ol> <li> <p><strong>Triage</strong>: Focus on questions where your hard work is likely to pay off; don't waste time on either easy "clocklike" questions (where simple rules of thumb can get you close to the right answer) or impenetrable "cloud-like" questions (where even fancy statistical models can't be the dart-throwing champ). Concentrate on questions in the Goldilocks zone of difficulty, where effort pays off the most. ... Triage judgment calls get harder as we come closer to home. ... Certain classes of outcomes have well-deserved reputations for being radically unpredictable, but we usually don't discover how unpredictable outcomes are until we have spun our wheels for a time trying to gain analytical traction. Bear in mind the two basic errors it is possible to make here: We could fail to try to predict the potentially predictable, or we could waste our time trying to predict the unpredictable. Which error would be worse in the situation you face?</p> </li> <li> <p><strong>Break seemingly intractable problems into tractable sub-problems.</strong> Decompose the problem into its knowable and unknowable parts. Flush ignorance into the open. Expose and examine your assumptions. Dare to be wrong by making your best guesses. Better to discover errors quickly than to hide them behind vague verbiage.</p> </li> <li> <p><strong>Strike the right balance between inside and outside views.</strong> Nothing is 100% "unique"; uniqueness is a matter of degree. Conduct creative searches for comparison classes even for seemingly unique events. Superforecasters are in the habit of posing the outside-view question: How often do things of this sort happen in situations of this sort? (Summers' estimates: Larry Summers, a Harvard professor knows about the planning fallacy--estimators tend to underestimate the time they need, often by factors of two or three--so he doubles the estimate, then moves to the next higher time unit. "One hour" becomes "two days", and "two days" becomes "four weeks".)</p> </li> <li> <p><strong>Strike the right balance between under- and over-reacting to evidence.</strong> Belief updating is necessary. Don't suppose that belief updating is always easy because it sometimes is; skillful updating requires teasing subtle signals from noisy news flows, all the while resisting the lure of wishful thinking. Savvy forecasters learn to ferret out telltale clues before the rest of us, snooping for nonobvious lead indicators--what would have to happen before "X" could. The best forecasters tend to be incremental belief updaters, often moving from probabilities of 0.4 to 0.35 or from 0.6 to 0.65. Yet superforecasters also know how to jump (move their probability estimates fast in response to diagnostic signals).</p> </li> <li> <p><strong>Look for the clashing causal forces at work in each problem.</strong> For every good policy argument, there is typically a counterargument that is at least worth acknowledging. Each side should list, in advance, the signs that would nudge them toward the other. In classical dialectics, thesis meets antithesis, producing synthesis. Synthesis is an art that requires reconciling irreducibly subjective judgments.</p> </li> <li> <p><strong>Strive to distinguish as many degrees of doubt as the problem permits but no more.</strong> Few things are either certain or impossible, and "maybe" isn't all that informative. The more degrees of uncertainty you can distinguish, the better a forecaster you are likely to be. Translating vague-verbiage hunches into numeric probabilities feels unnatural at first but it can be done with patience and practice. Don't reserve rigorous reasoning for trivial pursuits.</p> </li> <li> <p><strong>Strike the right balance between under- and over-confidence, between prudence and decisiveness.</strong> Superforecasters understand the risks of both rushing to judgment and of dawdling too long near "maybe". Long-term accuracy requires getting good scores on both calibration and resolution.</p> </li> <li> <p><strong>Look for the errors behind your mistakes but beware of rearview-mirror hindsight biases.</strong> Don't try to justify or excuse your failures--own them! Conduct unflinching postmortems on both failures and successes: Where exactly did I go wrong (or right)? Remember that although the more common error is to learn too little from failure and to overlook flaws in your basic assumptions, it is also possible to learn too much (you may have been basically on the right track but made a minor technical mistake that had big ramifications).</p> </li> <li> <p><strong>Bring out the best in others and let others bring out the best in you.</strong> Master the fine arts of team management, especially perspective-taking (understanding the arguments of the other side so well that you can reproduce them to the other's satisfaction), precision questioning (helping others to clarify their arguments so they are not misunderstood), and constructive confrontation (learning to disagree without being disagreeable).</p> </li> <li> <p><strong>Master the error-balancing bicycle.</strong> Implementing each commandment (these 10) requires balacing opposing errors. You can't become a superforecaster by reading training manuals. Learning requires doing with good feedback that leaves no ambiguity about whether you are succeeding or whether you are failing. Practice is not just going through the motions of making forecasts, or casually reading the news and tossing out probabilities--superforecasting is the product of deep, deliberative practice.</p> </li> <li> <p><strong>Don't treat commandments like commandments.</strong> "It is impossible to lay down binding rules because two cases will never be exactly the same." --Helmuth von Moltke Guidelines are the best we can do in a world where nothing is certain or exactly repeatable. Superforecasting requires constant mindfulness, even when you are dutifully following these commandments.</p> </li> </ol> https://tedneward.github.io/Research/thinking/the-edge-of-sentience/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/the-edge-of-sentience/index.html <p><a href="https://www.edgeofsentience.com/formats">Website</a> | <a href="">PDF</a></p> <h2>Abstract</h2> <p>Can octopuses feel pain and pleasure? What about crabs, shrimps, insects, or spiders? How do we tell whether a person unresponsive after severe brain injury might be suffering? When does a fetus in the womb start to have conscious experiences? Could there even be rudimentary feelings in miniature models of the human brain, grown from human stem cells? And what about AI? These are questions about the edge of sentience, and they are subject to enormous, disorienting uncertainty. The stakes are immense, and neglecting the risks can have terrible costs. We need to err on the side of caution, yet it’s often far from clear what ‘erring on the side of caution’ should mean in practice. When are we going too far? When are we not doing enough? The Edge of Sentience presents a comprehensive precautionary framework designed to help us reach ethically sound, evidence-based decisions despite our uncertainty.</p> <h2>Analysis</h2> <p>... from <a href="https://fivebooks.com/best-books/best-philosophy-books-2024-nigel-warburton/">"Five Books"</a>:</p> <p>"Jonathan Birch is a philosopher with a research team at the London School of Economics. He’s particularly interested in the idea that we ought to base our ethical interactions with animals on scientific studies of what animals are actually like. So he works with zoologists, ethologists, people who look very closely at animal behaviour.</p> <p>"There are amazing studies that reveal for instance, that individual honey bees are able to learn and are even capable of playing. An insect, which doesn’t seem highly intelligent, and which normally functions as part of a hive, can exhibit surprising abilities if you look closely at what it’s actually doing, or put it in laboratory situations that allow those capabilities to emerge.</p> <p>"Jonathan’s interested in a very wide range of species, and often in species that don’t have cute faces. We don’t tend to attribute the same sensitivity and capacity to feel pain to an insect that we might to a sweet puppy or a baby chimp. Take lobsters. They are killed in quite gruesome ways for culinary purposes. He’s been involved in research that suggests that lobsters have quite sophisticated neural networks and seem to exhibit pain-like behaviour in certain situations. There is sufficient evidence, he believes, to exercise what he calls ‘the precautionary principle.’"</p> <p><em>That we should assume they feel pain, just in case?</em></p> <p>"Yes. The precautionary principle is this: once you get over a threshold of evidence, treat animals as if they are sentient even though there is still some doubt. He’s not saying they are sentient, but that there is sufficient evidence to be more careful about how we treat them. So, you don’t want to drop them in boiling water and let them die slowly, for example. If we are set on killing them and it’s possible to stun lobsters or kill them electronically, then we should do this rather than resort to the boiling water in a saucepan method of killing them.</p> <p>"As I say, he’s not conclusively arguing that they do feel pain. This is the driving force in the way he approaches moral issues around how we treat other animals. If it seems they have the capacity, and there’s scientific evidence to back that up,, then let’s be more careful than we have been to date. Obviously, a lot of factory farming would be beyond the pale for him."</p> <p><em>Quite rightly, I think.</em></p> <p>"Agreed. But many people find it difficult to imagine that, say, fish or insects have sophisticated capacity for pain, even though the evidence supports the idea that they probably do.</p> <p>"Birch is a superbly clear writer, and he’s very careful to adjust his belief according to the evidence. He’s not a sentimental animal ethicist who thinks, gosh, killing a mosquito is the same as killing a fox. He bases everything that he writes on the available science and sound reasoning from the best data we have.</p> <p>"He’s one of the most interesting public philosophers around today because of his willingness to engage with ideas which have a very broad interest across the world. He’s also adept at dissemination those ideas. By making his book freely available online he has guaranteed a wide readership. He appears on podcasts, radio, and television and is frequently interviewed, more often these days on the potential sentience of AI than on animals. But his deep interest is in sentience in animals."</p> <p><em>(by Jonathan Birch, ISBN 9780192870421)</em></p> https://tedneward.github.io/Research/thinking/simple-rules/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/simple-rules/index.html <p><em>(by Donald Sull, Kathleen M. Eisenhardt)</em></p> <h3>Introduction</h3> <p>Effective simple rules share four common traits:</p> <ul> <li> <p>They are limited to a handful</p> </li> <li> <p>Tailored to the person or organization using them</p> </li> <li> <p>Apply to a well-defined activity or decision</p> </li> <li> <p>Provide clear guidance while conferring the latitude to exercise discretion</p> </li> </ul> <p>"Weaver argued that simple and uncertain problems have largely been solved, and that the greatest challenges of the future would be problems of complexity. … At the personal level, many of us struggle to manage complexity every day. We have to call a teenager to navigate the three remote controls required to turn on ESPN, an accountant to fill out our tax forms, and the IT help desk for guidance every time Microsoft introduces a new version of its software." (p 10/Kindle location 143)</p> <p>"People often attempt to address complex problems with complex solutions. … Meeting complexity with complexity can create more confusion than it resolves. [US tax code example]" (p11/Kloc161)</p> <p>"Applying complicated solutions to complex problems is an understandable approach, but flawed. The parts of a complex system can interact with one another in many different ways, which quickly overwhelms our ability to envision all possible outcomes. … How many ways can you combine six Lego blocks? For one block, the answer is trivial: 1. With some work, most people can calculate that two blocks combine 24 ways, but by three blocks the calculation becomes tricky (the correct answer is 1,560). When you get to six blocks, the number of possible combinations is daunting. For decades, the commonly accepted answer for six blocks was 103 million combinations, until two mathematicians revisited the problem with some massive computer power and discovered 915 million combinations." (p13/Kloc 178)</p> <h3>Why Simple Rules Work</h3> <p>Four common traits:</p> <ul> <li> <p><strong><em>Simple rules consist of a handful of guidelines applied to a specific activity or decision.</em></strong> Keeping the number of rules to a handful forces you to focus on what matters most. To be used, rules have to be remembered, and limited the number to a handful makes this possible.</p> </li> <li> <p><strong><em>Simple rules are tailored to the situations of the particular people who will use them.</em></strong></p> </li> <li> <p><strong><em>Simple rules are applied to a single well-defined activity or decision.</em></strong></p> </li> <li> <p><strong><em>Simple rules give concrete guidance without being overly prescriptive.</em></strong> Simple rules leave room to exercise creativity and pursue unanticipated opportunities.</p> </li> </ul> <h4>Simple rules to seize opportunities</h4> <p>Jesuit order’s success; the "Formula":</p> <ul> <li> <p>carry out whatever task they were assigned</p> </li> <li> <p>favor education</p> </li> <li> <p>released Jesuits from their obligation to recite their daily prayers in unison</p> </li> </ul> <p>To be effective, simple rules must fit the task at hand. Simple rules work best when flexibility matters more than consistency. … Both flexibility and consistency have their advantages, but increasing one reduces the other. Detailed rules are particularly useful for avoiding catastrophic errors, such as plane crashes; pilots are fond of saying that "checklists are written in blood". Detailed rules enhance efficiency in executing routine activities; McDonald’s requires that each franchisee follow the 386-page operating manual. This detailed rule book reduces discretion to a bare minimum, and tips the balance to consistency to ensure predictability, efficiency, and few mistakes.</p> <h4>Simple rules produce better decisions</h4> <p>By using simple rules, people can function without constantly stopping to rethink every aspect of a decision every time they make it.</p> <p>There are some situations where simple rules are not just quick and easy mental shortcuts, they also produce surprisingly accurate decisions. This is especially true in situations when the links between cause and effect are poorly understood, when important variables are highly correlated, when a few factors matter most, and when a gap exists between knowing what to do and actually doing it. Simple rules do not trump complicated models every time, but they do so more often than you might think.</p> <ul> <li> <p>Simple rules can avoid "overfitting" (fitting a model to closely to historical data hardwires error into the model)</p> </li> <li> <p>Simple rules can focus only on the most critical variables. By ignoring peripheral factors and tenuous correlations, rules of thumb eliminate a great deal of noise.</p> </li> <li> <p>Simple rules also capture correlated information about context. … Psychologists have found that people tend to overweigh peripheral variables at the expense of critical ones when they try to take all factors into account. Simple rules minimize the risk of overweighing peripheral considerations by focusing on the criteria most crucial for making good decisions.</p> </li> <li> <p>Simple rules also make it more likely that people will act on their decisions, because they are easy to remember and less cumbersome to follow than complex guidelines for action.</p> </li> </ul> <p>Simple rules not only trigger people to act, they also keep them from abandoning a decision once they have made it.&nbsp;Willpower, it turns out, is more like a reservoir than a river. If we deploy willpower on one decision, we have less self-control available for our next decision. But research finds that people in periods of low willpower can [make good decisions], provided they follow a simple rule.</p> <h4>Simple rules promote collective behavior</h4> <p>Example: honeybees and how they select their new home; "Bees achieve this impressive feat of coordination by following simple rules, such as "Dance longer for better sites", "Follow the first dancer you bump into", and "Head-butt scouts promoting other sites". The rules provide individual scouts with guidance on what to do, while leaving them the freedom to explore unexpected opportunities." (p40/Kloc 433)</p> <p>Simple rules impose a minimal level of coordination, while leaving ample room for individuals to pursue their own objectives.&nbsp;Zipper uses six simple rules for members sharing a car: (1) report damage, (2) keep it clean, (3) no smoking, (4) fill 'er up, (5) return on time, and (6) pets in carriers. When the rules that guide collective behavior are simple and clear, members of a community can monitor one another and sanction anyone who violates the social norms.</p> <p>Simple rules work because they do three things well:</p> <ul> <li> <p>They allow for flexibility in the pursuit of new opportunities, avoiding the rigidity of too many rules and the chaos of none at all. They are particularly effective when the situation is in flux, flexibility trumps consistency, and the benefits of seizing opportunities exceed the cost of making mistakes.</p> </li> <li> <p>Simple rules can produce decisions that match or outperform more sophisticated decision models across a wide variety of possible scenarios, and do so quickly, with limited data requirements, and when cause and effect are imperfectly understood.</p> </li> <li> <p>Collective action, like the honeybees’ choice of nest, can emerge from simple rules even when individuals’ mental capacity is limited and no one member understands the situation in its entirety. By following a few simple rules, members of a community can produce results, like the choice of a safe nest or the protection of intellectual property, far better than what individuals could do on their own.</p> </li> </ul> <h3>Making Better Decisions&nbsp;(Rules for Decisions)</h3> <p>[Bail decisions]</p> <p><em>Boundary rules</em> help you decide between two mutually exclusive alternatives. Boundary rules also help you pick which opportunities to pursue and which to reject when faced with a large number of alternatives. Prioritizing rules rank options to decide which alternatives will receive limited resources, such as medical care during battle or cash in a startup. Prioritizing rules are particularly useful when you lack sufficient resources or time to do everything, or when people hold conflicting views about what to do. Stopping rules dictate when to reverse a decision. They provide guidance, for example, on when to sell a stock, end the search for a mate, or descend from a treacherous mountaintop.</p> <h4>Boundary rules</h4> <p>Because these rules define the boundaries of inclusion or exclusion, they sometimes take the form of negative prohibitions, like the "thou shalt nots" of the Commandments. … Boundary rules narrow down the alternatives, helping people decide which opportunities to pursue in the face of an overwhelming number of choices. Boundary rules provide a quick screen to select the most promising targets based on readily available information which is highly correlated with what makes an opportunity attractive.</p> <p>Example: DARPA’s research initiatives; two boundary rules: the project must further the quest for fundamental scientific understanding, and second, it must have a practical use.</p> <p>Example: clinical depression; four boundary rules: Have you cried more than usual within the last week? Have you been disappointed in yourself or hated yourself within the last week? Have you felt discouraged about the future within the last week? Have you felt that you failed in your life within the last week? Patients who answer yes to all four questions are likely to be clinically depressed, and the rules correctly classified depressed patients over 97% of the time.</p> <p>Boundary rules are also easier to understand and communicate than the results of complicated statistical studies. … Complex models, which make simple statistical problems like [the breast cancer calculation percentage] look like child’s play, are difficult to understand and explain, leading to dangerous misunderstandings. Boundary rules can translate statistical findings into easy-to-use decision aids.</p> <p>Boundary rules can also translate broad policies into practical guidelines. [Example: Obama’s three rules for drone strikes.]</p> <p>Boundary rules guide the choice of what to do (and not do) without requiring a lot of time, analysis or information. Boundary rules work well for categorial choices, like a judge’s yes-or-no decision on a defendant’s bail, and decisions requiring many potential opportunities to be screened quickly. These rules also come in handy when time, convenience, and cost matter. Boundary rules cover the basics of what to do.</p> <h4>Prioritizing rules</h4> <p><em>Prioritizing rules</em> can help you rank a group of alternatives competing for scarce money, time, or attention. Prioritizing rules are also relevant in our personal lives for guiding everything from deciding how we spend our limited free time to ranking home-improvement projects. Prioritizing rules are useful when a large number of opportunities meet the threshold of the boundary rules, but resources are limited.</p> <p>[Example: Brazilian rail system decisions]</p> <p>[Example: Valentinian III, providing guidance on how to resolve Rome’s massive conflicting set of laws and interpretations, issues the Law of Citations: first specified a rule that limited the use of historical opinions to those written by five jurists widely acknowledged to be the greatest legal thinkers Rome had ever produced; then laid out four prioritizing rules that further clarified what judges should do: (1) when the jurists unanimously agreed on the issue, the judge was to follow their opinion; (2) if there was disagreement, the judge should follow the majority; (3) if the historical opinions were divided evenly, the judge should follow the opinion of Papinian; (4) if there was a tie among jurists and Papinian did not express an opinion, the judge could rely on his own discretion to decide the matter.]</p> <p><em>Stopping rules</em></p> <p>[Example: Crickets choosing mates] "Choose a mate who meets your quality threshold" is known as the fixed-threshold&nbsp;strategy; lowering the bar if they do not run across enough high-quality options adhere to what is known as the variable-threshold strategy.</p> <p>Whenever alternatives present themselves sequentially (as opposed to all at once), the question of when to stop searching and make a choice arises. Knowing when to stop searching is a thorny problem. Continuing the quest costs time, effort, and opportunity—you forgo listening to a good song while trying to find a great one. But the next tune might be worth the wait. You just don’t know. These are stopping rules. "Nobel Prize-winning economist Herbert A. Simon argued that individuals lack the information, time, and brainpower to determine the single best option when faced with a sequence of alternatives. Instead, like crickets, they rely on a rule of thumb to stop searching when they find an alternative that is good enough. Simon called this rule&nbsp;satisficing, an inelegant but descriptive combination of "satisfy" and "suffice"." (p64/Kloc829)</p> <p>A well-documented tendency among human decision makers, known as the status quo bias, leads individuals to hold 'em when they should fold 'em across a range of decisions.</p> <p>[Example: Loeb’s investing strategy rule: "If an investment loses 10 percent of its initial value, sell it."]</p> <p>[Example: Chicago diners vs Paris eaters; Parisians used rules like "Stop eating when I start feeling full", whereas Chicago were more likely to follow rules linked to external factors, such as "Stop eating when I run out of a beverage" or "Stop eating when the TV show I’m watching is over".]</p> <p>Stopping rules are particularly critical in situations when people tend to double down on a losing hand. ... Several factors increase the odds of this error, such as proximity to completion, the desire to save face, reluctance to write off sunk costs, and the fear of living with regret of not trying.</p> <p>[Example: Mt Everest climbing disaster]</p> <h3>Doing Things Better&nbsp;(Rules for Actions)</h3> <p>[Example: Jack’s bar, author-as-bouncer; simple rules for keeping the peace: "Don’t let trouble in the door"; "Stay sober until the last patron leaves"; "Double up for heavy metal, ska, and punk bands"; "Keep the bikers on your side"]</p> <p><em>Process rules</em> are more about how to do things better, and center on getting the job at hand done. Process rules work because they steer a middle path between the chaos of too few rules that can result in confusion and mistakes, and the rigidity of so many rules that there is little ability to adapt to the unexpected or take advantage of new opportunities. Simple put, process rules are useful whenever flexibility trumps consistency.</p> <p><em>How-to rules</em> guide the basics of executing tasks; coordination rules center on getting something done when multiple actors—people, organizations or nations—have to work together; timing rules center on getting things done in situations where temporal factors such as rhythms, sequences and deadlines are relevant.</p> <h4>How-to rules</h4> <ul> <li> <p>How-to rules can provide robust guidance.</p> <p>[Example: sports announcing; Seymour Joly de Lotbiniere, the "Lenin of the commentary revolution". A good commenter should: (1) set the scene, (2) describe the action, (3) give the score or results, regularly and succinctly, (4) explain, without interrupting, the stadium’s reaction to the game’s event, (5) share "homework", such as historical facts and figures or personal information, and (6) assess the significance of the occasion and key moments.]</p> </li> <li> <p>How-to rules are particularly helpful when there is extreme pressure and severe time constraint.</p> <p>[Example: the Mann Gulch tragedy; four simple rules for how to deal with out-of-control fires: (1) start an escape fire in the path of the advancing fire if possible; (2) go to where the fuel is thinner; (3) turn toward the fire and try to work through it; (4) don’t let the fire choose the spot where it hits you]</p> </li> <li> <p>How-to rules can also foster artistic creativity.</p> <p>"At first glance, this seems counterintuitive, because following the rules and being creative are often viewed as antithetical. The reality is, however, that a blank canvas and no rules on how to fill it can overwhelm an artist with too many degrees of freedom."</p> <p>[Examples of Chinese creativity: first group was simply given the assignment; second group received "Please try to be creative"; third group received simple rules on how to complete the activity, such as "Fold or tear the stickers to vary the shape and size of the materials".]</p> <p>Simple rules are not just for beginners, but rather can also stimulate the creativity of master artists.&nbsp;</p> <p>"Patricia Stokes, a painter and psychologist at Columbia University, studied how influential artists like Claude Monet, William Motherwell, and Piet Mondrian produced their groundbreaking work. She argues that truly original artists work by imposing constraints on themselves, in terms of the subjects they paint, materials they use, and artists they draw upon for inspiration. Monet, for example, purposefully limited his subjects, repeatedly painting pictures, by the dozens, of subjects like grain stacks and water lilies. This self-imposed constraint allowed him to focus on exploring how light changes, and his exploration helped spark a transition in the art world from representation to impressionism, setting the stage for twentieth-century artists like Picasso. By constraining the infinite possibilities, simple rules allow creativity to flourish, less from thinking outside the box and more from deciding how to draw the box in the first place." (p77/Kloc 1000)</p> </li> <li> <p>How-to rules can also accelerate creativity.</p> <p>[Example: White Stripes album. Five rules: (1) no blues; (2) no guitar solos; (3) no slide guitar; (4) no covers; (5) no bass. 18 songs in 10 days.) "By restricting their creative process, how-to rules freed the White Stripes to follow a short, clear path to their favorite patch of creativity.</p> <p>Simple rules can help balance novelty and continuity and add a dash of efficiency to the creative process.</p> <p>[Example: Elmore Leonard’s forty-six novels. "Avoid prologues", "Never use a verb other than 'said’ to carry dialogue", "Try to leave out the part that readers tend to skip".]</p> <p>How-to rules address the basics of getting things done without prescribing every detail of what to do. They work well in situations where the unexpected is the expected. They can stimulate creativity, shape action when there is no time to plan, and create structure that allows entrepreneurs to grow their company.</p> </li> </ul> <h4>Coordination rules</h4> <p>[Example: flocking behavior of starlings and other birds; Craig Reynolds avian avatar flocking rules: (1) avoid collisions; (2) head in the same direction as your nearest neighbors; (3) stay close to your nearest neighbors.]</p> <ul> <li> <p>Coordination rules work by clarifying what to do in relation to others.</p> <p>[Example: locust swarms: When the density of locusts exceeds a threshold, they begin to sneak up behind others to bit them. When forced into tight quarters, they follow two simple rules: (1) flee from the locusts chasing you from behind, and (2) try to eat the locust in front of you if it gets too close.]</p> <p>[Example: improv comedy]</p> </li> <li> <p>Coordination rules can be followed by using only local information. Coordination rules also allow for adaptation to local conditions while still recognizing the group objectives.</p> <p>[Example: Combat. Napoleon: "march toward the sound of gunfire"]</p> </li> <li> <p>Coordination rules delineate what to do, and likewise what others should do, so that collective objectives can be achieved. Coordination rules do not deal with when&nbsp;to do something.</p> </li> </ul> <h4>Timing rules</h4> <p>[Example: insomnia. Four simple rules: (1) get up at the same time every morning; (2) avoid going to bed until you feel sleepy; (3) do not stay in bed if you are not sleeping; (4) reduce the time spent in bed.]</p> <p><em>Timing rules</em> sometimes taking action when some triggering event happens; event pacing, this kind of rule links actions to events. Other timing rules call for action at a particular time on the clock or calendar—time pacing, creating deadlines and rhythms. Timing rules clarify when to do something and work best when temporal considerations like deadlines, rhythms, and sequences are relevant.</p> <p>[Example: dragonfly migration: (1) fly only when the nighttime temp falls for two consecutive nights; (2) stay put on windy days.]</p> <p>Time pacing rules dictate acting at a particular time on the clock or the calendar, and thereby create rhythms and deadlines.</p> <p>[Example: Pixar studios put into place timing rules: release a new movie every year; release movies at Thanksgiving.]</p> <p>Timing rules are particularly relevant in competitive situations when rivals go out of their way to disrupt their opponents.</p> <p>[Example: Riitta Katila has studied the global robotics industry. Finds that the best firms purposefully stay out of sync with their rivals. They stay ahead by introducing their newest products before their competitors, but they also stay behind by exploiting well-known technology to the fullest, and avoid releasing products when their rivals do.]</p> <p>[Example: Rothschild’s investing rule: "Buy when there is blood in the streets, even if that blood is your own."]</p> <p>Timing rules can also guide activities that follow predictable patterns or sequences.</p> <h3>Where Simple Rules Come From</h3> <p>[Example: "Hilltopping"--butterfly orgies; (a) "Fly uphill most of the time", (b) "Fly toward the highest slope in sight", (c) "Pause to check out local peaks, even if they are not the highest, but leave if you do not get lucky right away", (d) "Periodically make a random movement"]</p> <h4>Natural selection</h4> <p>Simple rules evolve in communities much as they do in societies as a whole; standup comedians and "Don't steal jokes"; "Rules evolve to address the most pressing issues in communities." "While evolved rules benefit from legitimacy and relevance, they also have weaknesses. Evolved rules are often implicit and deeply entrenched, making it difficult to examine them critically when circumstances change, or abandon them when they become dysfunctional. ... Without a guiding hand, rules often take on a life of their own." (p101/Kloc 1298)</p> <p>Four approaches that people commonly use to formulate rules:</p> <ul> <li> <p>People can develop their rules through thoughtful engagement with their own experience, a particularly effective approach for those with lots of relevant history</p> </li> <li> <p>People can also develop their rules by adopting the experiences of others, as conveyed through firsthand advice, books and analogies</p> </li> <li> <p>When high-quality scientific evidence exists, distilling it into simple rules can be the best approach</p> </li> <li> <p>Simple rules can arise through negotiation, when diverse stakeholders have divergent aims and views of what to do</p> </li> </ul> <h4>Codifying personal experience</h4> <p>[Example: Tina Fey's nine rules for managing a comedy show; (4) states, "When hiring, mix Harvard nerds with Chicago improvisers and stir"; (9) is "Never tell a crazy person he's crazy".]</p> <p>Values are another source of rules that often arise from personal experience. Personal values can dictate what is correct and even essential to do. "By drawing on their own experience, people are more likely to create relevant rules and to understand where the rules came from and why they matter. When people develop rules themselves and base them on values that matter to them, they also feel a greater ownership over their rules. People are more likely to use rules devised by themselves, reflecting their own values, rather than those imposed on them by someone else. Moreover, when people invest the time to reflect on their experience and codify it into rules, they typically do so with some goal that matters deeply to them." (p.103/Kloc 1333)</p> <p>[Example: UMichigan study looking at team of 45 employees looking to design a laser printer; need for engineering flow time leads to rules making sure engineers are uninterrupted]</p> <h4>Drawing on the experience of others</h4> <p>Advice from experienced people about how to ..., whether given directly or conveyed through a book or magazine article, can prove useful in formulating one's own approach.</p> <h4>Personal observation is another effective way to learn what works for others.</h4> <p>We also use analogies, finding similarities between our own experience and others', which can streamline the process of devising our own rules. ... Analogies involve three elements: (a) a new situation, (b) a prior example that is similar in important respects to the new situation, and (c) rules that can be transferred from the prior example to the current situation.</p> <p>Analogies to existing companies and other competitive domains like sports and battle are a favorite approach of entrepreneurs looking for guidance in attacking new opportunities. ... Analogies can be a powerful source of rules even when the comparison involves two very different domains.</p> <p>It's possible to be misled by superficial features when deciding when an analogy works. [Example: fictitious scenario involving conflict between small democratic country and large totalitarian one--red herring similarities to WW2 and Vietnam highly color the participants' results.]</p> <h4>Distilling scientific evidence</h4> <p>Scientific knowledge accumulates over time as researchers conduct diverse studies looking at complex phenomena from unique vantage points and with distinctive methodologies. Scientific evidence, however, is not inherently simple. The resulting knowledge is often full of qualifications and contingencies that hold under some circumstances, but not others. One way to develop simple rules is to review a body of scientific research, sort through to determine the most consistent findings across studies, and distill these findings into a few simple rules.</p> <p>[Example: Doctors distilling rules of infection for children.]</p> <p>[Example: Scientists distilling data for rules around sea life.]</p> <p>Distilling the evidence works when there is sufficient, high-quality evidence and when stakeholders view the situation in similar ways.</p> <h4>Negotiating an agreement</h4> <p>[Example: Author's experience with Executive Education at London Business School] "The key to moving past the political impasse was shifting the debate from, "Which programs should we run?" to "What simple rules should we use to decide which programs to run?" The Executive Education team met with faculty members, explaining the financial situation and drop in rankings, and asked the professors what guidelines they would suggest to manage the program portfolio. The team emphasized that they were aiming for a general set of rules that would apply to every program. The professors argued with one another, but ultimately converged on a handful of robust rules that would work across many situations. ... Of course, individual professors were not pleased when their own programs were cut. But since they had actively participated in negotiating the rules, they were much less likely to oppose the change." (p114/Kloc 1474)</p> <p>Negotiating simple rules works particularly well when it is cumbersome or impossible to resolve conflicts by escalating them to a higher authority. James Buchanan ... argued that when there are competing interests, it is critical for stakeholders to negotiate decision rules before haggling over specific decisions. An agreed set of rules provides a clear framework for settling contentious issues. When negotiating rules to guide future decisions, stakeholders cannot anticipate every situation or what their interests might be in every case. Since they cannot forecast every contingency, they are more likely to converge on general rules that a wide range of stakeholders can live with and consider fair across a broad set of choices.</p> <p>Negotiation can be a helpful approach even if there is a higher authority, when multiple interest groups are involved, their values conflict, and some facts are hard to come by. [Example: whale-watching rules]</p> <h3>Strategy as Simple Rules</h3> <p>[Example: eToro going from currency trading game platform to social network for traders]</p> <h3>Studying simple rules in action</h3> <p>"The best way to understand something is to try to change it."</p> <p>"Developing a strategy and implementing it are often viewed as two distinct activities--first you come up with the perfect plan and then you worry about how to make it happen. This approach, common though it is, creates a disconnect between what a company is trying to accomplish and what employees do on a day-to-day basis. ... Strategy and execution, in our view, cannot be separated, because they represent two sides of the same coin. ... 'Strategy as Simple Rules' argued that companies can close the gap between strategic intent and day-to-day action by adopting a strategy of simple rules, applying a handful of guidelines to critical activity or decision within the organization. A strategy of simple rules provides the flexibility to seize opportunities, produce good decisions when data and time are scarce, and help the various parts of the organization coordinate their activities to achieve common objectives." (p121/Kloc 1554)</p> <h4>Strategic process:</h4> <ul> <li> <p><strong><em>Figure out what will move the needles</em></strong>: identify the critical choices that will drive a wedge between revenues and costs to increase profits and sustain them over time</p> </li> <li> <p><strong><em>Choose a bottleneck</em></strong>: identify a bottleneck, a decision or activity that is preventing the company from achieving profitability</p> <p>In nearly every case, the company's goals were misunderstood, and top executives disagreed on what mattered most for the company's success. ... Without a clear understanding of what their company is trying to achieve, executives struggle to identify where to apply simple rules, let alone what the rules should be.</p> <p>The best bottlenecks to focus on share three characteristics: (a) they have a direct and significant impact on value creation--what moves the needles will vary from company to company and so should the chosen bottleneck; (b) chosen bottlenecks should represent recurrent decisions (rather than one-off choices), so the rules can be tested, refined, and used many times; (c) bottlenecks, as their name implies, arise when opportunities exceed available resources</p> <p>"The how&nbsp;from a who, what, how analysis is often a good place to start, and often points in the right direction. ... A company's how&nbsp;will often point to a broad process, such as marketing and sales or new-product development, crucial to creating economic value. These overarching processes consist of several discrete steps, each of which is a candidate for simple rules."</p> <p>"When searching for the right bottleneck, it helps to look for a critical activity where the number of opportunities exceeds the available resources, such as when sales opportunities outstrip a company's ability to meet demand. ... Choosing external partners is another bottleneck where opportunities often exceed a firm's capacity to pursue them. ... Decisions that require coordination across different departments or teams are another good place to look for bottlenecks."</p> <p>"When it comes to deciding where to apply simple rules, the most obvious activity is not always the right answer."</p> </li> <li> <p><strong><em>Craft the rules</em></strong>: a set of simple rules that, when applied to the bottleneck, improves profitability</p> <p>Developing rules from the top down is a big mistake. When leaders rely on their gut instincts, they overemphasize recent events, build in their personal biases, and ignore data that doesn't fit with their preconceived notions. It is much better to involve a team (4-8 members) and use a structured process to harness members' diverse insights and points of view. When drafting the team to develop simple rules, it is critical to include some of the people who will be using them on a day-to-day basis. ... Having users make the rules confers several advantages: (a) they are closest to the facts on the ground and best positioned to codify experience into usable rules; (b) because they will make decisions based on the rules, they can strike the right balance between guidance and discretion, avoiding rules that are overly vague or restrictive; (c) users can also phrase the rules in language that resonates for them, rather than relying on business jargon; (d) by actively participating in the process, users are more likely to buy into the final rules and therefore apply them in practice; (e) firsthand knowledge also makes it easier to explain the rules, and their underlying rationale, to colleagues who did not participate in the process.</p> <p>It is critical to test your first-cut rules in a rigorous fashion, and refine them in light of your findings.</p> </li> </ul> <h3>Getting Personal</h3> <p>[Applying the above three-step process to personal goals; same process, different scope; fill in any details later]</p> <p>[Example: dating]</p> <p>[Example: managing depression]</p> <p>[Example: winning friends and influencing people]</p> <h3>Rules for Improvement</h3> <h3>Breaking the Rules</h3> <h3>Conclusion</h3> <p>Overcoming the barriers to simplicity</p> <p>The effort required to develop simple rules.</p> <p>The people who benefit from complexity.</p> <p>The "myth of requisite complexity".</p> <p>Whether simple rules or detailed regulations are more effective is ultimately an empirical question that should be resolved by testing them to see which works better in the real world. ... Often, complex rules and regulations arise out of a distrust of human nature. If people cannot be trusted to do the right thing, detailed regulations are necessary to prevent malfeasance. [Example: Netflix's HR policies. "After studying their human resources policies, Netflix determined that 97% of their employees were trustworthy. ... Rather than continue to produce binders of detailed regulations, Netflix executives concentrated on not hiring people who would cause problems, and removing them quickly when hiring mistakes were made. This change allowed the company to replace thick manuals with simple rules. The company's policy for expenses, travel, gifts, and conducting personal business at work, for example, was reduced to four rules: (1) expense what you would not otherwise spend, (2) travel as if it were your own money, (3) disclose nontrivial gifts from vendors, and (4) do personal stuff at work when it is inefficient not to." (p227/Kloc2924)</p> <p>Rules to liberate</p> <p>Rules enable as well as constrain. They provide a threshold level of structure while leaving ample scope to exercise discretion. ... Close to the facts on the ground, individuals can draw on their judgment and creativity to manage risk and seize unexpected opportunities. The latitude to exercise discretion not only makes simple rules effective, it makes them attractive. People thrive when given the opportunity to apply their judgment and creativity to situations they face from day to day. And if they benefit from simple rules, they are more likely to use them and use them well.</p> https://tedneward.github.io/Research/thinking/sway/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/sway/index.html <p><em>(by (author), (publisher))</em></p> <h2>Preface</h2> <p>Asbestos &amp; open-heart surgery -&gt; ignoring the O-ring -&gt; diagnosing the wrong patient -&gt; where business &amp; psychology collide</p> <h2>Chapter 1: Anatomy of an accident</h2> <p>Taking off at Tenerife -&gt; the oversensitive egg shoppers -&gt; would you like insurance with that? -&gt; so long, Martha's Vineyard<br> * hidden forces/currents behind some of our decision making:<br> &nbsp;&nbsp; * loss aversion (tendency to go to great lengths to avoid possible losses)<br> &nbsp;&nbsp; * value attribution (inclination to imbue a person/thing with certain qualities based on an initial perceived value)<br> &nbsp;&nbsp; * diagnosis bias (blindness to all evidence that contradicts our initial assessment)</p> <h2>Chapter 2: The swamp of commitment</h2> <p>Playing not to lose -&gt; fun-n-gun -&gt; only the Gators walked out alive -&gt; the $204 twenty-dollar bill -&gt; the end of the Great Society -&gt; "we don't even know where the tunnel is"<br> * loss aversion + commitment =&gt; Vietnam</p> <h2>Chapter 3: The Hobbit &amp; the Missing Link</h2> <p>the real-life Indiana Jones -&gt; the hunt for the missing link -&gt; the Stradivarius on the subway -&gt; what's in a five-cent hot dog, anyway? -&gt; Homer Simpson &amp; Piltdown Man -&gt; can a discount drink decrease IQ? -&gt; Shakespeare was wrong -&gt; a paleontological lineup</p> <h2>Chapter 4: Michael Jordan &amp; the first-rate interview</h2> <p>the curse of the low draft pick -&gt; the "cold" professor -&gt; what lovesick college freshmen have in common w/ HR managers -&gt; when a pretty face equals a higher interest rate -&gt; the "mirror, mirror" effect -&gt; the Joe Friday solution</p> <h2>Chapter 5: The Bipolar Epidemic &amp; the Chameleon Effect</h2> <p>A psychiatric outbreak -&gt; sugar pills &amp; prozac -&gt; tricking Israeli army commanders -&gt; how to sound beautiful -&gt; how old do you feel? -&gt; the love bridge</p> <h2>Chapter 6: In France, the Sun revolves around the Earth</h2> <p>Who wants to trick a millionaire? -&gt; splitting the pie -&gt; sentimental car dealers -&gt; the talking cure for felons &amp; venture capitalists -&gt; russian justice -&gt; the rational machiguenga<br> * fairness is important in dealings, but fairness is relative to society</p> <h2>Chapter 7: Compensation &amp; cocaine</h2> <p>Switzerland's toxic conundrum -&gt; the GMAT rebels -&gt; the tower of the pleasure center &amp; hijacking altruism -&gt; fast times at "Commie High" -&gt; the anticipation factor<br> * pleasure center &amp; altruism centers cannot both operate simultaneously<br> * the anticipation factor drives the addictive behavior &amp; suppresses the altruism center</p> <h2>Chapter 8: dissenting justice</h2> <p>The Supreme Court conference -&gt; peer pressure &amp; coke-bottle glasses -&gt; Ferris Bueler &amp; the blocker -&gt; "we are not focusing ont he name you give to potatoes" -&gt; the captain is not God -&gt; not just thinking out loud -&gt; justice has been served<br> * Group dynamics: initiator, blocker, supporter, observer<br> * Power of even just one dissenter to break the tyranny of the majority<br> * Crew Resource Management (CRM): facts, challenge, action</p> https://tedneward.github.io/Research/thinking/thinking-fast-and-slow/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/thinking-fast-and-slow/index.html <p><em>(by Daniel Kahneman)</em></p> <p><em>I, <a href="http://omgitsmgp.com/">Michael Parker</a>, own this book and took these notes to further my own learning. If you enjoy these notes, please <a href="http://www.amazon.com/Thinking-Fast-Slow-Daniel-Kahneman/dp/0374533555">purchase the book</a>!</em></p> <h3>Introduction</h3> <ul> <li>We use resemblance as a simplifying heuristic to make difficult judgment, causing predictable biases in predictions.</li> <li>Social scientists in the 1970s broadly accepted that people are generally rational, and emotions such as fear, affection, and hatred explain departures from rationality.</li> <li>People tend to assess the relative importance of issues by the ease with which they are retrieved from memory, which is largely determined by the media.</li> <li>Accurate intuitions of experts are better explained by the effects of prolonged practice than by heuristics.</li> <li>Valid intuitions develop when experts have learned to recognize familiar elements in a new situation and to act in a manner that is appropriate to it.</li> <li>When faced with a difficult question, we often answer an easier one instead, usually without noticing the substitution.</li> <li>When intuition fails, because neither an expert solution nor a heuristic answer comes to mind, we resort to slower, deliberate, and effortful thinking.</li> </ul> <h3>Part 1: Two Systems</h3> <h4>Ch 1: The Characters of the Story</h4> <ul> <li>System 1 operates automatically and quickly, with no effort and no voluntary control.</li> <li>System 2 allocates attention to effortful mental activities. It's associated with the subjective experience of agency, choice, and concentration.</li> <li>All operations of System 2 require attention and are disrupted when attention is drawn away.</li> <li>System 2 has some ability to change the way System 1 works, by programming the normally automatic functions of attention and memory.</li> <li>System 1 generates impressions, intuitions, intentions, and feelings. Those endorsed by System 2 turn into beliefs and voluntary actions.</li> <li>Most of what System 2 thinks and does originates in your System 1, but System 2 takes over when things get difficult, and it normally has the last word.</li> <li>One of the tasks of System 2 is to overcome the impulses of System 1. System 2 is in charge of self-control.</li> <li>System 2 is too slow and inefficient to substitute for System 1. The best we can do is to recognize when mistakes are likely, and to try harder to avoid significant mistakes when the stakes are high.</li> </ul> <h4>Ch 2: Attention and Effort</h4> <ul> <li>Pupils are sensitive indicators of mental effort. The more System 2 exerts mental effort, the more they dilate.</li> <li>We decide what to do, but we have limited control over the effort of doing it. The task at hand decides this.</li> <li>Orienting and responding quickly to the gravest threats or most promising situations improved the chance of survival.</li> <li>In the economy of action, effort is a cost, and the acquisition of skill is driven by balancing benefits and costs. Laziness is in our nature.</li> <li>System 2 is the only one that can follow rules, compare objects on several attributes, and make deliberate choices between objects.</li> <li>A crucial capability of System 2 is that it can program memory to obey an instruction that overrides habitual responses.</li> <li>Multitasking is effortful. Time pressure is another driver. Any task that requires keeping several ideas in mind simultaneously has the same hurried character.</li> </ul> <h4>Ch 3: The Lazy Controller</h4> <ul> <li>Mihaly's flow is a state of effortless concentration so deep that people lose a sense of time, of themselves, and of their problems.</li> <li>This flow separates the two forms of effort: Concentration on the task and the deliberate control of attention.</li> <li>People who are cognitively busy are more likely to yield to temptation, make selfish choices, use sexist language, and make superficial judgments in social situations.</li> <li>Controlling thoughts and behaviors is one of the tasks that System 2 performs.</li> <li>If you exert self-control for a task, then you are less willing or able to exert self-control for a following task. This is called <em>ego depletion</em>.</li> <li>When you are thinking hard or exerting self-control, your blood glucose level drops. The implication is that you can undo ego depletion by ingesting glucose.</li> <li>When people believe a conclusion is true, they are also very likely to believe arguments that appear to support it, even when those arguments are unsound.</li> <li>Intelligence is not only the ability to reason; it is also the ability to find relevant material in memory and to deploy attention when needed.</li> <li>"Engaged" people are more alert, less willing to be satisfied with superficially attractive answers, and more skeptical about their intuitions.</li> <li>People who are not "engaged" are impulsive, impatient, and keen to receive immediate gratification.</li> </ul> <h4>Ch 4: The Associative Machine</h4> <ul> <li>The responses by System 1 are <em>associatively coherent</em>, yielding a self-reinforcing pattern of cognitive, emotional, and physical responses.</li> <li>Cognition is embodied. You think with your body, not only with your brain.</li> <li>Ideas are nodes in a vast network called associative memory, where causes link to effects, things to their properties, and things to their categories.</li> <li>Priming is not restricted to concepts and words. Events that you are not even aware of prime your actions and emotions.</li> <li>The influence of an action by the idea is called the ideomotor effect. It also works in reverse. For example, thinking of old age makes you act old, and vice versa.</li> <li>Money primes individualism: a reluctance to be involved with others, to depend on others, or to accept demands from others.</li> <li>Feeling that one's soul is stained triggers a desire to cleanse one's body, an impulse that is dubbed the Lady Macbeth Effect.</li> <li>System 1 provides the impressions that often turn into your beliefs, and is the source of impulses that often become the choices of our actions.</li> </ul> <h4>Ch 5: Cognitive Ease</h4> <ul> <li>Cognitive strain is affected by both the current level of effort and the presence of unmet demands. This mobilizes System 2.</li> <li>A repeated experience, clear display, primed idea, and good mood all increase cognitive ease. This in turn makes things feel familiar, true, good, and effortless.</li> <li>When strained, you are vigilant, suspicious, invest more effort, feel less comfortable, and make fewer errors. But you are less intuitive and less creative.</li> <li>Predictable illusions occur if judgment is based on an impression of cognitive ease or strain. For example, frequent repetition makes people believe lies.</li> <li>To craft a persuasive message, use high quality paper, bright colors, simple words, memorable verse, and quote the source with the simpler name.</li> <li>Cognitive strain, whatever the source, mobilizes System 2, which is more likely to reject the intuitive answer suggested by System 1.</li> <li>The mere exposure effect links the repetition of an arbitrary stimulus and the mild affection that people have for it. It's stronger for stimuli that we don't consciously see.</li> <li>Mood affects the operation of System 1. When we are uncomfortable and unhappy, we lose touch with our intuition.</li> </ul> <h4>Ch 6: Norms, Surprises, and Causes</h4> <ul> <li>The main function of System 1 is to maintain and update a model of your personal world, which represents what is normal in it.</li> <li>Norm theory is when a surprising event happens, and subsequent surprising events will appear more normal because they are interpreted in conjunction with it.</li> <li>We have norms for a vast number of categories, which provide the background for the immediate detection of anomalies.</li> <li>System 1 is adept at finding a coherent causal story that links the fragments of knowledge at its disposal.</li> <li>We are ready from birth to have <em>impressions</em> of causality, which do not depend on reasoning about patterns or causation. They are products of System 1.</li> <li>We are prone to apply causal thinking to situations that require statistical reasoning, but System 1 cannot do this, and System 2 requires necessary training.</li> </ul> <h4>Ch 7: A Machine for Jumping to Conclusions</h4> <ul> <li>Jumping to conclusions is efficient if the jump saves time and effort, the conclusion is likely correct, and the cost of an occasional mistake is acceptable.</li> <li>System 1 bets on answers, where recent events and the current context have the most weight in determining and interpretation. Otherwise, more distant memories govern.</li> <li>System 1 is gullible and biased to believe, while System 2 is in charge of doubting and unbelieving, but System 2 is sometimes busy and often lazy.</li> <li>Unlike scientists, which test hypotheses by trying to refute them, we seek data that are likely to be compatible with the beliefs that we currently hold.</li> <li>The tendency to like or dislike everything about a person, including things you have not observed, is known as the <em>halo effect</em>.</li> <li>The halo effect increases the weight of first impressions, sometimes to the point that subsequent information is mostly wasted.</li> <li>To derive the most useful information from multiple sources of evidence, you should always try to make these sources independent of each other.</li> <li>The standard practice of open discussion gives too much weight to the opinions of those who speak early and assertively, causing others to line up behind them.</li> <li>System 1 excels at constructing the best possible story that incorporates ideas currently activated, but it cannot allow for information that it does not have.</li> <li>Jumping to conclusions facilitates the achievement of coherence and of the cognitive ease that causes us to accept a statement as true. It explains overconfidence.</li> </ul> <h4>Ch 8: How Judgments Happen</h4> <ul> <li>System 1 continually assesses the problems that an organism must solve to survive. We equate good mood and cognitive ease with safety and familiarity.</li> <li>Faces with a strong chin and a slight confident-appearing smile exude confidence.</li> <li>A <em>judgment heuristic</em> is falling back on a simpler assessment that is made quickly and automatically and is available when System 2 must make its decision.</li> <li>Because System 1 represents categories by a prototype or a set of typical exemplars, it deals well with averages but poorly with sums.</li> <li>System 1 allows <em>matching</em> intensity across diverse and unrelated dimensions. This mode of prediction by matching is statistically wrong, although acceptable to both systems.</li> <li>The control over intended computations is far from precise, and we often compute much more than we want or need. This is called the <em>mental shotgun</em>.</li> </ul> <h4>Ch 9: Answering an Easier Question</h4> <ul> <li>If we can't satisfactorily answer a hard <em>target question</em>, then System 1 invokes <em>substitution</em> by recalling and answering an easier <em>heuristic question</em>.</li> <li>After answering a heuristic question, System 1 uses intensity matching to translate this answer to an answer of the target question.</li> <li>The dominance of conclusions over arguments is most pronounced when emotions are involved.</li> <li>An <em>affect heuristic</em> is when people let their likes and dislikes determine their beliefs about the world.</li> <li>While self-criticism is one of the functions of System 2, it is more of an apologist for than a critic of the emotions of System 1.</li> </ul> <h3>Part 2: Heuristics and Biases</h3> <h4>Ch 10: The Law of Small Numbers</h4> <ul> <li>System 1 is inept when faced with statistical facts, which change the probability of outcomes but do not cause them to happen.</li> <li>Extreme outcomes, both high and low, are most likely to be found in small than in large samples.</li> <li>Even statistical experts pay insufficient attention to sample size, and have poor intuitions of sampling effects.</li> <li>System 2 is capable of doubt, but sustaining doubt is harder work than sliding into certainty.</li> <li>System 1 runs ahead of the facts and constructs a rich image based on scraps of evidence, causing us to exaggerate the consistency and coherence of what we see.</li> <li>The associative machine seeks causes. But instead of focusing on how the event came to be, the statistical view relates it to what could have happened instead.</li> <li>We do not expect to see regularity produced by a random process. When we detect what appears to be a rule, we quickly reject the idea that the process is truly random.</li> <li>We pay more attention to the content of messages than to information about their reliability. Consequently we view the world in a simpler and more coherent way than the data justify.</li> </ul> <h4>Ch 11: Anchors</h4> <ul> <li>An <em>anchoring effect</em> occurs when people consider a particular value for an unknown quantity before estimating that quantity.</li> <li>Adjusting your estimate away from the anchor is an effortful activity. Insufficient adjustment, where we accept the anchor, is a sign of a weak or lazy System 2.</li> <li>Anchoring is also a priming effect, which selectively evokes compatible evidence. This is the automatic operation of System 1.</li> <li>The <em>anchoring index</em> is 100% for people who adopt the anchor as an estimate, and zero for people who are able to ignore the anchor altogether.</li> <li>Anchors that are obviously random can be just as effective as potentially informative anchors.</li> <li>When negotiating, don't make an outrageous counteroffer to an outrageous proposal, but make a scene and make it clear that you won't continue with that number on the table.</li> <li>To resist anchoring effects, search your memory for arguments against the anchor. This negates the biased recruitment of thoughts that produces these effects.</li> <li>System 2 is susceptible to the biasing effect of some anchors that makes some information easier to retrieve. It has control over or knowledge of the effect.</li> </ul> <h4>Ch 12: The Science of Availability</h4> <ul> <li>The <em>availability heuristic</em> replaces estimating the size of a category or frequency of an event with the ease with which instances come to mind.</li> <li>Salient events, dramatic events, and personal experiences versus experiences by others bias the ease with which instances come to mind.</li> <li>This explains why everyone in a group may feel as though he or she does more than his or her fair share.</li> <li>By asking people to provide more instances of a given behavior, you increase their struggle, and consequently they conclude that they don't adopt that behavior.</li> <li>Judgment is influenced more by the ease of retrieval than the number of instances retrieved. Increasing the number of requested instances therefore weakens judgment.</li> <li>When you provide a spurious reason for the difficulty of retrieving a large number of instances, judgment is again strengthened. The surprise is eliminated.</li> <li>People who are personally involved in the judgment are more likely to consider the number of instances and less likely to go by fluency.</li> <li>Fluency of instances is a System 1 heuristic, which is replaced by a focus on content when System 2 engages.</li> <li>Merely reminding people of a time when they had power increases their apparent trust in their own intuition.</li> </ul> <h4>Ch 13: Availability, Emotion, and Risk</h4> <ul> <li>Our expectations about the frequency of events are distorted by the prevalence and emotional intensity of the messages to which we are exposed.</li> <li>Jonathan Haidt said "the emotional tail wags the rational dog."</li> <li>An <em>availability cascade</em> is the self-sustaining chain of events through which a minor event leads to public panic and large-scale government action.</li> <li>Policy is about what people want and what is best for them. The availability cascade is the mechanism through which biases flow into policy.</li> <li>When dealing with small risks, we either ignore them or give them far too much weight, with no middle ground.</li> <li>Availability cascades may have a long-term benefit of calling attention to classes of risks and by increasing the risk-reduction budget.</li> </ul> <h4>Ch 14: Tom W's Speciality</h4> <ul> <li>The proportion of a particular class in a population is called the <em>base rate</em> of that class.</li> <li>How much an instance conforms to the stereotype of a particular class is called the <em>representativeness</em> of that instance.</li> <li>To determine how likely an instance belongs to a class, we ignore the base rate of the class and focus on the representativeness of the instance.</li> <li>Probability by representativeness is more accurate than chance guesses, but neglecting base rate information that points in another direction is a statistical sin.</li> <li>When endorsing representativeness, System 1 will automatically process the available information as if it were true, unless you decide immediately to reject it.</li> <li>Bayesian statistics govern how we should adjust the base rates given an account of representativeness.</li> </ul> <h4>Ch 15: Linda: Less is More</h4> <ul> <li>A <em>conjunction fallacy</em> is when we judge a conjunction of two events to be more probable than one of the events in a direct comparison.</li> <li>The most coherent stories are not the most probable, but they are plausible. And we confuse the notions of coherence, plausibility, and probability.</li> <li>Consequently, adding details to a scenario can make it more persuasive, but less likely to come true.</li> <li>When performing single evaluation instead of joint evaluation, the <em>less is more</em> principle evaluates a collection of items by its average and not its sum.</li> <li>The sum-like nature of a variable is less obvious for probability than something more enumerable like money.</li> <li>A question phrased as "how many" makes you think of individuals, while "what percentage" does not. This decreases the incidence of the conjunction fallacy.</li> </ul> <h4>Ch 16: Causes Trump Statistics</h4> <ul> <li><em>Statistical base rates</em> are facts about a population to which a case belongs, but they are not relevant to the individual case.</li> <li><em>Causal base rates</em> change your view of how the case came to be.</li> <li>We neglect statistical base rates, and we easily combine causal base rates with other case-specific information.</li> <li>Resistance to stereotyping is a laudable moral position, but neglecting valid stereotypes inevitably leads to suboptimal judgments.</li> <li>System 1 can deal with stores in which the elements are causally linked, but it is weak in statistical reasoning.</li> <li>Individuals feel relieved of responsibility when they know that others have heard the same request for help.</li> <li>When the outcome surprises us, we are unwilling to deduce the particular from the general, but are willing to infer the general from the particular.</li> <li>Consequently, we are more likely to learn something by finding surprises in our own behavior than by hearing surprising facts about people in general.</li> </ul> <h4>Ch 17: Regression to the Mean</h4> <ul> <li><em>Regression to the mean</em> is when poor performance is followed by improvement, and good performance is followed by deterioration.</li> <li>We tend to be nice to other people when they please us and nasty when they do not, we are statistically punished for being nice and rewarded for being nasty.</li> <li>The discrepancies between two trials does not need a causal explanation. Often luck explains why one is a significant outlier.</li> <li>The <em>correlation coefficient</em> between two measures is a measure of the relative weight of the factors they share.</li> <li>Whenever the correlation between two scores is imperfect, there will be regression to the mean.</li> <li>Causal explanations will be evoked when we detect regression, but they will be wrong because regression to the mean has an explanation but does not have a cause.</li> </ul> <h4>Ch 18: Taming Intuitive Predictions</h4> <ul> <li>We are capable of rejecting information as irrelevant and false, but adjusting for smaller weaknesses in the evidence is not something System 1 can do.</li> <li>If we are asked for a prediction but substitute an evaluation of the evidence, we generate biased predictions that completely ignore regression to the mean.</li> <li>To create an unbiased prediction, start with a baseline estimate and an estimate derived from the evidence, and choose the value between them that is proportional to your estimate of correlation.</li> <li>Such unbiased predictions make errors, but they are smaller and do not favor either higher or lower outcomes.</li> <li>Unbiased predictions permit predicting rare or extreme cases only when the information is very good, so you'll never have the satisfaction of calling an extreme case.</li> <li>Unbiased predictions are less preferred when error has varying types and severity, and extreme cases must be called correctly even if it accumulates error elsewhere.</li> <li>Your intuitions will deliver predictions that are too extreme and you will be inclined to put far too much faith into them.</li> </ul> <h3>Part 3: Overconfidence</h3> <h4>Ch 19: The Illusion of Understanding</h4> <ul> <li>We are always ready to interpret behavior as a manifestation of general propensities and personality traits, or causes that you readily match to effects.</li> <li>In a story, many of the important events that involve choices tempts us to exaggerate the role of skill and underestimate the part of luck.</li> <li>When you adopt a new view of the world, you immediately lose much of your ability to recall what you used to believe before your mind changed.</li> <li>This causes us to underestimate the extent to which we were surprise by past events, also known as <em>hindsight bias</em>.</li> <li>Hindsight bias leads us to asses the quality of a decision not by whether the process was sound but by whether its outcome was good or bad.</li> <li>The sense-making System 1 makes us see the world as more tidy, simple, predictable, and coherent than it really is. So we think that we can predict the future.</li> <li>The comparison of firms that have been more or less successful is to a significant extent a comparison between firms that have been more or less lucky.</li> </ul> <h4>Ch 20: The Illusion of Validity</h4> <ul> <li>Declarations of high confidence tell you that an individual has constructed a coherent story in his or her mind, not necessarily that the story is true.</li> <li>The persistence of individual differences in achievement is the measure by which we confirm the existence of skill in someone.</li> <li>Facts that challenge basic assumptions, and thereby threaten our livelihood and self-esteem, are simply not absorbed. The mind does not digest them.</li> <li>People can maintain an unshakeable faith, however absurd, when they are surrounded by a community of like-minded believers.</li> <li>A person who acquires more knowledge develops an enhanced illusion of skill and becomes unrealistically overconfident. They have many excuses ready when proven wrong.</li> <li>Errors in prediction are inevitable because the world is unpredictable. And high subjective confidence is not an indicator of accuracy.</li> </ul> <h4>Ch 21: Intuitions vs Formulas</h4> <ul> <li>Roughly 60% of studies have shown significantly better accuracy for algorithms in comparisons of clinical and statistical predictions.</li> <li>One reason experts may be inferior is that they try to be clever, think outside the box, and consider complex combinations of features. This actually reduces validity.</li> <li>Another reason is that when asked to evaluate the same information twice, we frequently give different answers. Unnoticed stimuli can substantially influence System 1.</li> <li>Assigning equal weights to all predictors is often superior than varying weights found by multiple regression, because it's not affected by accidents of sampling.</li> <li>Consequently, we can develop useful algorithms without any prior statistical research. And back of the envelope judgments are often good enough.</li> <li>The aversion to algorithms making decisions is rooted in the strong preference that many people have for the natural over the synthetic or artificial.</li> <li>Intuition adds value, but only after a disciplined collection of objective information and disciplined scoring of separate traits.</li> <li>When creating your own formula for an interview procedure, pick at most six dimensions, and develop a 1 to 5 scale for each one.</li> </ul> <h4>Ch 22: Expert Intuition: When Can We Trust It?</h4> <ul> <li>In the <em>recognition-primed decision</em> model, System 1 comes up with a plan. System 2 simulates it. If it works, it's implemented. Otherwise it's tweaked or discarded.</li> <li>Emotional learning may be quick, but expertise takes time to develop because it requires building a large collection of mini-skills.</li> <li>Confidence does not imply truth. The associative machine suppresses doubt and evokes ideas that are compatible with the current dominant story.</li> <li>Skilled intuitions come from environments where the environment is sufficiently regular to be predictable, and we can learn these regularities through prolonged practice.</li> <li>Human learning is normally efficient. If a strong predictive cue exists, human observers are likely to find it, given a sufficient opportunity to do so.</li> <li>Algorithms perform better in noisy environments because they detect weakly valid cues, and they use such cues consistently.</li> <li>The unrecognized limits of professional skill help explain why experts are often overconfident.</li> <li>Judgments that answer the wrong question can also be made with high confidence.</li> </ul> <h4>Ch 23: The Outside View</h4> <ul> <li>Confidentially collecting the judgment of each person in a group makes better use of the knowledge of its members.</li> <li>Our <em>inside view</em> judgment is overly optimistic, while the <em>outside view</em> judgment rightly adjusts a baseline prediction.</li> <li>We routinely discard statistical information, such as that offered by the outside view, when it's incompatible with personal impressions of a case.</li> <li>A <em>planning fallacy</em> is unrealistically close to a best-case scenario, and could be improved by consulting the statistics of similar cases.</li> <li>To counter the planning fallacy, <em>reference class forecasting</em> uses distributional information to create a baseline prediction. This adopts an outside view.</li> <li>People often, but not always, take on risky projects because they are overly optimistic about the odds they face.</li> </ul> <h4>Ch 24: The Engine of Capitalism</h4> <ul> <li>The people who have the greatest influence on the lives of others are likely to be optimistic and overconfident, and to take more risks than they realize.</li> <li>The optimistic risk taking of entrepreneurs contributes to the economic dynamism of society, even if most risk takers end up disappointed.</li> <li>We rate ourselves below average on any task we find difficult, and so we are overly optimistic about our standing on any activity we do moderately well.</li> <li>Entrepreneurs imagine a future where their actions determine the outcome of the firm, and not their competitors. This is because they know so little about these competitors.</li> <li>This <em>competition neglect</em> begets more competitors than the market can profitably sustain, and so the average outcome is a loss.</li> <li>A wide confidence interval is a confession of ignorance, which is not socially acceptable for someone who is paid to be knowledgeable. So we must be overconfident.</li> <li>Optimism is highly valued, both socially and in the market, and so we reward the providers of dangerously misleading information more than we reward truth tellers.</li> <li>Optimism contributes to resilience by defending one's self image, where we take credit for successes but little blame for failures.</li> <li>A <em>premortem</em> assumes that a year has passed, you implemented the plan as it exists, and the outcome was a disaster. It explains why.</li> <li>This counters overconfident optimism by escaping groupthink, and unleashing the imagination of knowledgeable individuals in a much needed direction.</li> </ul> <h3>Part 4: Choices</h3> <h4>Ch 25: Bernoulli's Errors</h4> <ul> <li>Choices between simple gambles provide a simple model that shares important features with the more complex decisions that researchers actually aim to understand.</li> <li>Expected utility theory defined axioms of rationality. Prospect theory examines why we deviate this theory under risk.</li> <li>A risk-averse decision maker will choose a sure thing that is less than the expected value of a gamble, paying a premium to avoid uncertainty.</li> <li>Bernoulli proposed that the psychological gamble isn't the weighted average of the monetary values, but the weighted average of the utility of these outcomes.</li> <li>Prospect theory states that happiness is determined by the recent change in wealth, and not its absolute value. It defines a reference point for happiness.</li> </ul> <h4>Ch 26: Prospect Theory</h4> <ul> <li>You know you have made a theoretical advance when you can no longer reconstruct why you failed for so long to see the obvious.</li> <li>Your personal wealth does not determine your attitudes to gains and losses. We like winning and dislike losing. And we dislike losing more than you like winning.</li> <li>In financial outcomes, the reference point can be the status quo, what you expect, or what you feel entitled to. Better outcomes are gains. Worse outcomes are losses.</li> <li>A principle of diminishing sensitivity applies to changes of wealth, both for gains and losses.</li> <li>We typically only accept a bet if its <em>loss aversion ratio</em>, or its ratio of gains to losses, is a minimum between 1.5 and 2.5.</li> <li>In mixed gambles, where both a gain and a loss are possible, loss aversion causes extremely risk-averse choices.</li> <li>In bad choices, where a sure loss is compared to a larger loss that is merely possible, diminishing sensitivity causes risk seeking.</li> <li>But prospect theory cannot deal with disappointment. It also cannot deal with regret, such as losing the gamble and foregoing the sure option.</li> </ul> <h4>Ch 27: The Endowment Effect</h4> <ul> <li>Indifference curves assume that your utility depends entirely on the present situation, and that the evaluation of a possible job does not depend on your current job.</li> <li>In labor negotiations and bargaining, the reference point and loss aversion is well understood. But their omission in most scenarios is <em>theory-induced blindness</em>.</li> <li>Loss aversions says the disadvantages of a change loom larger than its advantages. This introduces a bias that favors the status quo.</li> <li>Under the <em>endowment effect</em>, owning a good increases its value. Your minimum selling price greatly exceeds your maximum buying price, violating rational economic behavior.</li> <li>The endowment effect applies to goods "for use," or consumed or enjoyed. It doesn't apply to goods "for exchange," or traded for other goods.</li> <li>Loss aversion is built into the automatic evaluations of System 1. Consider the baby who holds on fiercely to a toy and shows agitation when it is taken away.</li> <li>Selling goods activates regions of the brain that are associated with disgust as pain. Buying activates these areas if the price is too high.</li> <li>Veteran traders are unaffected by the endowment effect. They ask "How much do I want to <em>have</em> this good, compared with other things I could have instead?"</li> <li>Being poor is living below one's reference point. Small amounts of money they receive are a reduced loss, not a gain. All their choices, then, are between losses.</li> </ul> <h4>Ch 28: Bad Events</h4> <ul> <li>Our brain prioritizes threats above opportunities. It can process hostile images that we can't consciously see, and can quickly pick hostile faces out from a crowd.</li> <li>Even symbolic threats evoke in attenuated form many reactions of the real thing, including fractional tendencies to avoid or approach, recoil or lean forward.</li> <li>Given a goal, not achieving it is a loss, while exceeding it is a gain. But the aversion to the failure of not reaching a goal is stronger than the desire to exceed it.</li> <li>Negotiations are difficult because losses by one side induce pain that exceeds the pleasure of the gains by the other side.</li> <li>An existing wage, price, or rent sets a reference point that is entitled. We think that exploiting market power to impose losses on others is unacceptable.</li> <li>A firm has its own entitlement, which is to retain its current profit. But we think it's not unfair for a firm to reduce its workers' wages when its profitability is falling.</li> <li>If a merchant lowers the price of a good, customers who bought at the higher price think of themselves as having sustained a loss that is more than appropriate.</li> <li>Punishing one stranger for behaving unfairly to another stranger activates the pleasure centers of the brain.</li> </ul> <h4>Ch 29: The Fourfold Pattern</h4> <ul> <li>When mapping decision weights to outcome probabilities, they are not equal in value, contrary to the <em>expectation principle</em>.</li> <li>We overweight unlikely events, which illustrates the <em>possibility effect</em>. We underweight highly likely events, which illustrates the <em>certainty effect</em>.</li> <li>We also have inadequate sensitivity to intermediate probabilities. The range of decision weights is much smaller than the range of probabilities.</li> <li>Paying a premium to eliminate a worry with certainty is compatible with the psychology of worry but not with the rational model.</li> <li>Between a sure loss and a gamble with a high probability of a larger loss, diminishing sensitivity makes the sure loss more aversive, and the certainty effect reduces the aversiveness of the gamble.</li> <li>This explains why people accept a high probability of making things worse for a small hope of avoiding a large loss, which can turn manageable failures into disasters.</li> <li>These same two factors enhance the attractiveness of the sure thing and reduce the attractiveness of the gamble when the out come is positive.</li> <li>Systematic deviations from expected value are costly in the long run. This rule applies to both risk aversion and to risk seeking.</li> </ul> <h4>Ch 30: Rare Events</h4> <ul> <li>Emotion and vividness influence fluency, availability, and judgments of probability, and thus account for our excessive response to the few rare events we don't ignore.</li> <li>People overestimate the probabilities of unlikely events, and overweight unlikely events in their decisions.</li> <li>If the event we are asked to estimate is very unlikely, we instead focus on its alternative. We focus on the odd, different, and unusual.</li> <li>A rich and vivid representation of the outcome, whether or not it is emotional, reduces the role of probability in the evaluation of an uncertain prospect.</li> <li>The <em>denominator effect</em> is when your attention is drawn to one outcome that "stands out," and you do not assess the alternative outcome with the same care.</li> <li>We weight low-probability events more when stated in terms of relative frequencies (how many) than when stated in more abstract terms of "chances," "risk," and "probability" (how likely).</li> <li>In <em>choice from experience</em>, instead of <em>choices from description</em>, we are exposed to variable outcomes from the same source, and so we do not overweight rare events.</li> </ul> <h4>Ch 31: Risk Policies</h4> <ul> <li>Every simple choice formulated as gains and losses can be deconstructed in innumerable ways into a combination of choices, yielding preferences that are likely to be inconsistent.</li> <li>If paying a premium for sure gains and to avoid a sure loss come out of the same pocket, the discrepant attitudes are unlikely to be optimal.</li> <li>We prefer <em>narrow framing</em>, or considering a sequence of simple decisions, even when we must entertain <em>broad framing</em>, or considering the decisions jointly.</li> <li>Narrow framing of gambles leads to loss aversion. Broad framing treats each gamble as one of many, blunting the emotional reaction to loss and increasing the tolerance of risk.</li> <li>Closely following daily fluctuations is a losing proposition, because the pain of small losses exceeds the pleasure of equally frequent small gains.</li> <li>A <em>risk policy</em> eliminates the pain of occasional loss by the thought that the policy that left you exposed to it will be advantageous to you over the long run.</li> <li>While an outside view protects you from the exaggerated optimism of the planning fallacy, a risk policy protects you from the exaggerated caution induced by loss aversion.</li> </ul> <h4>Ch 32: Keeping Score</h4> <ul> <li>The emotion that people attach to the state of our <em>mental accounts</em> are not acknowledged in standard economic theory.</li> <li>The <em>disposition effect</em> is the bias in finance to sell winners rather than losers, and is an instance of narrow framing.</li> <li>The <em>sunk-cost fallacy</em> invests additional resources in a losing account when better investments are available. It prefers an unfavorable gamble to a sure loss.</li> <li>Members of a board will replace a CEO with one who does not carry the same mental accounts and is therefore better able to ignore sunk costs of past involvements.</li> <li>A poignant story evokes more regret if it involves unusual events, because such events attract attention and are easier to undo in our imagination.</li> <li>People expect to have stronger emotional reactions to an outcome that is produced by action than to the same reaction when it is produced by inaction.</li> <li>When you deviate from the default, you can easily imagine the norm. If the default is associated with bad consequences, the discrepancy can be a source of painful emotions.</li> <li>You will be more loss averse in situations that are more important than money, and more reluctant to sell important endowments when it might lead to an awful outcome.</li> <li>To inoculate yourself against regret, remind yourself of its possibility and that things can go badly, and preclude any hindsight that might cause it.</li> </ul> <h4>Ch 33: Reversals</h4> <ul> <li>We normally experience situations in which contrasting alternatives are absent, and so moral intuitions that come to your mind in different scenarios are inconsistent.</li> <li>Preference reversal can occur if joint evaluation focuses attention on a situational aspect that is less salient than in single evaluation.</li> <li>The emotional reactions of System 1 likely determine single evaluation, while the comparison and careful evaluation required by joint evaluation calls for System 2.</li> <li>Judgments and preferences are coherent within categories but potentially incoherent when the objects are evaluated belonging to different categories.</li> <li>The <em>evaluability hypothesis</em> states that some attributes are only given weight in a joint evaluation because they are not evaluable on their own.</li> <li>Be wary of joint evaluation when someone who controls what you see has a vested interest in what you choose.</li> </ul> <h4>Ch 34: Frames and Reality</h4> <ul> <li><em>Losses</em> evoke stronger negative feelings than <em>costs</em> do, and so the cost of a lottery ticket that did not win is more acceptable than losing a gamble.</li> <li>"Rational" subjects, which are least susceptible to framing effects, showed enhanced activity in the frontal area of the brain that is implicated in combining emotion and reasoning.</li> <li>Reframing is effortful and System 2 is normally lazy. Most of us passively accept decisions as they are framed, and so our preferences are <em>frame-bound</em> and not <em>reality-bound</em>.</li> <li>Broader frames and inclusive accounts generally lead to more rational decisions.</li> <li>Opt-in versus opt-out is another framing effect. We check a box if we've already decided what we wish to do. But if unprepared for the question, laziness prefers the default.</li> </ul> <h3>Part 5: Two Selves</h3> <h4>Ch 35: Two Selves</h4> <ul> <li>The decision maker who pays different amounts to achieve the same gain or be spared the same loss is making a mistake.</li> <li>The <em>peak-end rule</em> states that a global retrospective rating is dominated by the highest score (the peak) and the final score (the end).</li> <li>The <em>duration neglect</em> states that the length of the evaluated activity has no effect whatsoever on its evaluation.</li> <li>The <em>experiencing self</em>, which evaluates each moment, does not have a voice. The <em>remembering self</em>, which keeps score and governs what we learn, makes our decisions.</li> <li>What we learn from the past is to maximize the qualities of our future memories, not necessarily our future experience.</li> <li>This is a feature of System 1, which represents sets by averages, norms, and prototypes. Not by sums.</li> <li>In some cases rats who can stimulate their brain by pressing a lever will die of starvation without taking a break to feed themselves.</li> <li>A memory that neglects duration will not serve our preference for long pleasures and short pains.</li> </ul> <h4>Ch 36: Life as a Story</h4> <ul> <li>Duration neglect is normal in a story. A story is about significant events and memorable events like its ending, not about time passing.</li> <li>The peak-end rule and duration effect also influence our evaluation of others' lives.</li> <li>Vacation pictures may be important to the remembering self. The photographer does not view the scene as a moment to be savored but as a future memory to be designed.</li> <li>We use the word "memorable" to describe vacation highlights, which explicitly reveals the goal of the experience.</li> </ul> <h4>Ch 37: Experienced Well-Being</h4> <ul> <li>A small fraction of the population endures most of the suffering, whether because of illness, an unhappy temperament, or misfortunes or tragedies.</li> <li>Our emotional state is largely determined by what we attend to, and we are normally focused on our current activity and immediate environment.</li> <li>Less commuting, greater availability of child care, and improved socializing opportunities can reduce the "unhappiness index" of a society.</li> <li>From Gallup data, some life aspects, such as education, is associated with a higher evaluation of one's life, but not with greater experienced well-being.</li> <li>Religion has favorable impact on both positive affect and stress reduction than on life evaluation, but provides no reduction of feelings of depression or worry.</li> <li>Being poor makes one miserable. A salary beyond $75,000 may enhance one's life satisfaction, but it does not improve experienced well-being.</li> <li>Higher income allows the purchase of many pleasures, but it also reduces the ability to enjoy the smaller things in life. Therefore the emotional experience is unchanged.</li> </ul> <h4>Ch 38: Thinking About Life</h4> <ul> <li><em>Affective forecasting</em> is the forecast of one's personal state in the future. An error happens when you think statistics don't apply to you.</li> <li>The score you assign to your life is determined by a small sample of highly available ideas, and not a careful weighing of the domains of your life.</li> <li>Both experienced temperament and life satisfaction are largely determined by the genetics of temperament.</li> <li>Setting goals that are especially difficult to attain leads to a dissatisfied adulthood. We must consider what people want in the concept of well-being.</li> <li>The <em>focusing illusion</em> states that nothing in life is as important as you think it is when you are thinking about it.</li> <li>This illusion can cause people to be wrong about their present state of well-being as well as the happiness of others, and about their own happiness in the future.</li> <li>The term <em>miswanting</em> describes bad choices that arise from the errors of affective forecasting. The focusing illusion is a rich source of it.</li> <li>The focusing illusion favors goods or experiences that are initially exciting but may lose their appeal. It appreciates less experiences that retain their attention value.</li> </ul> https://tedneward.github.io/Research/thinking/good-strategy-bad-strategy/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/good-strategy-bad-strategy/index.html <p><em>(by Richard Rumelt; ISBN )</em></p> <blockquote> <p>The most basic idea of strategy is the application of strength against weakness. Or, if you prefer, strength applied to the most promising opportunity.</p> <p>A Good Strategy identifies the one or two critical issues — and then focuses and concentrates action and resources on them. It doesn’t just draw on existing strength; it creates strength through the coherence of its design.</p> </blockquote> <h1>Introduction: Overwhelming obstacles</h1> <p>Core of strategy work: Discovering the critical factors in a situation and designing a way of coordination and focusing actions to deal with those factors.</p> <p>Good strategy honestly acknowledges the challenges being faces and provides an approach to overcoming them.</p> <p>Strategy selects the path, identifying how, why, and where determination and leadership are to be applied.</p> <p>Strategy must include concrete actions (“Implementation”).</p> <p>Goal setting != strategy.</p> <h1>Part 1: Good and bad strategy</h1> <p>Basic idea of strategy: apply strength to opportunity.</p> <p>A strategy is like a lever that magnifies force. Yes, you might be able to drag a giant block of rock across the ground with muscles, ropes and motivation. But it is wiser to build levers and wheels and then move the rock. Unlike a stand-alone decision or a goal, a strategy is a coherent set of analyses/concepts/policies/arguments/actions that respond to a high-stakes challenge.</p> <p>Sources of strength:<br> * Having a coherent strategy, one that creates strength.<br> * Subtle shifts in viewpoints.</p> <p>A leader identifies the one or two critical issues in the situation — and then focuses and concentrates action and resources on them.</p> <p>Good strategy almost always looks simple and obvious; doesn't require huge #s of Powerpoint slides to explain, and is not auto-generated from "strategic management" tools (matrix, chart, triangle, or fill-in-the-blanks).</p> <p><strong><em>A good strategy honestly acknowledges the challenges being faced and provides an approach to overcoming them. And the greater the challenge, the more a strategy focuses and coordinates efforts to achieve a powerful competitive punch or problem-solving effect.</em></strong></p> <p>End result should be a strategy that is aimed at channelling energy into what seem to be one or two of the most attractive opportunities, where it looks like you can make major inroads or breakthroughs.</p> <h2>Chapter 1 - Good strategy is unexpected</h2> <p>The first natural advantage of good strategy arises because other organisations often don’t have one. And because they don’t expect you to have one. ... Instead, they have multiple goals and initiatives to symbolize progress, but no coherent approach to accomplish that progress other than "spend more and try harder".</p> <p>Apple strategy: survive and wait for the next big thing. Succeed on that next thing.</p> <p>Good strategy requires leaders who are willing and able to say no to a wide variety of actions and interests. Strategy is at least as much about what an organisation does not do as it is about what it does.</p> <h2>Chapter 2 - Discovering power</h2> <p>When there is a successful company, there is always failing competition. Learn from both. What the competition was not able to copy?</p> <p>Wal-Mart: did not break conventional wisdom; it broke the definition of store.</p> <p>US Department of Defense: From pure military capabilities to asymmetric cost on opponent.</p> <h2>Chapter 3 - Bad strategy</h2> <p>Hallmarks of bad strategy:</p> <ul> <li>Fluff: Origin in […] and recently in the IT industry.</li> <li>Failure to face the challenge.</li> <li>Mistaking goals for strategy.</li> <li>Bad strategic objectives: <ul> <li>Fail to address critical issues.</li> <li>Impracticable.</li> </ul> </li> </ul> <p>True expertise is making a complex subject understandable.</p> <p>If a challenge is not defined, it is impossible to assess the quality of the strategy.</p> <p>Strategy work is episodic, not annual.</p> <p>One of the challenges of being a leader is mastering the shift from having others define your goals to being the architect of the organization’s purpose and objectives.</p> <p><strong>Goal</strong>: overall values and desires.<br> <strong>Objective</strong>: specific operational target.<br> <strong><em>Strategy transforms goals into coherent actionable objectives.</em></strong></p> <p>Good strategy focuses on one or very few pivotal objectives.</p> <p>Underperformance is not a challenge; is a result. The true challenges are the reasons for it.</p> <h2>Chapter 4 - Why so much bad strategy?</h2> <p>Bad strategy is the active avoidance of the hard work of crafting a good strategy.</p> <p>If you fail to identify and analyse the obstacles you don’t have a strategy. All you have is a stretch goal, a budget, or a list of things you wish would happen.</p> <h3>Some Forms of Bad Strategy</h3> <ul> <li> <p><strong>Dog’s Dinner Objectives</strong>: A long list of "things to do," often mislabeled as "strategies" or "objectives." These lists usually grow out of planning meetings in which stakeholders state what they would like to accomplish, then they throw these initiatives onto a long list called the "strategic plan" so that no one’s feelings get hurt, and they apply the label "long-term" so that none of them need be done today.</p> <p>A lot of companies conflate OKRs (Objectives and Key Results) with strategy. OKRs shouldn't replace strategy work. The process typically looks like this:<br> * once a year, each department head is asked to come up with their own departmental OKRs, which are supposed to be connected to company goals (increase revenue, decrease costs, etc.).<br> * Then each department breaks down their OKRs into sub-OKRs for their teams to carry out,<br> * which are then broken down into sub-sub-OKRs for sub-teams and/or specific people,<br> * so on down the chain.</p> <p>This process just perpetuates departmental silos and are rarely cohesive or mutually supportive of each other (if this does happen, it’s usually a happy accident). Department and team leaders often throw dependencies on other departments and teams, which causes extra work for teams that they often haven’t planned for and aren’t connected to their own OKRs, which drags down the efficiency and effectiveness of the entire organization. It’s easy for leaders to underestimate this drag since it’s hard to measure, and what isn’t measured isn’t managed.</p> <p><em>Setting objectives is not the same as creating a strategy to reach those goals</em>. You still need to do the hard strategy work of making a diagnosis of what obstacle is holding you back, creating a guiding policy for overcoming the obstacle, and breaking that down into coherent actions for the company to take (which shouldn’t be based on what departments or people or expertise you already have, but instead you should look at what competencies you need to carry out your strategy and then apply existing teams and people to carrying them out, if they exist, and hire where you’re missing expertise, and get rid of competencies that are no longer needed in the strategy). OKRs can be applied at the top layer as company goals to reach, then applied again to the coherent actions (i.e. what’s the objective of each action, and how will you know if you reached it?), and further broken down for teams and people as needed. You still need an actual strategy before you can set OKRs, but most companies conflate OKRs with strategy.</p> </li> <li> <p><strong>Blue Sky Objectives</strong>: A blue-sky objective is a simple restatement of the desired state of affairs or of the challenge. It skips over the annoying fact that no one has a clue as to how to get there.</p> <pre><code>For example, "underperformance" isn’t a challenge, it’s a result. It’s a restatement of a goal. The true challenge are the reasons for the underperformance. Unless leadership offers a theory of why things haven’t worked in the past (a.k.a. a diagnosis), or why the challenge is difficult, it is hard to generate good strategy. </code></pre> </li> <li> <p><strong>The Unwillingness or Inability to Choose</strong>: Any strategy that has universal buy-in signals the absence of choice. Because strategy focuses resources, energy, and attention on some objectives rather than others, a change in strategy will make some people worse off and there will be powerful forces opposed to almost any change in strategy (e.g. a department head who faces losing people, funding, headcount, support, etc., as a result of a change in strategy will most likely be opposed to the change). Therefore, strategy that has universal buy-in often indicates a leader who was unwilling to make a difficult choice as to the guiding policy and actions to take to overcome the obstacles.</p> </li> </ul> <p>This is true, but there are ways of mitigating this that he doesn’t discuss, which I talk about in the "Closing Thoughts" section below.</p> <ul> <li> <p><strong>Template-style "strategic planning"</strong>: Many strategies are developed by following a template of what a "strategy" should look like. Since strategy is somewhat nebulous, leaders are quick to adopt a template they can fill in since they have no other frame of reference for what goes into a strategy.</p> <ul> <li><strong>The Vision</strong>: Your unique vision of what the org will be like in the future. Often starts with "the best" or "the leading."</li> <li><strong>The Mission</strong>: High-sounding politically correct statement of the purpose of the org.</li> <li><strong>The Values</strong>: The company’s values. Make sure they are non-controversial.</li> <li><strong>The Strategies</strong>: Fill in some aspirations/goals but call them strategies.</li> </ul> <p>This template-style strategy skips over the hard work of identifying the key challenge to overcome, and setting out a guiding policy and actions to overcome the obstacle. It mistakes pious statements of the obvious as if they were decisive insights. The vision, mission, and goals are usually statements that no one would argue against, but that no one is inspired by, either.</p> </li> <li> <p><strong>New Thought</strong>: This is the belief that you only need to envision success to achieve it, and that thinking about failure will lead to failure. The problem with this belief is that strategy requires you to analyze the situation to understand the problem to be solved, as well as anticipating the actions/reactions of customers and competitors, which requires considering both positive and negative outcomes. Ignoring negative outcomes does not set you up for success or prepare you for the unthinkable to happen. It crowds out critical thinking.</p> </li> </ul> <h2>Chapter 5 - The Kernel of good strategy</h2> <p>Strategy is <em>designing a way to deal with a challenge</em>. A good strategy, therefore, <em>must identify the challenge to be overcome</em>, and <em>design a way to overcome it</em>. Good strategy has an essential logical structure called Kernel, that has:</p> <ul> <li> <p><strong>Diagnosis</strong>: Simplifies reality by identifying the critical aspects. Leaders must absorb a large part of a situation/problem’s complexity and ambiguity and pass on to the organization a simpler problem, one that is solvable. Answers: “What is going on here?”. At minimum:</p> <ul> <li>Names/classifies situation.</li> <li>Links facts into patterns.</li> <li>Highlights important issues.</li> </ul> <p>It defines a domain of action: If no action can be derived from it, the diagnosis is useless.</p> <p>If a challenge is ill-structured, the diagnosis has to be an educated guess.</p> <p>In business, most deep strategic changes are brought about by a change in diagnosis.</p> </li> <li> <p><strong>Guiding policy</strong>: Overall approach to cope or overcome the obstacles identified in the diagnosis. Drawn upon or creates sources of advantage. It channels action in certain direction without defining exactly what shall be done. Not goals or vision. In nonprofit and public policy, good strategy creates advantage by magnifying the effects of resources and actions.</p> <p>A good guiding policy itself can be a source of advantage by: Anticipating actions of others. Reducing complexity and ambiguity. Concentrating effort on critical aspects. Create coherent actions.</p> </li> <li> <p><strong>Coherent action</strong>: Non conflicting and coordinated. Resource commitments. Coordination by itself can be a source of advantage.</p> </li> </ul> <p>Strategy is visible as <strong><em>coordinated action imposed on a system</em></strong>. Imposed = "exercise in centralized power used to overcome the natural workings of a system".</p> <h1>Part 2: Sources of power</h1> <h2>Chapter 6 - Using leverage</h2> <p>Strategic leverage arises from a mixture of:<br> * anticipation: buyer demand, competition reactions. Anticipation does not require psychic powers. In many circumstances, anticipation simply means considering the habits, preferences and policies of others, as well as various inertias and constraints in change.<br> * pivot points: small adjustment magnifies effect of effort<br> * concentration: focus on a few objectives, due to limited resources, limited leadership cognition, perceived effectiveness (a 100% improvement in one department seems more effective than a 10% improvement in 100 departments)</p> <p>To achieve leverage, you must have insight into a pivot point that with magnify the effects of focused energy and resources. A pivot point magnifies the effect of effort. It is a natural or created imbalance in a situation, a place where a relatively small adjustment can unleash much larger pent-up forces.</p> <h2>Chapter 7 - Proximate objectives</h2> <p>Proximate objective: one that is close enough at hand to be feasible, guided by forecast of the future; the more uncertain the future, the more proximate objective must be.</p> <p>How proximate an objective is depends on the skills and accumulated resources of the organization.</p> <h2>Chapter 8 - Chain-link systems.</h2> <p>Strengthen the weakest link.</p> <p>Quality matters when quantity is an inadequate substitute.</p> <p>Find limiting factors.</p> <p>Quality matching: when each link is managed separately, the system can get stuck in a low-effectiveness state:<br> * Quality of the chain is equal to the lowest link.<br> * Improving on any other link is a waste.</p> <p>Problems with chain-linked:<br> * Identify bottlenecks.<br> * Incremental change may not pay off and even make things worse: Focus success measurement on change itself.</p> <p>Excellence in a chain-linked system is difficult to replicate.</p> <h2>Chapter 9 - Using Design</h2> <p>Strategy is not about choice/decision but about design, more constructed than chosen.</p> <p>In design, issue of mutual adjustment: Sharp gain or cost on getting the combinations right or wrong.</p> <p>Most of the work in system design is figuring out the interactions.</p> <p>A design-type strategy is an adroit configuration of resources and actions that yields an advantage in a challenging situation.</p> <p>Given a set bundle of resources, the greater the competitive challenge, the greater the need for the clever, tight integration of resources and actions.</p> <p>Given a set level of challenge, higher-quality resources lessen the need for the tight integration of resources and actions.</p> <p>Resources and tight coordination are partial substitutes for each other.</p> <p>Tight integration cost:<br> * Harder to create.<br> * Narrower focus.<br> * More fragile.<br> * Less flexible to change.</p> <p>Strategic resource is one that competitors cannot duplicate without suffering a net economic loss. High quality strategic resource yielding a powerful competitive advantage makes for great strategy simplicity (think useful patents). But it can impede innovation. Current profits are rarely associated with recent action, but with actions from the past.</p> <p>Success leads to laxity and bloat, and these lead to decline.</p> <h2>Chapter 10 - Focus</h2> <p>Coordination of policies that produces extra power through their interacting and overlapping effects<br> Demands application of that power to the right target<br> At the core, strategy is about focus, and most complex organisations don’t focus their resources. Instead, they pursue multiple goals at once, not concentrating enough resources to achieve a breakthrough in any of them.</p> <h2>Chapter 11 - Growth</h2> <p>Healthy growth is not engineered (ie through acquisition). It is the outcome of growing demand for special capabilities or of expanded or extended capabilities. It is the outcome of having superior products or skills.</p> <h2>Chapter 12 - Using advantage</h2> <p>An advantage is the result of differences – an asymmetry between rivals. Knowing your relative strengths and weaknesses, as well as the relative strengths and weaknesses of your competitors, can help you find an advantage. Strengths and weaknesses are "relative" because a strength you have in one context, or against one competitor, may be a weakness in another context, or against a different competitor. Press where you have advantage and side-step situations in which you do not. Exploit rivals’ weaknesses; avoid leading with your own.</p> <p>The most basic advantage is producing at a lower cost than your competitors, or delivering more perceived value than your competitors, or a mix of the two. The difficult part is sustaining an advantage. To do that, you need an "isolating mechanism" that prevents competitors from duplicating it. Isolating mechanisms include patents, reputations, commercial and social relationships, network effects, dramatic economies of scale, and tacit knowledge and skill gained through experience.</p> <p>Once you have an advantage, you should strengthen it by:<br> * <em>deepening it</em><br> * <em>broadening it</em><br> * <em>creating higher demand</em> for your products and services<br> * <em>strengthening your isolating mechanisms</em></p> <h2>Chapter 13 - Using dynamics</h2> <p>Dynamics are waves of change that roll through an industry. They are the net result of a myriad of shifts and advances in technology, cost, competition, politics, and buyer perceptions. Such waves of change are largely exogenous – that is, beyond the control of any one organization.</p> <p>Most industries, most of the time, are fairly stable. When change occurs, understand the forces underlying the main effect to find out the second order effects. These second effects are the ones to focus on.</p> <p>If you can see them coming, they are like an earthquake that creates new high ground and levels what had previously been high ground, leaving behind new sources of advantage for you to exploit. To discern a wave of change, you need to understand the gritty details well enough to question experts.</p> <p>There are 5 guideposts to look out for:</p> <ol> <li>Rising fixed costs</li> <li>Deregulation</li> <li>Predictable Biases <ul> <li>Ignore the fact that all business trends peak and then decline.</li> <li>Market leaders will duke it out for supremacy.</li> <li>The future winners will be the current apparent winners.</li> </ul> </li> <li>Incumbent Response</li> <li>Attractor States (i.e. where an industry "should" go).</li> </ol> <p>Attractor states are especially interesting because he defines it as where an industry "should" end up in the light of technological forces and the structure of demand. By "should," he means to emphasize an evolution in the direction of efficiency – meeting the needs and demands of buyers as efficiently as possible. They’re different from corporate visions because the attractor state is based on overall efficiency rather than a single company’s desire to capture most of the pie. Attractor states are what pundits and industry analysts write about. There’s no guarantee, however, that the attractor state will ever come to pass. As it relates to strategy, you can anticipate most players to chase the attractor state. This leads many companies to waste resources chasing the wrong vision, and faltering as a result (e.g. Cisco rode the wave of "dumb pipes" and "IP everywhere" that AT&amp;T and other telecom companies should have exploited). If you "zig" when other companies "zag", you can build yourself an advantage.</p> <p>As a strategist, you should seek to do your own analysis of where an industry is going, and create a strategy based on that (rather than what pundits "predict" will happen). Combining your own proprietary knowledge of your customers, technology, and capabilities with industry trends can give you deeper insights that analysts on the outside can’t see. Taking that a step further, you should also look for second-order effects as a result of industry dynamics. For example, the rise of the microprocessor was predicted by many, and largely came true. But what most people didn’t predict was the second-order effect that commoditized microprocessors getting embedded in more products led to increased demand for software, making the ability to write good software a competitive advantage.</p> <h2>Chapter 14 - Inertia and entropy</h2> <p>Entropy: weakly managed organizations tend to be less organized and focused.</p> <p>Inertia: Even when change programs at full speed, it can take years to alter a large company.</p> <ul> <li>Routine: If top management are convinced that new routines are essential, change can be quick.</li> <li>Culture: <ul> <li>To break: <ul> <li>Simplification: it will highlight obsolete units, waste, bad behaviour.</li> <li>Fragment the operation units.</li> <li>Triage units: close, repair or empower.</li> </ul> </li> <li>To change the culture, replace the alpha members.</li> </ul> </li> <li>By proxy: Abandon old profit streams.</li> </ul> <h2>Chapter 15 - Putting it together</h2> <p>(NVidia's strategy)</p> <p>Good strategy</p> <p>Intelligent anticipation</p> <p>Guiding policy that reduces complexity</p> <p>Power of design, focus, using advantage, riding a dynamic wave of change, and the important role played by inertia and disarray of rivals</p> <h1>Part 3 - Thinking like a strategist</h1> <p>Change in viewpoint is important when creating strategy.</p> <p>Most useful shift in viewpoint: thinking about your own thinking.</p> <h2>Chapter 16 - The science of strategy</h2> <p>A new strategy is an hypothesis and its implementation an experiment.</p> <p>Best knowledge to build good strategy is the one available only to your company.</p> <p>If new insights or ideas are not needed, deduction is sufficient.</p> <p>Strategy requires induction.</p> <p>In creating strategy, it is often important to take on the viewpoints of others, seeing how the situation looks to a rival or to a customer. Advice to do this is both often given and taken. Yet the advice skips over what is possibly he most useful shift in viewpoint: thinking about your own thinking.</p> <h2>Chapter 17 - Using your head</h2> <p>Make a short list of the most important and actionable. Things to do, not things to worry about.</p> <p>It is unnatural, even painful, to question your own ideas.</p> <p>Good strategy emphasize focus over compromise.</p> <p>Judgment can be practiced: Write down before every meeting what issues will arise and who will take which positions.</p> <h2>Chapter 18 - Keeping your head</h2> <p>Independent, not eccentric. Doubting without curmudgeon.</p> <p>Errors in judgment:<br> * Engineering overreach: when failure modes and consequences are not understood.<br> * Smooth-sailing fallacy: when lack of recent failures make people overconfident.<br> * Working under risk-seeking malincentives: You profit if things go well, others pay if things go bad.<br> * Social herding.<br> * Inside view: believe “this case is different” despite data.</p> <h3>Improve on strategy</h3> <ol> <li>Have tools to fight myopia and guide attention</li> <li>Question your own judgement</li> <li>Record judgements so you can learn from them</li> </ol> <h3>Tools to help thinking</h3> <p>Think about first principles — not what is done but by WHY it’s done<br> Create-Destroy — try hard to destroy current alternative</p> https://tedneward.github.io/Research/thinking/prag-thinking-learning/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/prag-thinking-learning/index.html <p><em>(by Andy Hunt)</em></p> <h3>Chapter 1: Introduction</h3> <p>Programming is all about problem solving. It requires creativity, ingenuity, and invention.</p> <p>Software isn't designed in an IDE or other tool. It’s imagined and created in our heads.</p> <p>Two of the most important modern skills are communication, and learning and thinking.</p> <h3>Chapter 2: Journey from Novice to Expert</h3> <ul> <li>Event theories can be measured, verified, and proven. Construct theories are intangible abstractions, cannot be proven, and are evaluated by usefulness.</li> <li>It's often difficult for experts to explain their actions to a fine level of detail; many of their responses are so well practiced that they become preconscious actions.</li> <li>Novices need clear, context-free rules by which they can operate, just as the expert would be rendered ineffective if he were constrained to operate under those same rules.</li> <li>When becoming an expert, you don't just know more, but you experience fundamental differences in how you perceive the world, how you approach problem solving, and the mental models you form and use.</li> <li>At stage 1 of the Dreyfus model, novices can be somewhat effective by following context-free rules, but are vulnerable to confusion when mistakes happen or things go awry.</li> <li>At stage 2, advanced beginners try things on their own, take advice, and want information fast, but still have problems troubleshooting and don't see a big picture.</li> <li>At stage 3, competent practitioners can develop and use mental models, troubleshoot problems on their own, and find and apply advice from experts.</li> <li>At stage 4, proficient practitioners can correct previous poor task performance, learn from the experience of others, and know what likely happens next or what can break.</li> <li>At stage 5, experts are primary sources of knowledge, continually look for better methods, and work from intuition instead of reason.</li> <li>Novices following rules by experts thrive, while experts following rules by other experts suffer. Their tool is intuition.</li> <li>Three most important changes are moving from rules to intuition, equal weighting of parts to distinguishing important and unimportant ones, and detached observer to involved influencer.</li> <li>Most people are advanced beginners who overstate their abilities (the Dunning-Kruger effect) because metacognitive abilities tend to only appear at higher skill levels.</li> <li>A competent practitioner might be better suited to teach a novice than an expert, who are often unable to articulate why they reached a particular decision.</li> <li>Deliberate practice requires tackling a well-defined and appropriately difficult task, in an environment that provides feedback and provides opportunities for repetition and correction of errors.</li> <li>To help promote the skills of advanced beginners to competent practitioners, have good exemplars in the environment; they will be watched and imitated.</li> <li>Trumpeter Clark Terry told students the secret to learning music was to imitate, assimilate, and then innovate.</li> <li>Tools and models are useful in the right environments, but none are a panacea, and their misapplication has probably done more damage than good.</li> <li>Avoid formal rules if you need creativity, intuition, or inventiveness.</li> <li>Context matters, as the expert uses context-dependent intuition, but the lower stages of the Dreyfus model aren't skilled enough to use it.</li> </ul> <h3>Chapter 3: This Is Your Brain</h3> <ul> <li>The R-mode of your brain is critical for intuition, problem solving, and creativity. The L-mode gives you the power to work through the details and make it happen.</li> <li>Memory is actively maintained by an executing loop. Like dynamic RAM, your brain needs constant refreshing, or its memories fade.</li> <li>Like describing a dream, R-mode yields many results that can't be verbalized. Additionally, it's like peripheral vision, outside your direct control.</li> <li>R-mode is unpredictable, so you need to be ready to capture any insight or idea at any time.</li> <li>Once you start keeping track of ideas, you'll get more of them. Everyone has good ideas to capture.</li> <li>L-mode gives you verbal, analytic, symbolic, abstract, temporal, rational, digital, logical, and linear thinking abilities.</li> <li>R-mode is non-verbal, non-rational, synthetic to compose and form wholes, spatial, concrete, intuitive, analogic, and holistic to recognize patterns and structures. This is where intuition lies.</li> <li>Building things you can learn from, like prototypes and unit tests, is R-mode synthesis; inspecting and dissecting is L-mode analysis.</li> <li>When you are fearful or angry, filled with negative emotions, your body shuts down extra resources in preparation for for fight or flight.</li> <li>Creativity comes from the selection of the right components in the right presentation to create the work. And selection comes from pattern matching.</li> <li>R-mode helps you find global, holistic patterns. L-mode lets you analyze parts and look into the detail.</li> <li>Believing in the plasticity of your brain, or the amount you can learn and number of skills you can attain, increases those abilities.</li> </ul> <h3>Chapter 4: Get in Your Right Mind</h3> <ul> <li>Activate more neural pathways than usual to involve more of your brain. For example, fiddle with a tactile puzzle while pondering a tricky problem.</li> <li>Allow several senses to interact, like writing a problem down, drawing it, describing it verbally, engaging in open discussion, and act out roles.</li> <li>Listening to music, drawing, meditation, jogging, needlework, rock climbing, and so on can engage the R-mode that shuts off the L-mode.</li> <li>Once you experience the shift to R-mode, you'll better know what R-mode processing feels like, and it will become easier over time.</li> <li>You want to let the R-mode lead and then switch to the L-mode to "productize" it.</li> <li>Working together is an effective way to discover helpful and interesting abstractions. In pair programming, the driver is in L-mode, while the navigator is in R-mode.</li> <li>A strong metaphor comes from suddenly switching from one frame of reference to a different, unexpected one. That junction between the two frames is the bisociation.</li> <li>The farther away the ideas are, the harder it is to join them in a metaphor that works. But such imprinting from one frame to another is very powerful.</li> <li>Programmers are sloppy in their use of metaphors. For example, using <code>insert()</code> instead of <code>add()</code> on a container, tables that aren't like tables, threads that aren't like threads.</li> <li>Humor comes from drawing or extending relationships beyond the norm, which also helps build stronger metaphors.</li> <li>Your brain stores every input it receives, although does not necessarily index them. Your R-mode asynchronously scans these memories, albeit inefficiently.</li> <li>Image streaming harvests R-mode imagery: Pose a question, then close your eyes, and describe out loud every image that crosses your mind. Imagine it using all five senses.</li> <li>With the morning pages technique, write three uncensored pages long hand before you "wake up." This gives you an unguarded, unconscious R-mode brain dump.</li> <li>Insight can come at odd times like taking a shower or mowing the lawn because the L-mode gets bored and tunes out, leaving the R-mode to present its findings.</li> <li>As Poincaré did, write down everything you know about a problem, and then go on a walk without focusing on it, allowing the R-mode to dominate.</li> <li>Making source code human-readable means making the larger patterns in the code easier to see.</li> <li>Use consistent typographic cues to aid visual perception. Conflicting styles are contentious because they represent conflicting patterns.</li> <li>Change your viewpoint on a problem by viewing it in reverse, exaggerating an idea, combining disparate ideas, and non-goal-directed playing with ideas.</li> <li>Even small changes in your routine, like where you park, where you part your hair, when you eat, and what days you shave help prevent neural ruts.</li> </ul> <h3>Chapter 5: Debug Your Mind</h3> <ul> <li>Cognitive biases are mental "bugs" that affect your decision making, memory, perception, rationality, and so on. There are a lot of them.</li> <li>Symbolic reduction, or reducing large, complex systems to simple, easily manipulated symbols, is essential to computer programming and knowledge-based work.</li> <li>Humans are bad at trying to extrapolate future events from previous ones. We assume a stable, linear progression, with easily defined cause and effect.</li> <li>Don't discount unobserved or rare phenomena as impossible. And homogeneity and randomness are different things.</li> <li>On an exploratory or inventive project, you learn as you go, and so you know most at the end. So defer closure in order to make better decisions later.</li> <li>Agile development embraces uncertainty; it is not a bad thing. Chip away at the details, and don't nail them down until ready.</li> <li>Write things down. As a Chinese proverb says, the palest ink is better than the best memory.</li> <li>Attitudes on risk, individualism vs teamwork, stability vs freedom, and family vs work are affected by your cohort and your parents.</li> <li>Generation X form the greatest entrepreneurial generation in U.S. history. They're fiercely individualistic, and resist being labeled at all costs.</li> <li>The four generational archetypes are the prophet, with vision, values; the nomad, with liberty, survival, honor; the hero, with community, affluence; and the artist, with pluralism, expertise, due process.</li> <li>With Howe/Strauss generational archetypes, the hero begets the artist, who begets the prophet, who begets nomad, who begets the hero.</li> <li>The best way to keep from falling victim to your generation’s particular set of biases is to embrace diversity.</li> <li>Within the MBTI framework, sensing (S) vs intuition (N) is probably the largest source of miscommunication and misunderstanding. 75 percent of folks are sening.</li> <li>Work out a compromise. Do not try to change the other person’s temperament to match your own.</li> <li>Our primeval characteristics include immediate fight or flight, wanting everything now, being dominant, defending territory, crying foul, and considering anyone different evil.</li> <li>Attitudes, beliefs, behaviors, and emotions are all contagious.</li> <li>It's easy to write a short and snippy e-mail because it doesn't give the neocortex time to intervene, unlike writing a longhand letter.</li> <li>Lead with intuition, but follow up with provable, linear feedback. Your intuition may be wrong.</li> <li>Expectations color reality. So ask hard questions about a problem, measure all you can, get the opinion of others, and define its opposite to draw a more critical eye.</li> </ul> <h3>Chapter 6: Learn Deliberately</h3> <ul> <li>Learning isn't done to you, it's something you do. "Education" comes from "educare," which literally means "led out." And knowledge alone, without experience, isn't effective.</li> <li>To develop competence and expertise, you need continuing goals, feedback to understand your progress, and a deliberate approach to the whole thing.</li> <li>An objective is something you do to get you closer to a goal. It should be specific, measurable, achievable, relevant, and time-boxed. Together, SMART objectives.</li> <li>By being deliberate about your learning, by allocating appropriate time, and by using that time wisely, you can be much more efficient in your learning.</li> <li>All knowledge investments have some value. Even if you never use a particular technology vocationally, it will impact the way you think and solve problems.</li> <li>Not all learning sessions will be equally productive, but by scheduling them regularly, you will win out in the long run.</li> <li>Visual learners need to see the material, auditory learners need to hear the material, and kinesthetic learners need to physically experience the material.</li> <li>Howard Gardner defined seven facets of intelligence: kinesthetic, linguistic, logical/mathematical, visual/spatial, musical, interpersonal, and intrapersonal.</li> <li>The team that studies together learns together, teaches each other, and learns more effectively.</li> <li>Written instructions are considered the least efficient because many parts of the brain and body you want to train or educate aren't the parts that process language.</li> <li>After surveying the work, forming questions, and then reading, <em>recite</em> the information by playing with it, exercising your R-mode. Finally, review it.</li> <li>Reading the same material over and over doesn't help you remember it. Instead, test yourself by trying to recall the material over and over.</li> <li>Redrawing a mind map from memory helps strengthen the connections and may expose additional insights in the process.</li> <li>For exploratory mind maps, use words as titles, draw icons to represent key ideas, and make important lines thick. Do it quickly, giving the R-mode unfettered access.</li> <li>For a collaborative mind map, write ideas on stick notes, cluster related notes together, and draw circles around each group and connect them with lines.</li> <li>Documenting makes you turn your attention inward, as you would when working with a mind map, promoting insight later in the project.</li> <li>Teaching others clarifies your own understanding of something and reveals many of your underlying assumptions.</li> </ul> <h3>Chapter 7: Gain Experience</h3> <ul> <li>Non-goal-directed exploring, or playing with a problem, doesn't make it any easier, but it gets us closer to how we're wired to learn.</li> <li>Working with new material or solving a problem in a playful manner makes it more enjoyable and also easier to learn.</li> <li>When learning from similarities, clinging to them thinking may prevent us from fully embracing a new skill. And our notion of "similar" may be completely wrong.</li> <li>"I don't know" is a fine answer, but don't let it end there. And it's not important to get something right the first time; it's important to get it right the last time.</li> <li>Version control allows backtracking; unit testing measures progress; automation ensures repetition. Together, they give you the ability to experiment with little risk.</li> <li>Inner game stresses learning from your own experience. It's difficult to learn something by words; we learn something better by discovery, not instruction.</li> <li>When playing the inner game, just be aware. Don't try to get it right, but notice when it's wrong, and act to correct it.</li> <li>When debugging, be fully aware of how the system is behaving, and then decide what part is wrong before moving on to devise a solution.</li> <li>Not only does your creativity suffer with a deadline looming, but it suffers for two days after the deadline. So end a project iteration on a Friday.</li> <li>Once you create an environment where it's acceptable to fail, the pressure is off, and you can be attentive, comfortable, and observant. See unit testing and prototyping.</li> <li>Much of perception is based on prediction, which is based on context and past experience, so much that current, real-time input takes a backseat.</li> <li>You can imagine experiences and learn from as effectively as if you had lived them for real, because your brain has difficulty telling the difference.</li> <li>Intuition grows by having more patterns to draw on and apply, and a growing body of tacit knowledge of what to look for and when.</li> </ul> <h3>Chapter 8: Manage Focus</h3> <ul> <li>Information is raw data in a given context, while knowledge imparts meaning to that information, from the application of time, attention, or skill.</li> <li>Much of meditation, yoga, and similar practices offer some relief from the idle chatter of your L-mode, to live in the moment, and to not divide your mental energy unnecessarily.</li> <li>With Vipassana meditation, explicitly focus on your breathing. Words can come, but just let them go, and return attention to your breath.</li> <li>With the segmented breath controlled breathing approach, on the inhale, fill the lower belly first, then the chest, and then finally up to the collar bones.</li> <li>Handing over work to the unconscious only works if you have some data to fill up on.</li> <li>Tools you use for mental support outside your brain become part of your operating mind. They help form your exocortex.</li> <li>Whether a wiki or a shoebox, once you have a place or page to put some type of thought, you'll get more thoughts of that type.</li> <li>Transcribing notes to the wiki or cleaning them up on the wiki helps get your head around the material.</li> <li>Multitasking can cost you 20 to 50 percent of your productivity, while after an interruption it make 20 minutes to pick up where you left off.</li> <li>GTD suggests scanning the input queue only once, processing each pile of work in order, and not keeping lists in your head.</li> <li>Set up times of the day when you aren't to be interrupted, and other times for collaboration, daily stand-up meetings, and other assorted contingencies.</li> <li>To control the flood of email, remember that you'll receive less email if you send less email, and that your reply speed sets the tempo for a conversation.</li> <li>To try and keep context, raise the physical cost of entry and exit to remind you of the mental cost.</li> <li>In the few seconds before an interruption takes hold, leave some breadcrumbs or cues that you can pick up on when you get back to resuming the task.</li> <li>You can instantly recognize productivity gains of 20 to 30 percent with a second monitor. Get one of the same size and brand to minimize distractions.</li> <li>Organize virtual desktops by task, such as communications, writing, coding, surfing, and music.</li> </ul> <h3>Chapter 9: Beyond Expertise</h3> <ul> <li>To manage effective change, remember: Start with a plan, fear inaction and not error, give it a fair chance, believe that change is possible, and take small and achievable steps.</li> <li>The professional kiss of death for the expert is to act like one. Your own judgment and views, instead of supporting you, can imprison you.</li> <li>Leonardo Da Vinci said: "People look without seeing, hear without listening, eat without awareness of taste, touch without feeling, and talk without thinking."</li> </ul> https://tedneward.github.io/Research/thinking/elephant-in-the-brain/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/elephant-in-the-brain/index.html <p><em>(by Kevin Simler, Robin Hanson)</em></p> <h1>Introduction</h1> <h4>First example: health care</h4> <p>Why do patients spend so much on medical care? To get healthier: That's their one and only goal, right? Maybe not. Consider data points:</p> <ul> <li>People in developed countries consume way too much medicine, well beyond what's useful for staying healthy. <em>(I'd be curious to know which studies... citation?)</em></li> <li>Large randomized studies, for example, find that people given free healthcare consume way too much medicine, yet don't end up noticably healthier. <em>(Again, studies/citations?)</em></li> <li>Meanwhile, <em>non-medical</em> interventions--such as efforts to alleviate stress or improve diet, exercise, sleep or air quality--have a much more apparent effect on health, and yet patients and policymakers are far less eager to pursue them.</li> <li>Patients are also easily satisfied with the <em>apperance</em> of good medical care, and show shockingly little interest in digging beneath the surface (by getting second opinions or asking for outcome statistics from their doctors or hospitals).</li> <li>People spend exorbitantly on heroic end-of-life care even though cheap, palliative care is usually just as effective at prolonging life and usually better and preserving quality of life.</li> </ul> <p>Altogether, these puzzles cast considerable doubt on the simple idea that medicine is strictly about health.</p> <p>People might have other motives for buying medicine (beyond getting healthy) and that these motives are largely unconscious. Medicine isn't just about health--it's also an exercise in <em>conspicuous caring</em>.</p> <p><strong>We're suggesting that key human behaviors are often driven by multiple motives; secondly, we're suggesting that some of these motives are unconscious; and these are elephant-sized motives large enough to leave footprints in national economic data.</strong></p> <h4>Second example: startup</h4> <p>Inspired by <em>Hierarchy in the Forest</em> (Christopher Boehm): parallels between chimpanzee societies and startup/business societies.</p> <p>People don't typically think or talk in terms of maximizing social status or showing conspicuous care. And yet we all instinctively act this way. In fact, we're able to act quite skillfully and strategically, pursuing our self-interest without explicitly acknowledging it, even to ourselves.</p> <h4>The Core Idea</h4> <p><strong>We, human beings, are a species that's not only capable of acting on hidden motives--we're designed to do it.</strong> Our brains are built to act in our self-interest while at the same time trying hard not to appear selfish in front of other people. And in order to throw them off the trail, our brains often keep "us", our conscious minds, in the dark. The less we know of our own ugly motives, the easier it is to hide them from others.</p> <p><strong>Self-deception is therefore <em>strategic</em>, a ploy our brains use to look good while behaving badly.</strong> As long as we continue to tiptoe around it, we'll be unable to think clearly about human behavior.</p> <blockquote> <h4>Box 1: "The Elephant"</h4> <p>The elephant in the brain, in a word, is selfishness--the selfish parts of our psyches. But it's also broader than that. Selfishness is just the heart, if you will, and an elephant has many other parts, all interconnected. "The elephant", therefore, refers not just to human selfishness, but to a whole cluster of related concepts: competitiveness fighting for power, status, and sex; our willingness to lie and cheat to get ahead; that we hide some of our motives and that we do so to mislead others.</p> </blockquote> <h4>The Basic Argument</h4> <p>At least four strands of research all lead to the same conclusion: that we are "strangers to ourselves":</p> <ol> <li> <p><em>Microsociology.</em> The depth and complexity of our social behaviors (laughing, blushing, tears, eye contact, body language)--our brains choreograph these interactions on our behalf, and with surprising skill.</p> </li> <li> <p><em>Cognitive and social psychology.</em> Our brains aren't just hapless and quirky--they're <em>devious</em>. "At every stage [of processing information]--from its biased arrival, to its biased encoding, to organizing it around false logic, to misremembering and then misrepresenting it to others--the mind continually acts to distort information flow in favor of the usual goal of appearing better than one really is." The <em>introspection illusion</em>, the fact that we don't know our own minds nearly as well as we pretend to.</p> </li> <li> <p><em>Primatology.</em> Humans are primates, specifically apes. When we study primate groups, we notice a lot of Machiavellian behavior; when asked to describe our own behavior, we mostly portray our motives as cooperative and prosocial.</p> </li> <li> <p><em>Economic puzzles.</em> When we study specific social institutions--medicine, education, etc--we notice they frequently fall short of their stated goals. In many cases, this is due to simple execution failures. But in other cases, the institutions behave as though they were designed to achieve other, unacknowledged goals.</p> </li> </ol> <p>We act on hidden motives together, in public, just as often as we do by ourselves, in private. And when enough of our hidden motives harmonize, we end up constructing stable, long-lived institutions that are designed, at least partially, to accommodate such motives.</p> <blockquote> <h4>Box 2: Our Thesis in Plain English</h4> <ol> <li>People are judging us all the time. One of the important things they're judging is our <em>motives</em>.</li> <li>Because others are judging us, we're eager to look good. So we emphasize our pretty motives and downplay our ugly ones.</li> <li>This applies not just to our words, but also to our thoughts. Why can't we be honest with ourselves? The answer is that our thoughts aren't as private as we imagine; in many ways, conscious thought is a rehearsal of what we're ready to say to others.</li> <li>In some areas of life (politics), we're quick to point out when others' motives are more selfish than they claim, But in other areas (medicine), we prefer to believe that almost all of us have pretty motives.</li> </ol> </blockquote> <h1>Part I: Why We Hide Our Motives</h1> <h2>Chapter 1: Animal Behavior</h2> <p>Let's start at a simpler beginning: two animal behaviors that are hard to decipher. In each case, the animals appear to be doing something simple and straightforward, but as we dig beneath the surface, we'll find extra layers of complexity.</p> <h3>Social Grooming</h3> <p>To keep their fur clean, primates enlist the aid of others to groom their backs, faces, and heads: <em>social grooming</em>.</p> <p>At first blush, social grooming looks like an act of hygiene, a way to keep one's fur clean. But we can't take it at face value:</p> <ul> <li> <p>Most primates spend far more time grooming each other than necessary to keep fur clean. (Gelada baboons spend 17% of their daylight hours; other primates spend 0.1%, and birds 0.01%.)</p> </li> <li> <p>Primates spend a lot more time grooming each other than they spend grooming themselves.</p> </li> <li> <p>We can correlate the average body size with the amount of time they spend grooming--if grooming were strictly a hygienic activity, we'd expect larger species (with more fur) to spend more time grooming each other. But in fact there's no correlation.</p> </li> </ul> <p>Primatologist Robin Dunbar suggests social grooming isn't about hygiene--it's also about politics. By grooming each other, primates help forge alliances that help them in other situations. "Grooming creates a platform off which trust can be built." This explains why higher-ranked individuals receive more grooming than lower-ranked individuals. It also explains why grooming time across species is correlated with the size of the social group, but not the amount of fur.</p> <h3>Competitive Altruism</h3> <p>The Arabian babbler (bird) lives in small groups of 3 to 20 members who collectively defend a small territory of trees, shrubs, and bushes that provide much-needed cover from predators. Babblers who live as part of a group do well; those who are kicked out of a group are in great danger. Male babblers arrange themselves into rigid dominance hierarchies. At first, these activities seem straightforwardly altruistic, but closer inspection suggests otherwise.</p> <ul> <li>Babblers compete to help each other and the group, often aggressively so. If the goal of these behaviors is to be helpful, why waste effort competing to perform them? <ul> <li>One theory is that higher-ranked babblers are stronger and better able to forego food and fight off predators.</li> <li>The problem: babblers compete primarily with the birds immediately above or below them in the hierarchy (#1 v #2, never #1 v #10).</li> <li>Babblers often intefere in the helpful behaviors of their rivals, which makes no sense if the goal is to benefit the group as a whole.</li> </ul> </li> <li>The answer: altruistic babblers develop a kind of "credit" among their groupmates: <em>prestige status</em>. This earns them two perks: <ul> <li>Mating opportunities--the alpha will allow a high-prestige beta to mate with some of the females, "bribing" the beta to stick around.</li> <li>Reduced risk of ostracization.</li> </ul> </li> </ul> <h3>Human Behaviors</h3> <p>To find out why we misconstrue animal motives, including our own, we have to look more carefully at how our brains were designed and what problems they're intended to solve.</p> <h2>Chapter 2: Competition</h2> <h2>Chapter 3: Norms</h2> <h2>Chapter 4: Cheating</h2> <h2>Chapter 5: Self-Deception</h2> <h2>Chapter 6: Counterfeit Reasons</h2> <h1>Part II: Hidden Motives in Everyday Life</h1> <h2>Chapter 7: Body Language</h2> <h2>Chapter 8: Laughter</h2> <h2>Chapter 9: Conversation</h2> <h2>Chapter 10: Consumption</h2> <h2>Chapter 11: Art</h2> <h2>Chapter 12: Charity</h2> <h2>Chapter 13: Education</h2> <h2>Chapter 14: Medicine</h2> <h2>Chapter 15: Religion</h2> <h2>Chapter 16: Politics</h2> <h2>Chapter 17: Conclusion</h2> https://tedneward.github.io/Research/thinking/inverted-computer-culture/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/inverted-computer-culture/index.html <p><em>(from <a href="http://viznut.fi/texts-en/inverted_computer_culture.html">http://viznut.fi/texts-en/inverted_computer_culture.html</a>)</em></p> <p>Imagine a world where computers are inherently old. Whatever you do with them is automatically seen as practice of an ancient and unchanging tradition. Even though new discoveries do happen, they cannot dispel the aura of oldness.</p> <p>Most computers are located in ancient buildings, "computer temples", situated in tranquil natural environments. They stand unchanged over centuries, like mountains, no matter what happens in the world around them.</p> <p>The idea of using computers in urban everyday life or for commercial purposes is something that doesn't even occur to people's minds – that would be just as absurd and wild as the idea of cleaning one's home with a 100-year-old tortoise.</p> <p>Mundane mechanical technology has limited lifespans, a couple of decades at most. Solid-state computer components, on the other hand, have no mechanical decay, so they are practically eternal. An average microchip is hundreds of years old and has thus seen many different uses in many different eras. It is difficult to even look at a computer without admiring the long history it has gathered within itself. Computers thus feel more rooted to the world than any other human creation – despite the fact that they have no practical use in the society.</p> <p>Computing is something stereotypically practiced by old women. Someone who is interested in computers is probably also interested in crocheting. Men and youngsters who visit computer temples are considered oddballs by their peers; young people in particular tend to see computers as something profoundly anti-cool. Movements that try to rid computing of its "grandmotherly bias" are usually seen as silly and embarrassing.</p> <p>It is commonly thought to be futile to even try to make youngsters interested in computers – they simply don't yet have the required patience or concentration. There's no addictivity or instant gratification, nothing flashy or punk that fascinates the young mind. The appreciation and understanding of computers is something that develops slowly over years, often via gateway interests such as meditation, cultural history or pure mathematics. The kind of people who want to settle in a monastery and dedicate their lives to science or art may also develop an interest in computing.</p> <p>The psychological effects of human-computer interaction are considered similar to those of meditation: concentration, awareness, mental clarity. Computers are also seen as amplifiers of wisdom and intelligence. However, it may take years for a user to attain these effects, so those who are after self-improvement are more likely to adopt more straightforward practices.</p> <p>It is considered essential to be in a properly alert and rested state of mind when using a computer. Even to seasoned users, every session is special, and the purpose of the session must be clear in mind before sitting down. The outer world is often hurried and flashy, but computers provide a "sacred space" for relaxing, slowing down and concentrating on a specific idea without distractions. Breathing slows down, thought processes become more wholesome and contemplative. People who have spent a lot of time in a computer temple often find it difficult to readjust to the fast pace, distractivity and intellectual one-dimensionality of the non-computer world.</p> <p>Computers are seldom privately owned – they are considered essentially communal rather than personal – but their usage patterns are often highly individualistic. Programming is the most essential element of all computer use, and it is not uncommon to find users who have created all of their software from scratch. Centuries of computer science research has valued simplicity, elegance, smallness and clarity over mundane values such as efficiency or scalability, so there are many beautiful ideas to base one's individual programs on.</p> <p>A small percentage of computer use consists of long-distance human communication. The topics of non-technical discussion are often big and timeless and seldom concern daily news events. Heated arguments and polarization are considerably less common than in the non-computer "world of illusions".</p> <p>There are computer games, but they are never framed as "entertainment". They are more often seen as learning experiences that help see the world from different viewpoints. Children would find most of the games utterly boring, if someone were crazy enough to try to get them interested.</p> <p>Common people have all kinds of weird misconceptions about computing. They often see computers as far more spiritual or even divine than they actually are, so computer people often need to bring them down to earth from their fantasies. Even though computers often do help improve minds, they do not turn people into omniscient gods or saints or grant them eternal lives. And the computers themselves aren't omniscient either, and cannot manipulate the fabric of reality.</p> <hr> <p>The purpose of this thought experiment was to invert the current mainstream cultural aspects of computing. It is not intended to be a utopian fantasy, even though some of it does align with what I have written about permacomputing.</p> <p>In the real world, people associate computers with many different things: corporate dehumanization, overwhelming consumer capitalism, alienation from the material world, shortened attention spans, ridiculously short obsolescence cycles, etc. etc. It is often difficult to tell these cultural biases apart from the "essence" of computing, and it is even more difficult to envision alternatives due to the lack of diversity. Thought experiments like this may be helpful for widening the perspective.</p> <p>So, what is the "essence" of computing then? I'd say universality. The universality of computing makes it possible to bend it to reflect and amplify just about any kind of ideology or cultural construct. In the recent decades, some ideas have just been so overwhelming that they feel very essential even though they are not. In an alternate timeline, other ideas might be dominant.</p> https://tedneward.github.io/Research/thinking/farnam-street-blog/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/farnam-street-blog/index.html <p>Eleven-plus years of thinking about thinking. I'm in Heaven. Notes and quotes from each blog entry, most-recent first.</p> <hr> <h2><a href="https://fs.blog/2020/11/how-julia-child-used-first-principles-thinking/">How Julia Child Used First Principles Thinking</a></h2> <blockquote> <p>There’s a big difference between knowing how to follow a recipe and knowing how to cook. If you can master the first principles within a domain, you can see much further than those who are just following recipes. That’s what Julia Child, "The French Chef", did throughout her career.</p> </blockquote> <p>Adhering to recipes will only get you so far, and it certainly won’t result in you coming up with anything new or creative. ... People who know how to cook understand the basic principles that make food taste, look, and smell good. ... There’s a reason many cooking competition shows feature a segment where contestants need to design their own recipe from a limited assortment of ingredients. Effective improvisation shows the judges that someone can actually cook, not just follow recipes.</p> <p><strong>If you want to learn how to think for yourself, you can’t just follow what someone else came up with.</strong> You need to understand <a href="https://fs.blog/2018/04/first-principles/">first principles</a> if you want to be able to solve complex problems or think in a unique, creative fashion. First principles are the building blocks of knowledge, the foundational understanding acquired from breaking something down into its most essential concepts. ... <strong>Once you understood how it actually worked, you could learn from mistakes instead of repeating them again and again. Looking for first principles is just a way of thinking. It’s a commitment to understanding the foundation that something is built on and giving yourself the freedom to adapt, develop, and create. Once you know the first principles, you can keep learning more advanced concepts as well as innovating for yourself.</strong></p> <hr> <h2><a href="https://fs.blog/2020/11/learning-through-play/">Learning Through Play</a></h2> <blockquote> <p>Play is an essential way of learning about the world. Doing things we enjoy without a goal in mind leads us to find new information, better understand our own capabilities, and find unexpected beauty around us. Arithmetic is one example of an area we can explore through play.</p> </blockquote> <p><strong>When the pressure mounts to be productive every minute of the day, we have much to gain from doing all we can to carve out time to play.</strong></p> <p><strong>Playing with symbols:</strong> (Arithmetic, and using different representations for the numbers, etc.) <strong>When we start to play with the representations, we connect to the underlying reasoning behind what we are doing.</strong></p> <p><strong>Stepping away from requirements:</strong> (Groupings of symbols, and groupings of groupings) "“You might think there is no question about it; we chose four as our grouping size, so that’s that. Of course we will group our groups into fours—as opposed to what? Grouping things into fours and then grouping our groups into sixes? That would be insane! But it happens all the time. Inches are grouped into twelves to make feet, and then three feet make a yard. And the old British monetary system had twelve pence to the shilling and twenty shillings to the pound.”" By reminding us of the options available in such a simple, everyday activity as counting, Lockhart opens a mental door. <a href="https://fs.blog/2019/03/stormtrooper-problem/">What other ways might we go about our tasks and solve our problems?</a> It’s a reminder that most of our so-called requirements are ones that we impose on ourselves.</p> <p><strong>Have fun with the unfamiliar:</strong> Getting stuck on anything can be incredibly useful. If forces you to stop and consider what it is you are really trying to achieve. Getting stuck can help you identify the first principles in your situation. In getting unstuck, we learn lessons that resonate and help us to grow. ... <strong>Play is often the exploration of the unfamiliar.</strong></p> <hr> <h2><a href="https://fs.blog/2020/10/descriptions-arent-prescriptions/">Descriptions Aren't Prescriptions</a></h2> <p>(The Malagasy language: official rules described in 19th-century, taught faithfully, even as its practitioners spoke along different sets of rules. " We find it dull to speak according to the official rules of our language. We seek out novelty in our everyday lives and do whatever it takes to avoid boredom. Even if each person only plays a little bit once in a while, the results compound. Graeber explains that 'this playing around will have cumulative effects.'" Languages still need conventions so people can understand each other. The higher the similarity between the versions of a language different people speak, the more they can communicate. At the same time, they cannot remain rigid. Trying to follow an unyielding set of strict rules will inevitably curtail the usefulness of a language and prevent it from developing in interesting and necessary ways. <strong>Languages need a balance: enough guidance to help everyone understand each other and provide an entry point for learners, and enough flexibility to keep updating the rules as actual usage changes.</strong>)</p> <p><a href="https://fs.blog/2015/11/map-and-territory/">“The map is not the territory”</a> means that any representation of reality has to be a simplification that may contain errors, become outdated, or reflect biases. Maps remove details that aren’t necessary for their intended use. Representations of complex systems may show expected behavior or ideal behavior. ... <strong>Sometimes maps are descriptive, and sometimes they’re prescriptive; often they’re a bit of both.</strong> We run into problems when we confuse one type for another and try to navigate an idealized territory or make the real territory fit an idealized image.</p> <p><strong>When you encounter a representation of something, it’s useful to consider which parts are descriptive and which parts are prescriptive. Remember that both prescriptions and descriptions can and should change over time.</strong></p> <hr> <h2><a href="https://fs.blog/2020/10/sharks-survivorship-bias/">What Sharks Can Teach Us About Survivorship Bias</a></h2> <blockquote> <p>Survivorship bias refers to the idea that we get a false representation of reality when we base our understanding only on the experiences of those who live to tell their story. Taking a look at how we misrepresent shark attacks highlights how survivorship bias distorts reality in other situations.</p> </blockquote> <p>We must be careful to not let a volume of survivors in one area blind us to the stories of a small number of survivors elsewhere. <strong>Most importantly, we need to ask ourselves what stories are not being told because no one is around to tell them.</strong> The experiences of the dead are necessary if we want an accurate understanding of the world.</p> <p><strong>Survivorship bias crops up all over our lives and impedes us from accurately assessing danger.</strong> ... <strong>Understanding survivorship bias prompts us to look for the stories of those who weren’t successful.</strong> A lack of visible survivors with memorable stories might mean we view other fields as far safer and easier than they are. ... <strong>Survivorship bias prompts us to associate more risk with industries that exhibit more public failures.</strong> ... <strong>If we don’t factor survivorship bias into our thinking we end up in a classic map is not the territory problem.</strong> ... <strong>Not recognizing survivorship bias can lead to faulty decision making.</strong></p> <hr> <h2><a href="https://fs.blog/2020/10/why-life-cant-be-simpler/">Why Life Can't Be Simpler</a></h2> <blockquote> <p>We’d all like life to be simpler. But we also don’t want to sacrifice our options and capabilities. <a href="https://lawsofux.com/teslers-law">Tesler’s law of the conservation of complexity</a>, a rule from design, explains why we can’t have both. Here’s how the law can help us create better products and services by rethinking simplicity.</p> </blockquote> <p>In <a href="https://www.amazon.com/Living-Complexity-Press-Donald-Norman/dp/0262528940/">Living with Complexity</a>, Donald A. Norman explains that complexity is all in the mind. Our perception of a product or service as simple or complex has its basis in the conceptual model we have of it. Norman writes that "A conceptual model is the underlying belief structure held by a person about how something works . . . Conceptual models are extremely important tools for organizing and understanding otherwise complex things." ... <strong>When we want something to be simpler, what we truly need is a better conceptual model of it.</strong></p> <p>Removing functionality doesn’t make something simpler, because it removes options. Simple tools have a limited ability to simplify processes. Trying to do something complex with a simple tool is more complex than doing the same thing with a more complex tool.</p> <p><strong>The total complexity of a system is a constant. If you make a user’s interaction with a system simpler, the complexity behind the scenes increases.</strong> ... <strong>Complexity is like energy. It cannot be created or destroyed, only moved somewhere else.</strong> "What is simple on the surface can be incredibly complex inside: what is simple inside can result in an incredibly complex surface. So from whose point of view do we measure complexity?"</p> <p><strong>From a user’s standpoint, the simplest products and services are those that are fully automated and do not require any intervention (unless something goes wrong.)</strong> ... <strong>On the other end of the spectrum, we have products and services that require users to control every last step.</strong></p> <p>Lessons from the conservation of complexity:<br> * <strong>How simple something looks is not a reflection of how simple it is to use.</strong> Perceived simplicity is not the same as operational (actual) simplicity.</p> <p>&gt; A very basic example of complexity trade-offs can be found in the history of arithmetic. For centuries, many counting systems all over the world employed tools using stones or beads like a tabula (the Romans) or soroban (the Japanese) to facilitate adding and subtracting numbers. They were easy to use, but not easily portable. Then the Hindu-Arabic system came along (the one we use today) and by virtue of employing columns, and thus not requiring any moving parts, offered a much more portable counting system. However, the portability came with a cost.</p> <p>&gt; Paul Lockhart explains in <em>Arithmetic</em>, "With the Hindu-Arabic system the writing and calculating are inextricably linked. Instead of moving stones or sliding beads, our manipulations become transmutations of the symbols themselves. That means we need to know things. We need to know that one more than 2 is 3, for instance. In other words, the price we pay [for portability] is massive amounts of memorization." Thus, there is a trade-off. The simpler arithmetic system requires more complexity in terms of the memorization required of the users. We all went through the difficult process of learning mathematical symbols early in life. Although they might seem simple to us now, that’s just because we’re so accustomed to them.</p> <p>Norman writes: "Perceived simplicity is not at all the same as simplicity of usage: operational simplicity. Perceived simplicity decreases with the number of visible controls and displays. Increase the number of visible alternatives and the perceived simplicity drops. The problem is that operational simplicity can be drastically improved by adding more controls and displays. The very things that make something easier to learn and to use can also make it be perceived as more difficult." Even if it receives a negative reaction before usage, operational simplicity is the more important goal.</p> <ul> <li> <p><strong>Things don’t always need to be incredibly simple for users.</strong> People have an intuitive sense that complexity has to go somewhere. When using a product or service is too simple, users can feel suspicious or like they’ve been robbed of control.</p> </li> <li> <p><strong>Products and services are only as good as what happens when they break.</strong></p> </li> <li> <p><strong>There is an importance of thinking about how the level of control you give your customers or users influences your workload.</strong></p> </li> </ul> <hr> <h2><a href="https://fs.blog/2020/09/being-smart-is-not-enough/">Being Smart is not Enough</a></h2> <blockquote> <p>When hiring a team, we tend to favor the geniuses who hatch innovative ideas, but overlook the butterflies, the crucial ones who share and implement them. Here’s why it’s important to be both smart AND social.</p> </blockquote> <p>In business, it’s never enough to have a great idea. For any innovation to be successful, it has to be shared, promoted, and bought into by everyone in the organization. Yet often we focus on the importance of those great ideas and seem to forget about the work that is required to spread them around. ... Joseph Henrich explores the role of culture in human evolution. One point he makes is that it’s not enough for a species to be smart. What counts far more is having the cultural infrastructure to share, teach, and learn:</p> <blockquote> <p>Consider two very large prehuman populations, the Geniuses and the Butterflies. Suppose the Geniuses will devise an invention once in 10 lifetimes. The Butterflies are much dumber, only devising the same invention once in 1000 lifetimes. So, this means that the Geniuses are 100 times smarter than the Butterflies. However, the Geniuses are not very social and have only 1 friend they can learn from. The Butterflies have 10 friends, making them 10 times more social.</p> <p>Now, everyone in both populations tries to obtain an invention, both by figuring it out for themselves and by learning from friends. Suppose learning from friends is difficult: if a friend has it, a learner only learns it half the time. After everyone has done their own individual learning and tried to learn from their friends, do you think the innovation will be more common among the Geniuses or the Butterflies?</p> </blockquote> <p>? Well, among the Geniuses a bit fewer than 1 out of 5 individuals (18%) will end up with the invention. Half of those Geniuses will have figured it out all by themselves. Meanwhile, 99.9% of Butterflies will have the innovation, but only 0.1% will have figured it out by themselves.</p> <hr> https://tedneward.github.io/Research/thinking/mental-models/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/mental-models/index.html <p><a href="https://fs.blog/mental-models/">Farnam Street Blog</a></p> <p>A mental model is <em>a simplified explanation of how something works</em>. Any idea, belief, or concept can be boiled down to its essence. Like a map, mental models highlight key information while ignoring irrelevant details. They’re tools for compressing complexity into manageable chunks. Mental models help us understand the world. For example, velocity shows that both speed and direction matter. Reciprocity reveals how being positive and taking initiative gets the world to do most of the work for you. Margin of Safety reminds us that things don’t always go as planned. Relativity exposes our blind spots and shows how a different perspective can reveal new information. These are just a few examples.</p> <h2>General Thinking</h2> <h3><a href="https://fs.blog/map-and-territory/">The Map is Not the Territory</a></h3> <p>The map is not the territory reminds us that our mental models of the world are not the same as the world itself. It cautions against confusing our abstractions and representations with the complex, ever-­shifting reality they aim to describe. It is dangerous to mistake the map for the territory. Consider the person with an outstanding résumé who checks all the boxes on paper but can’t do the job. Updating our maps is a difficult process of reconciling what we want to be true with what is true. In many areas of life, we are offered maps by other people. We are reliant on the maps provided by experts, pundits, and teachers. In these cases, the best we can do is to choose our mapmakers wisely and to seek out those who are rigorous, transparent, and open to revision. Ultimately, the map/territory distinction invites us to engage with the world as it is, not just as we imagine it. And remember, when you don’t make the map, choose your cartographer wisely.</p> <h3><a href="https://fs.blog/circle-of-competence/">Circle of Competence</a></h3> <p>The first rule of competition is that you are more likely to win if you play where you have an advantage. Playing to your advantage requires a firm understanding of what you know and don’t know. Your circle of competence is your personal sphere of expertise, where your knowledge and skills are concentrated. It’s the domain where you have a deep understanding, where your judgments are reliable, and your decisions are sound. The size of your circle isn’t as important as knowing the boundaries. The wise person knows the limits of their knowledge and can confidently say, “This falls within my circle,” or “This is outside my area of expertise.” While operating within your circle of competence is a recipe for confidence and effectiveness, venturing outside your circle of competence is a recipe for trouble. You’re like a sailor navigating unfamiliar waters without a map, at the mercy of currents and storms you don’t fully understand. This isn’t to say that you should never venture outside your circle. Learning new things, gaining new skills, and mastering new domains is one of the most beautiful things about life.</p> <p>Celebrate your expertise, but also acknowledge your limitations.</p> <h3><a href="https://fs.blog/first-principles/">First Principles Thinking</a></h3> <p>First principles thinking is the art of breaking down complex problems into their fundamental truths. It’s a way of thinking that goes beyond the surface and allows us to see things from a new perspective. Thinking in first principles allows us to identify the root causes, strip away the layers of complexity, and focus on the most effective solutions. Reasoning from first principles allows us to step outside the way things have always been done and instead see what is possible. First principles thinking is not easy. It requires a willingness to challenge the status quo. This is why it’s often the domain of rebels and disrupters who believe there must be a better way. It’s the thinking of those willing to start from scratch and build from the ground up.</p> <h3><a href="https://fs.blog/thought-experiment/">Thought Experiment</a></h3> <p>Thought experiments are the sandbox of the mind, the place where we can play with ideas without constraints. They’re a way of exploring the implications of our theories, of testing the boundaries of our understanding. They offer a powerful tool for clarifying our thinking, revealing hidden assumptions, and showing us unintended consequences.</p> <p>The power of thought experiments lies in their ability to create a simplified model of reality where we can test our ideas. In the real world, confounding factors and messy details obscure the core principles at work. Thought experiments allow us to strip away the noise and focus on the essence of the problem.</p> <p>Thought experiments remind us that some of the most profound insights and innovations start with a simple question: What if?</p> <h3><a href="https://fs.blog/second-order-thinking/">Second-Order Thinking</a></h3> <p>Second-­order thinking is a method of thinking that goes beyond the surface level, beyond the knee-­jerk reactions and short-­term gains. It asks us to play the long game, to anticipate the ripple effects of our actions, and to make choices that will benefit us not just today but in the months and years to come. Second-order thinking demands we ask: And then what? Think of a chess master contemplating her next move. She doesn’t just consider how the move will affect the next turn but how it will shape the entire game. She’s thinking many steps ahead. She’s considering her own strategy and her opponent’s likely response. In our daily lives, we’re often driven by first-­order thinking. We make decisions based on what makes us happy now, what eases our current discomfort, or satisfies our immediate desires. Second-­order thinking asks us to consider the long-­term implications of our choices to make decisions based not just on what feels good now but on what will lead to the best outcomes over time. In the end, second-­order thinking is about playing the long game. It’s about making choices for the next move and the entire journey.</p> <h3><a href="https://fs.blog/probabilistic-thinking/">Probabilistic Thinking</a></h3> <p>Probabilistic thinking is the art of navigating uncertainty. Successfully thinking in shades of probability means roughly identifying what matters, calculating the odds, checking our assumptions, and then deciding. The challenge of probabilistic thinking is that it requires constant updating. As new information emerges, the probabilities change. What seemed likely yesterday may seem unlikely today. This explains why probabilistic thinkers always revise their beliefs with new data and why it’s uncomfortable for many people. It’s much easier to believe something false is accurate than to deal with the fact that we might be wrong. Being a probabilistic thinker means being willing to say, “I don’t know for sure, but based on the evidence, I think there’s a 63 percent chance of X.” The rewards of probabilistic thinking are immense. By embracing uncertainty, we can make better decisions, avoid the pitfalls of overconfidence, and navigate complex situations with greater skill and flexibility. We can be more open-­ minded, more receptive to new ideas, and more resilient in the face of change.</p> <h3><a href="https://fs.blog/inversion/">Inversion</a></h3> <p>Much of success comes from simply avoiding common paths to failure. Inversion is not the way we are taught to think. We are taught to identify what we want and explore things that will move us closer to our objective. However, avoiding things that ensure we don’t get what we want dramatically increases our odds of success. We can get fixated on solving problems one way, missing simpler solutions. Inversion breaks us out of this tunnel vision. Instead of “How do I solve this?”, inversion asks, “What would guarantee failure?” Rather than “How can I achieve this?”, it asks “What’s preventing me from achieving it?” This flip reveals insights our usual thinking overlooks. When facing a tricky problem or ambitious goal, try inverting. Ask how you’d guarantee failure. The answers may surprise you—and unlock new solutions.</p> <h3><a href="https://fs.blog/occams-razor/">Occam’s Razor</a></h3> <p>Occam’s razor is the intellectual equivalent of “keep it simple.” When faced with competing explanations or solutions, Occam’s razor suggests that the correct explanation is most likely the simplest one, the one that makes the fewest assumptions. This doesn’t mean the simplest theory is always true, only that it should be preferred until proven otherwise. Sometimes, the truth is complex, and the simplest explanation doesn’t account for all the facts. The key to wielding this model is understanding when it works for you and against you. A theory that is too simple fails to capture reality, and one that is too complex collapses under its own weight.</p> <h3><a href="https://fs.blog/mental-model-hanlons-razor/">Hanlon’s Razor</a></h3> <p>Hanlon’s razor is a mental safeguard against the temptation to label behavior as malicious when incompetence is the most common response. It’s a reminder that people are not out to get you, and it’s best to assume good faith and resist the urge to assign sinister motives without overwhelming evidence. This isn’t to say that genuine malice doesn’t exist. Of course, it does. But in most interactions, stupidity is a far more common explanation than malevolence. People make mistakes. They forget things. They speak without thinking. They prioritize short-­term wins over long-term wins. They act on incomplete information. They fall prey to bias and prejudice. These actions might appear like deliberate attacks from the outside, but the reality is far more mundane. Hanlon’s razor’s real power lies in how it shifts our perspective. When we assume stupidity rather than malice, we respond differently. Instead of getting defensive or lashing out, we approach the situation with empathy and clarity. For most daily frustrations and confusion, Hanlon’s razor is a powerful reminder to approach problems with a spirit of generosity. It’s a way to reduce drama and stress and find practical solutions instead of descending into blame and escalation.</p> <h2>Systems Thinking</h2> <h3>Feedback Loops</h3> <p>Feedback loops are the engines of growth and change. They’re the mechanisms by which the output of a system influences its input.</p> <p>Complex systems often have many feedback loops, and it can be hard to appreciate how adjusting to feedback in one part of the system will affect the rest.</p> <p>Using feedback loops as a mental model begins with noticing the feedback you give and respond to daily. The model also provides insight into the value of iterations in adjusting based on the feedback you receive. With this lens, you gain insight into where to direct system changes based on feedback and the pace you need to go to monitor the impacts.</p> <p>Feedback loops are what make systems dynamic. Without feedback, a system does the same thing over and over. Understand them, respect them, and use them wisely.</p> <h3>Equilibrium</h3> <p>Equilibrium is the state of balance, where opposing forces cancel each other out. It’s the calm in the storm’s center, the stable point around which the chaos swirls. In a system at equilibrium, there’s no net change. Everything is in a steady state, humming along at a constant pace.</p> <p>However, systems are rarely static. They continuously adjust toward equilibrium but rarely stay in balance for long.</p> <p>Equilibrium is a ­ double-edged sword, both stability and stagnation. In our lives, we often act like we can reach an equilibrium: once we get into a relationship, we’ll be happy; once we move, we’ll be productive; once X thing happens, we’ll be in Y state. But things are always in flux. We don’t reach a certain steady state and then stay there forever. The endless adjustments are our lives. The trick is to find the right balance, strive for equilibrium where it’s needed, and know when to break free and embrace the dis-equilibrium that drives progress.</p> <h3>Bottlenecks</h3> <p>Bottlenecks are the choke points, the narrow parts of the hourglass where everything slows down. They’re the constraints that limit the flow, the weakest links in the chain that determine the strength of the whole. In any system, the bottleneck is the part holding everything else back.</p> <p>The tricky thing about bottlenecks is that they’re not always obvious. It’s easy to focus on the parts of the system that are moving quickly and assume everything is fine. But the real leverage is in finding and fixing the bottlenecks. Speed up the slowest part, and you speed up the whole system.</p> <p>This is the theory of constraints in a nutshell. Figure out your bottleneck and focus all your efforts on alleviating it. Don’t waste time optimizing the parts that are already fast. They’re not the limiting factor.</p> <p>However, bottlenecks aren’t always the villains we make them out to be. Sometimes, they’re a necessary part of the system. Think of a security checkpoint at an airport. It slows everything down, but it’s there for a reason. Remove it, and you might speed things up, but at the cost of safety.</p> <p>The key is to be intentional about your bottlenecks. Choose them wisely, and make sure they’re serving a purpose. A deliberate bottleneck can be a powerful tool for focusing effort and maintaining quality. An accidental bottleneck is just a drag on the system.</p> <p>Bottlenecks are leverage points where a little effort can go a long way.</p> <h3>Scale</h3> <p>Systems change as they scale up or down; neither is intrinsically better or worse. The right scale depends on your goals and the context. If you want to scale something up, you need to anticipate that new problems will keep ­ arising—​­ problems that didn’t exist on a smaller scale. Or you might need to keep solving the same problems in different ways.</p> <p>Think about a recipe. If you’re making a cake for four people, you use a certain amount of ingredients. But if you want to make a cake for four hundred people, you don’t just multiply the ingredients by one hundred. That’s not how scale works. You need to change the process and use bigger mixers and bigger ovens. You need a system that can handle the increased volume without breaking down.</p> <p>The challenge with scale is that it’s not always obvious how to achieve it. What works for a small system often breaks down at larger volumes. You have to anticipate the bottlenecks and the points where the system will strain under the increased load. And you have to be ready to re‑engineer your processes as you grow.</p> <p>If you’re building something, always be thinking about scale. How will this work when you have ten times as many customers? One hundred times? One thousand times? Build with scale in mind from the start, and you’ll be ready for the growth when it comes.</p> <h3>Margin of Safety</h3> <p>Margin of safety is a secret weapon. It’s the buffer, the extra capacity, the redundancy that you build into a system to handle unexpected stress. It’s the difference between a bridge that can barely handle the expected load and one that can handle ten times that load without breaking a sweat.</p> <p>You can apply a margin of safety to any area of life with uncertainty and risk. The key is always to ask yourself: What if I’m wrong? What if things don’t go as planned? How much extra capacity must I build to handle the unexpected?</p> <p>But here’s the rub: margin of safety isn’t free. It means spending more upfront. In the short term, you’ll look overly cautious and leave immediate profits on the table. But in the long run, this apparent overcaution lets you survive when others break – and thrive when others merely survive.</p> <p>Margin of safety is the unsung hero of ­ long-​­term success. It’s not flashy. It’s not exciting, but it’s the foundation on which everything else is built. Master it, and you’ll be well on your way to navigating the uncertainties of life with confidence and stability.</p> <h3>Churn</h3> <p>Churn is the silent killer of businesses. It’s the slow leak, the constant drip of customers slipping away, of users drifting off to find something new. The attrition eats away at your growth, forcing you to keep running just to stay in place. The thing about churn is that it’s often hidden. It’s not like a sudden crisis that grabs your attention. It’s a slow, quiet process that happens in the background.</p> <p>Churn can present opportunity. Like a snake shedding its skin, replacing components of a system is a natural part of keeping it healthy. New parts can improve functionality.</p> <p>When we use this model as a lens, we see that new people bring new ideas, and counterintuitively, some turnover allows us to maintain stability. Replacing what is worn out also allows us to upgrade and expand our capabilities, creating new opportunities. Some churn is inevitable. Too much can kill you.</p> <h3>Algorithms</h3> <p>Algorithms are recipes. A list of crisp, unambiguous steps that tell you how to get from point A to point B. But they’re more than just directions. Algorithms are if‑then machines for tuning out the noise and zeroing in on the signal. Have the specs been met? Fol- low the algorithm and find out. Thinking algorithmically means searching for processes that reliably spit out the desired results, like a vending machine dispensing the same candy bar every time someone punches in E4.</p> <h3>Critical mass</h3> <p>Critical mass isn’t just a science term; it’s a guide for understanding that often things happen slowly and then all at once. It’s the moment when a system goes from sputtering along to explosive growth. Like a nuclear chain reaction, once you hit critical mass, the reaction becomes self-sustaining.</p> <p>Through this lens we gain insight into the amount of material needed for a system to change from one state to another. Material can be anything from people and effort to raw material. When enough material builds up, systems reach their tipping point. When we keep going, we get sustainable change.</p> <p>Using critical mass as a lens for situations where you want different outcomes helps you identify both the design elements you need to change and the work you need to put in.</p> <h3>Emergence</h3> <p>Nearly everything is an emergent ­ effect—​­a table, a space shuttle, even ­ us—​­ combinations of ingredients that come together in a specific way to create something new. Emergence is the universe’s way of reminding us that when we combine different pieces in new ways, we get results that are more than the sum of their parts, often in the most unexpected and thrilling ways.</p> <p>Using this mental model is not about predicting emergent properties but acknowledging they are possible. There is no need to stick with what you know; mix it up and see what happens. Learn new skills, interact with new people, read new things.</p> <h3>Irreducibility</h3> <p>Irreducibility is about essence. It’s the idea that some things can’t be broken down into smaller parts without losing what makes them tick. It’s the idea that not everything can be explained by looking at its components. Emergent properties arise from complex systems that can’t be predicted by studying the individual parts.</p> <p>Grappling with irreducibility requires a shift in thinking. Instead of trying to break things down, sometimes you have to zoom out. Look at the big picture. Embrace the complexity. Because some problems don’t have neat, modular solutions. They’re irreducibly messy.</p> <p>Using irreducibility as a lens helps you focus on what you can change by understanding what really matters</p> <h3>Law of Diminishing Returns</h3> <p>Diminishing returns is the idea that the easy wins usually come first. The more you optimize a system, the harder it gets to eke out additional improvements, like squeezing juice from a lemon. The first squeeze is easy. The second takes a bit more work. By the tenth squeeze, you’re fighting for every last drop.</p> <p>Every bit of effort translates into significant gains when you’re a beginner. But as you level up, progress becomes more incremental. It takes more and more work to get better and better. That’s why going from good to great is much harder than going from bad to good.</p> <p>Understanding diminishing returns is crucial for allocating resources efficiently. You want to focus on where you can get the biggest bang for your buck. Sometimes, that means knowing when to stop optimizing and move on to something else.</p> <h2>The Mental Models of Military and War</h2> <h3>Seeing the Front</h3> <p>One of the most valuable military tactics is the habit of “personally seeing the front” before making decisions – not always relying on advisors, maps, and reports, all of which can be faulty or biased. The Map/Territory model, as does the incentive model, illustrates the problem of not seeing the front. Leaders of any organization can generally benefit from seeing the front, as it provides firsthand information and tends to improve the quality of secondhand information. (Also known as, "Eating Your Own Dog Food" or "Working the Phones")</p> <h3>Asymmetric Warfare</h3> <p>The asymmetry model leads to an application in warfare whereby one side seemingly “plays by different rules” than the other side due to circumstance. Generally, this model is applied by an insurgency with limited resources. Unable to out-muscle their opponents, asymmetric fighters use other tactics, as with terrorism creating fear that’s disproportionate to their actual destructive ability.</p> <h3>Two-Front War</h3> <p>The Second World War was a good example of a two-front war. Once Russia and Germany became enemies, Germany was forced to split its troops and send them to separate fronts, weakening their impact on either front. Opening a two-front war can often be a useful tactic, as can solving a two-front war or avoiding one, as in the example of an organization tamping down internal discord to focus on its competitors.</p> <h3><a href="https://fs.blog/counterinsurgency/">Counterinsurgency</a></h3> <p>Though asymmetric insurgent warfare can be extremely effective, competitors have developed counterinsurgency strategies over time. Recently and famously, General David Petraeus of the United States led the development of counterinsurgency plans involving no additional force but substantial gains. Tit-for-tat warfare or competition often leads to a feedback loop that demands insurgency and counterinsurgency.</p> <h3><a href="https://fs.blog/mutually-assured-destruction/">Mutually Assured Destruction</a></h3> <p>Somewhat paradoxically, the stronger two opponents become, the less likely they may be to destroy one another. This process of mutually assured destruction occurs not just in warfare, as with the development of global nuclear warheads, but also in business, as with the avoidance of destructive price wars between competitors. However, in a fat-tailed world, it is also possible that mutually assured destruction scenarios simply make destruction more severe in the event of a mistake (pushing destruction into the “tails” of the distribution).</p> <h2>Human Nature and Judgment</h2> <h3>Trust</h3> <p>Fundamentally, the modern world operates on trust. Familial trust is generally a given (otherwise we’d have a hell of a time surviving), but we also choose to trust chefs, clerks, drivers, factory workers, executives, and many others. A trusting system is one that tends to work most efficiently; the rewards of trust are extremely high.</p> <h3>Bias from Incentives</h3> <p>Highly responsive to incentives, humans have perhaps the most varied and hardest to understand set of incentives in the animal kingdom. This causes us to distort our thinking when it is in our own interest to do so. A wonderful example is a salesman truly believing that his product will improve the lives of its users. It’s not merely convenient that he sells the product; the fact of his selling the product causes a very real bias in his own thinking.</p> <h3>Pavlovian Association</h3> <p>Ivan Pavlov very effectively demonstrated that animals can respond not just to direct incentives but also to associated objects; remember the famous dogs salivating at the ring of a bell. Human beings are much the same and can feel positive and negative emotion towards intangible objects, with the emotion coming from past associations rather than direct effects.</p> <h3><a href="https://fs.blog/mental-model-bias-envy-jealousy/">Tendency to Feel Envy &amp; Jealousy</a></h3> <p>Humans have a tendency to feel envious of those receiving more than they are, and a desire “get what is theirs” in due course. The tendency towards envy is strong enough to drive otherwise irrational behavior, but is as old as humanity itself. Any system ignorant of envy effects will tend to self-immolate over time.</p> <h3><a href="https://fs.blog/mental-model-bias-from-liking-loving/">Tendency to Distort Due to Liking/Loving</a> or <a href="https://www.farnamstreetblog.com/2016/09/bias-from-disliking-hating/">Disliking/Hating</a></h3> <p>Based on past association, stereotyping, ideology, genetic influence, or direct experience, humans have a tendency to distort their thinking in favor of people or things that they like and against people or things they dislike. This tendency leads to overrating the things we like and underrating or broadly categorizing things we dislike, often missing crucial nuances in the process.</p> <h3>Denial</h3> <p>Anyone who has been alive long enough realizes that, as the saying goes, “denial is not just a river in Africa.” This is powerfully demonstrated in situations like war or drug abuse, where denial has powerful destructive effects but allows for behavioral inertia. Denying reality can be a coping mechanism, a survival mechanism, or a purposeful tactic.</p> <h3><a href="https://fs.blog/mental-model-availability-bias/">Availability Heuristic</a></h3> <p>One of the most useful findings of modern psychology is what Daniel Kahneman calls the Availability Bias or Heuristic: We tend to most easily recall what is salient, important, frequent, and recent. The brain has its own energy-saving and inertial tendencies that we have little control over – the availability heuristic is likely one of them. Having a truly comprehensive memory would be debilitating. Some sub-examples of the availability heuristic include the Anchoring and Sunk Cost Tendencies.</p> <h3>Representativeness Heuristic</h3> <p>The three major psychological findings that fall under Representativeness, also defined by Kahneman and his partner Tversky, are:</p> <ol> <li><a href="https://fs.blog/mental-model-bias-from-insensitivity-to-base-rates/">Failure to Account for Base Rates</a>: An unconscious failure to look at past odds in determining current or future behavior.</li> <li>Tendency to Stereotype: The tendency to broadly generalize and categorize rather than look for specific nuance. Like availability, this is generally a necessary trait for energy-saving in the brain.</li> <li>Failure to See False Conjunctions: Most famously demonstrated by the Linda Test, the same two psychologists showed that students chose more vividly described individuals as more likely to fit into a predefined category than individuals with broader, more inclusive, but less vivid descriptions, even if the vivid example was a mere subset of the more inclusive set. These specific examples are seen as more representative of the category than those with the broader but vaguer descriptions, in violation of logic and probability.</li> </ol> <h3><a href="https://fs.blog/mental-model-social-proof/">Social Proof</a> (Safety in Numbers)</h3> <p>Human beings are one of many social species, along with bees, ants, and chimps, among many more. We have a DNA-level instinct to seek safety in numbers and will look for social guidance of our behavior. This instinct creates a cohesive sense of cooperation and culture which would not otherwise be possible but also leads us to do foolish things if our group is doing them as well.</p> <h3><a href="https://fs.blog/narrative-fallacy/">Narrative Instinct</a></h3> <p>Human beings have been appropriately called “the storytelling animal” because of our instinct to construct and seek meaning in narrative. It’s likely that long before we developed the ability to write or to create objects, we were telling stories and thinking in stories. Nearly all social organizations, from religious institutions to corporations to nation-states, run on constructions of the narrative instinct.</p> <h3>Curiosity Instinct</h3> <p>We like to call other species curious, but we are the most curious of all, an instinct which led us out of the savanna and led us to learn a great deal about the world around us, using that information to create the world in our collective minds. The curiosity instinct leads to unique human behavior and forms of organization like the scientific enterprise. Even before there were direct incentives to innovate, humans innovated out of curiosity.</p> <h3>Language Instinct</h3> <p>The psychologist Steven Pinker calls our DNA-level instinct to learn grammatically constructed language the Language Instinct. The idea that grammatical language is not a simple cultural artifact was first popularized by the linguist Noam Chomsky. As we saw with the narrative instinct, we use these instincts to create shared stories, as well as to gossip, solve problems, and fight, among other things. Grammatically ordered language theoretically carries infinite varying meaning.</p> <h3>First-Conclusion Bias</h3> <p>As Charlie Munger famously pointed out, the mind works a bit like a sperm and egg: the first idea gets in and then the mind shuts. Like many other tendencies, this is probably an energy-saving device. Our tendency to settle on first conclusions leads us to accept many erroneous results and cease asking questions; it can be countered with some simple and useful mental routines.</p> <h3><a href="https://fs.blog/mental-model-bias-from-insensitivity-to-sample-size/">Tendency to Overgeneralize from Small Samples</a></h3> <p>It’s important for human beings to generalize; we need not see every instance to understand the general rule, and this works to our advantage. With generalizing, however, comes a subset of errors when we forget about the Law of Large Numbers and act as if it does not exist. We take a small number of instances and create a general category, even if we have no statistically sound basis for the conclusion.</p> <h3>Relative Satisfaction/Misery Tendencies</h3> <p>The envy tendency is probably the most obvious manifestation of the relative satisfaction tendency, but nearly all studies of human happiness show that it is related to the state of the person relative to either their past or their peers, not absolute. These relative tendencies cause us great misery or happiness in a very wide variety of objectively different situations and make us poor predictors of our own behavior and feelings.</p> <h3><a href="https://fs.blog/commitment-consistency-bias/">Commitment &amp; Consistency Bias</a></h3> <p>As psychologists have frequently and famously demonstrated, humans are subject to a bias towards keeping their prior commitments and staying consistent with our prior selves when possible. This trait is necessary for social cohesion: people who often change their conclusions and habits are often distrusted. Yet our bias towards staying consistent can become, as one wag put it, a “hobgoblin of foolish minds” – when it is combined with the first-conclusion bias, we end up landing on poor answers and standing pat in the face of great evidence.</p> <h3><a href="https://fs.blog/what-is-hindsight-bias/">Hindsight Bias</a></h3> <p>Once we know the outcome, it’s nearly impossible to turn back the clock mentally. Our narrative instinct leads us to reason that we knew it all along (whatever “it” is), when in fact we are often simply reasoning post-hoc with information not available to us before the event. The hindsight bias explains why it’s wise to keep a journal of important decisions for an unaltered record and to re-examine our beliefs when we convince ourselves that we knew it all along.</p> <h3><a href="https://fs.blog/kantian-fairness-tendency/">Sensitivity to Fairness</a></h3> <p>Justice runs deep in our veins. In another illustration of our relative sense of well-being, we are careful arbiters of what is fair. Violations of fairness can be considered grounds for reciprocal action, or at least distrust. Yet fairness itself seems to be a moving target. What is seen as fair and just in one time and place may not be in another. Consider that slavery has been seen as perfectly natural and perfectly unnatural in alternating phases of human existence.</p> <h3>Tendency to Overestimate Consistency of Behavior (<a href="https://fs.blog/fundamental-attribution-error/">Fundamental Attribution Error</a>)</h3> <p>We tend to over-ascribe the behavior of others to their innate traits rather than to situational factors, leading us to overestimate how consistent that behavior will be in the future. In such a situation, predicting behavior seems not very difficult. Of course, in practice this assumption is consistently demonstrated to be wrong, and we are consequently surprised when others do not act in accordance with the “innate” traits we’ve endowed them with.</p> <h3>Influence of Stress (Including Breaking Points)</h3> <p>Stress causes both mental and physiological responses and tends to amplify the other biases. Almost all human mental biases become worse in the face of stress as the body goes into a fight-or-flight response, relying purely on instinct without the emergency brake of Daniel Kahneman’s “System 2” type of reasoning. Stress causes hasty decisions, immediacy, and a fallback to habit, thus giving rise to the elite soldiers’ motto: “In the thick of battle, you will not rise to the level of your expectations, but fall to the level of your training.”</p> <h3><a href="https://fs.blog/survivorship-bias/">Survivorship Bias</a></h3> <p>A major problem with historiography – our interpretation of the past – is that history is famously written by the victors. We do not see what Nassim Taleb calls the “silent grave” – the lottery ticket holders who did not win. Thus, we over-attribute success to things done by the successful agent rather than to randomness or luck, and we often learn false lessons by exclusively studying victors without seeing all of the accompanying losers who acted in the same way but were not lucky enough to succeed.</p> <h3><a href="https://fs.blog/do-something-syndrome/">Tendency to Want to Do Something</a> (Fight/Flight, Intervention, Demonstration of Value, etc.)</h3> <p>We might term this Boredom Syndrome: Most humans have the tendency to need to act, even when their actions are not needed. We also tend to offer solutions even when we do not have knowledge to solve the problem.</p> <h3><a href="https://fs.blog/peter-cathcart-wason-falsification/">Falsification</a> / <a href="https://fs.blog/confirmation-bias/">Confirmation Bias</a></h3> <p>What a man wishes, he also believes. Similarly, what we believe is what we choose to see. This is commonly referred to as the confirmation bias. It is a deeply ingrained mental habit, both energy-conserving and comfortable, to look for confirmations of long-held wisdom rather than violations. Yet the scientific process – including hypothesis generation, blind testing when needed, and objective statistical rigor – is designed to root out precisely the opposite, which is why it works so well when followed.</p> <p>The modern scientific enterprise operates under the principle of falsification: A method is termed scientific if it can be stated so that a certain defined result would cause it to be proved false. Pseudo-knowledge and pseudo-science operate and propagate by being unfalsifiable. As with astrology, we cannot prove them either correct or incorrect because the conditions under which they would be shown false are never stated.</p> https://tedneward.github.io/Research/thinking/finite-infinite-games/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/finite-infinite-games/index.html <p>(by James Carse, 1986)</p> <h2>One: There Are at Least Two Kinds of Games</h2> <ol> <li> <p>A finite game is played for the purpose of winning, an infinite game for the purpose of continuing the play.</p> </li> <li> <p>If a finite game is to be won by someone it must come to a definitive end. It will come to an end when someone has won. We know that someone has won the game when all the players have agreed who among them is the winner. No other condition than the agreement of the players is absolutely required in determining who has won the game. … There is no finite game unless the players freely choose to play it. No one can play who is forced to play. It is an invariable principle of all play, finite and infinite, that whoever plays, plays freely. Whoever must play, cannot play.</p> </li> <li> <p>We can speak of finite games as having temporal boundaries. Players must agree to the establishment of spatial and numerical boundaries as well. That is, the game must be played within a marked area, and with specified players.</p> </li> <li> <p>To have such boundaries means that the date, place, and membership of each finite game are externally defined. The world is elaborately marked by boundaries of contest, its people finely classified as to their eligibilities.</p> </li> <li> <p>Only one person or team can win a finite game, but the other contestants may well be ranked at the conclusion of play.</p> </li> <li> <p>In one respect, but only one, an infinite game is identical to a finite game: Of infinite players we can also say that if they play they play freely; if they must play, they cannot play. Otherwise, infinite and finite play stand in the sharpest possible contrast. Infinite players cannot say when their game began, nor do they care. There are no spatial or numerical boundaries to an infinite game. While finite games are externally defined, infinite games are internally defined. The time of an infinite game is not world time, but time created within the play itself. Since each play of an infinite game eliminates boundaries, it opens to players a new horizon of time.</p> </li> <li> <p>Finite games can be played within an infinite game, but an infinite game cannot be played within a finite game. Infinite players regard their wins and losses in whatever finite games they play as but moments in continuing play.</p> </li> <li> <p>If finite games must be externally bounded by time, space, and number, they must also have internal limitations on what the players can do to and with each other. To agree on internal limitations is to establish rules of play. It is, in fact, by knowing what the rules are that we know what the game is. The rules of a finite game are the contractual terms by which players can agree who has won.</p> </li> <li> <p>The rules must be published prior to play, and the players must agree to them before play begins. The agreement of the players to the applicable rules constitutes the ultimate validation of those rules. They are valid only if and when players freely play by them.</p> </li> <li> <p>If the rules of a finite game are unique to that game it is evident that the rules may not change in the course of play--else a different game is being played. The rules of an infinite game must change in the course of play. The rules are changed when the players of an infinite game agree that the play is imperiled by a finite outcome--that is, by the victory of some players and the defeat of others. The rules of an infinite game are changed to prevent anyone from winning the game and to bring as many persons as possible into the play. If the rules of a finite game are the contractual terms by which the players can agree who has won, the rules of an infinite game are the contractual terms by which the players agree to continue playing.</p> </li> <li> <p>The rules of an infinite game are always designed to deal with specific threats to the continuation of play. Infinite players use the rules to regulate the way they will take the boundaries or limits being forced against their play into the game itself. The rule-making capacity of infinite players is often challenged by the impingement of powerful boundaries against their play--such as physical exhaustion, or the loss of material resources, or the hostility of nonplayers, or death. The task is to design rules that will allow the players to continue the game by taking these limits into play--even when death is one of the limits. Finite players play within boundaries; infinite players play with boundaries.</p> </li> <li> <p>Although it may be evident enough in theory that whoever plays a finite game plays freely, it is often the case that finite players will be unaware of this absolute freedom and will come to thin that whatever they do they must do.</p> </li> </ol> <h2>Two: No One Can Play a Game Alone</h2> <h2>Three: I Am the Genius of Myself</h2> <h2>Four: A Finite Game Occurs Within a World</h2> <h2>Five: Nature Is the Realm of the Unspeakable</h2> <h2>Six: We Control Nature for Societal Reasons</h2> <h2>Seven: Myth Provokes Explanation but Accepts None of It</h2> https://tedneward.github.io/Research/thinking/philosophy/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/philosophy/index.html <h2>Places</h2> <ul> <li><a href="https://plato.stanford.edu/">Stanford Encyclopedia of Philosophy</a> | <a href="https://plato.stanford.edu/contents.html">Table of Contents</a> <ul> <li><a href="https://plato.stanford.edu/entries/gratitude/">Gratitude</a>: Gratitude is the proper or called-for response in a beneficiary to benefits or beneficence from a benefactor. It is a topic of interest in normative ethics, applied ethics, moral psychology, and political philosophy. Despite its ubiquity in everyday life, there is substantive disagreement among philosophers over the nature of gratitude and its relationship to other philosophical concepts. The sections of this article address five areas of debate about what gratitude is, when gratitude is called for, and how the answers to those questions bear on other topics in moral philosophy and philosophy generally.</li> </ul> </li> </ul> https://tedneward.github.io/Research/thinking/brain-rules/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/brain-rules/index.html <p><em>(by John Medina; Pear Press, when?; ISBN ...)</em></p> <p>1) Exercise boosts brain power<br> 2) The human brain evolved, too<br> 3) Every brain is wired differently<br> 4) We don't pay attention to boring things<br> 5) Repeat to remember<br> 6) Remember to repeat<br> 7) Sleep well, think well<br> 8) Stressed brains don't learn the same way<br> 9) Stimulate more of the senses<br> 10) Vision trumps all other senses<br> 11) Male and female brains are different<br> 12) We are powerful and natural explorers</p> https://tedneward.github.io/Research/thinking/economics/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/economics/index.html <h2>The Economist</h2> <p><a href="https://www.economist.com/the-economist-reads/2022/08/09/what-to-read-to-understand-how-economists-think">What to read to understand how economists think</a>:</p> <ul> <li><strong><em>Capitalism and Freedom</em></strong>. <em>By Milton Friedman. University of Chicago Press; 272 pages</em> "This book is perhaps the best way to learn to think about trade-offs, because that was how Friedman always thought about the world."</li> <li><strong><em>The Worldly Philosophers</em></strong>. <em>By Robert Heilbroner. Touchstone; 368 pages; $18.99. Simon &amp; Schuster</em> "In a cheerful, conversational style, Heilbroner takes the reader through the writings of the earliest economists, explaining why their ideas were so revolutionary."</li> <li><strong><em>Africa: Why Economists Get It Wrong</em></strong>. <em>By Morten Jerven. Bloomsbury Academic; 176 pages</em> "A lot of economists plug data from Africa into huge statistical models, seeking to explain, for instance, why social trust is higher in one part of a country than another. But a lot of this research is based on shoddy data. ... One lesson from the book (which we reviewed in 2015) is to be upfront about the limitations of your data sources."</li> <li><strong><em>Capitalism, Alone</em></strong>. <em>By Branko Milanovic. Harvard University Press; 304 pages</em> "This is the book to read if you want to understand why capitalism—and economists’ way of thinking—has triumphed the world over. ... Capitalism is far from perfect, his book shows, yet it is hard to shake the notion that it is the only system that broadly works."</li> <li><strong><em>Freakonomics: A Rogue Economist Explores the Hidden Side of Everything</em></strong>. <em>By Stephen J. Dubner and Steven Levitt. HarperCollins; 352 pages</em> "This book is so famous that it seems a cliché to choose it, but if you haven’t read it, you really should."</li> </ul> <p><a href="https://www.economist.com/the-economist-reads/2022/07/04/what-to-read-to-understand-the-history-of-western-capitalism">What to read to understand the history of Western capitalism</a></p> https://tedneward.github.io/Research/thinking/category-theory/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/category-theory/index.html <p>Because apparently this is now "a thing" among serious programmers....</p> <p><a href="https://arxiv.org/abs/1803.05316">[1803.05316] Seven Sketches in Compositionality: An Invitation to Applied Category Theory</a> [Mar 2018]</p> <blockquote> <p>What’s neat is that no prior knowledge of category theory is assumed! They write for a broad audience: motivated HS students, machine learning researchers, retired programmers, pure math’ns…, &amp; each ch. is based on real-world topics like electrical circuits and control theory. I love what they share in the Preface: “The motto at MIT is ‘mens et manus,’ Latin for ‘mind and hand.’ We believe that category theory—and pure math in general—has stayed in the realm of mind for too long; it is ripe to be brought to hand.” – <a href="https://twitter.com/math3ma">via Twitter</a></p> </blockquote> <p><a href="https://www.slideshare.net/JordanParmer/category-theory-in-10-minutes-77309719">Category Theory in 10 Minutes</a> | <a href="http://rs.io/why-category-theory-matters/">Why Category Theory Matters</a> | <a href="https://markkarpov.com/post/category-theory-part-1.html">Category Theory Basics, Part I</a> | <a href="http://ncatlab.org/nlab/show/category+theory">Notebook on category theory at nLab</a> | <a href="http://arxiv.org/abs/1302.6946">Category Theory for Scientists</a> by David Spivak. | <a href="http://category-theory.mitpress.mit.edu/chapter001.html">Category Theory for the sciences (book)</a> by David Spivak. | <a href="http://www.logicmatters.net/resources/pdfs/GentleIntro.pdf">Category Theory - a gentle introduction (PDF)</a> by Peter Smith of Cambridge university. 171 pages. | <a href="http://plato.stanford.edu/entries/category-theory/bib.html">Pragammatic Reading guide</a> via <a href="http://redd.it/1ht4mf">this reddit thread</a>. | <a href="http://ct2015.web.ua.pt/talks.html">CT 2015 conf</a> | <a href="http://www.logicmatters.net/categories/">Reading lists</a> | <a href="https://mioalter.github.io/posts/2018-12-17-Yoneda.html">Yoneda, Currying, and Fusion</a>; Dec 2018.</p> <p><a href="https://bartoszmilewski.com/2014/10/28/category-theory-for-programmers-the-preface/">Bartosz Milewski: Category Theory for Programmers</a></p> <p><a href="https://www.quantamagazine.org/with-category-theory-mathematics-escapes-from-equality-20191010/">"With Category Theory, Mathematics Escapes from Equality"</a>: "Two monumental works have led many mathematicians to avoid the equal sign. Their goal: Rebuild the foundations of the discipline upon the looser relationship of “equivalence.” The process has not always gone smoothly."</p> https://tedneward.github.io/Research/thinking/concept-maps/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/concept-maps/index.html <p><a href="https://cmap.ihmc.us/docs/learn.php">Cmap</a></p> https://tedneward.github.io/Research/thinking/atomic-habits/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/atomic-habits/index.html <p><em>(???)</em></p> <h2><a href="https://www.chrisbehan.ca/posts/atomic-habits">Quick Summary</a></h2> <p>A habit is a routine or behavior that is performed so regularly it becomes automatic. From a biological perspective, the purpose of habits is to preserve energy by offloading the cognitive effort spent on repetitive tasks from the conscious to the subconscious. As you repeat a certain behavior, the connection between the neurons (the fundamental units of the brain) associated with that behavior become stronger. After enough repetitions, the connection becomes so strong that the behavior is automatic. When Stephen Curry picked up and shot a basketball for the first time, he had to focus on every aspect of the motion: hand position, balance, posture, and release. But after millions of practice shots, Curry can shoot the ball from anywhere on the court without thinking.</p> <p>When used effectively, habits empower you to become whoever you want in life. A habit of daily reading will eventually make you knowledgeable, a habit of daily exercise will eventually make you strong. The difficult part is sticking with the habit long enough to reap the benefits. You won't develop a wealth of knowledge from reading one book, and you won't get jacked after going to the gym for a week. Only after years of reading will you be able to compare and build upon ideas from different disciplines. Only after months of workouts will you be able to fully activate all the muscles associated with a lift. A 1% improvement every day for a year results in a 37x improvement from where you started. The initial 1 percent improvements are difficult to notice, but after a while they become astronomical. All big things come from small beginnings.</p> <p>Social media is filled with posts celebrating the achievements of individuals while glossing over the thousands of hours it took them to get there. What we don't see in pictures of Giannis Antetokounmpo hoisting the Larry O'Brien trophy are the years of grueling practice that preceded his championship run. We've been tricked into expecting instant results without putting in the reps. To effectively develop new habits, we need to break this mindset and acknowledge that the most powerful outcomes are delayed. We must shift our focus from achievement to trajectory. The strongest man in the world becomes weak if he stops using his muscles. The brightest man becomes dumb if he stops using his brain. It matters not where you were yesterday, but rather, where you're headed today.</p> <p>Trajectory &gt; Achievements</p> <h3>The Three Layers of Behavior Change</h3> <p>To effectively build new habits, you need to be aware of the three layers of behavior change:</p> <ol> <li>Identity</li> <li>Process</li> <li>Outcome</li> </ol> <p>Identity is what you believe, process is what you do, and outcome is what you get. The reason most people fall short of their aspirations in life is that they are solely focused on outcomes, without considering the systems that lead to those outcomes. Telling yourself "I want to get rich" or "I want to lose weight" will not contribute to either. Only through modifying our identity and processes can we change our behavior and bring about our desired outcomes.</p> <p>Your behaviors are an expression of your identity. If you exercise every day, you are an active person. If you watch Netflix all day, you are a lazy person. To change your outcomes, you must first change your identity and processes.</p> <p>To change your identity, start by asking how someone with your desired identity behaves. If you want to become an entrepreneur, ask "Who is the type of person that becomes an entrepreneur?" An entrepreneur is the type of person who creates things. An entrepreneur is the type of person who is willing to fail. An entrepreneur is the type of person who takes risks for what they believe in. Start acting like your desired identity, and eventually, you will become them. Write every day and you will become a writer. Program every day and you will become a programmer. Every action you take is a vote for the type of person you wish to become.</p> <p>As you've likely noticed, just telling yourself you're going to start doing something is not an effective way to change your behavior. Our brains are energy conservation machines, constantly analyzing the environment for ways to maximize immediate rewards while spending as little energy as possible. Effective processes manipulate your environment so that the energy required for the desirable behavior is less than the energy required for the undesirable behavior. It's tough to eat healthier if your fridge is stocked full of chocolate bars. But by manipulating your environment such that your fridge is filled with fruit and vegetables and eating a Cadbury requires a trip to the supermarket, you've suddenly tipped the scale in your favor. Every goal is doomed to fail if it goes against the grain of human nature.</p> <h3>The Four Steps of Habit Formation</h3> <p>There are four steps involved in the formation of every habit:</p> <ol> <li>Cue</li> <li>Craving</li> <li>Response</li> <li>Reward</li> </ol> <p>The cue is the initial signal that triggers the behavior (ex. the sight of a chocolate bar). The craving is the desired change in state that results from the cue (ex. a desire for the taste of chocolate). The response is the behavior that is performed to satisfy the craving (ex. eating the chocolate bar). The reward is the outcome delivered by the response (ex. the sugar rush from eating the chocolate bar). If a behavior is insufficient in any of the above steps, it will not become a habit. Remove the cue and the habit will never start. Reduce the craving and you won't be motivated enough to act. If you don't respond, there will be no reward. And if there is no reward, or it doesn't live up to your expectations, you won't repeat the behavior in the future.</p> <h3>The Four Laws of Behavior Change</h3> <p>The four laws of behavior change are a set of actionable principles for building habits. The laws are:</p> <ol> <li>Make it obvious</li> <li>Make it attractive</li> <li>Make it easy</li> <li>Make it satisfying</li> </ol> <p>Each law maps to one of the four steps of habit formation:</p> <ol> <li>Make it obvious → Cue</li> <li>Make it attractive → Craving</li> <li>Make it easy → Response</li> <li>Make it satisfying → Reward</li> </ol> <h3>Make it Obvious</h3> <p>To effectively build a new habit, make the cue for that habit blatantly obvious. If you want to start running every morning, place your runners next to your bed the night before. When you wake up, the first thing you'll see is your sneakers, signaling to your brain that it's time to get active. If you want to build a habit of reading before bed, place a book on your pillow every morning. When it's time for bed, the book will be waiting for you. Design your environment around the habits you wish to develop. The more clear and consistent the cues in your environment, the more likely you will be to develop the habit associated with the cue.</p> <h4>Implementation Intentions</h4> <p>Implementation intentions are used to increase your likelihood of performing a habit. The idea behind implementation intentions is to eliminate ambiguity around when and where you will perform a habit. Instead of just telling yourself "I'm going to start running", grab a piece of paper and write down "I will run at 7am in my neighborhood". The format of implementation intentions is:</p> <p>I will [BEHAVIOUR] at [TIME] in/at [LOCATION]</p> <p>Examples:</p> <ul> <li>I will write at 9 am in my home office</li> <li>I will play basketball at 6 pm at the University</li> <li>I will read at 9 pm on my balcony</li> </ul> <p>Write down a list of implementation intentions for all the habits you wish to develop and place the list somewhere you'll see every day (ex. on your bedroom door or the lock screen of your phone).</p> <h4>Habit Stacking</h4> <p>Habit stacking is a technique in which you consistently perform some behavior immediately after another behavior. The technique leverages the Diderot effect: the human phenomenon where one purchase leads to another (ex. buying a new desk prompts you to get a new keyboard, mouse, and monitor), and attempts to apply it to habits. Habit stacking works by explicitly tieing behaviors together. Say you currently have the habit of making coffee every morning but also wish to develop a habit of writing. You would use habit stacking by affirming "After I make coffee, I will sit down at my desk and write." The key behind habit stacking is to tie the desired behavior to something you already do. The format of habit stacking is:</p> <p>After [CURRENT HABIT], I will [NEW HABIT]</p> <p>Think of habit stacking as the connections between implementation intentions.</p> <h4>Habit Scorecard</h4> <p>You need to be aware of your habits before you can change them. The habit scorecard is a system for identifying and categorizing your existing habits as bad, neutral, or good. To create your scorecard, keep a pen and paper on you for an entire day and write down everything you do (ex. wake up, check phone, make coffee, shower, etc). At the end of the day, mark each behavior as either bad, neutral, or good.</p> <p>Example:</p> <ul> <li>Wake up - Neutral</li> <li>Check phone - Bad</li> <li>Make Coffee - Neutral</li> <li>Shower - Good</li> </ul> <p>To determine whether or not a habit is good, ask yourself it's helping you become the type of person you wish to be.</p> <h4>Make it Invisible</h4> <p>To break a bad habit, remove the cues associated with that habit from your environment. This is an inversion of the first law of behavior change.</p> <p>"Make it obvious" → "Make it invisible"</p> <p>To watch less YouTube, uninstall the app from your phone. To eat less sugar, remove all the sweets from your fridge. The purpose of "Making it invisible" is to eliminate the times when a behavior is initialized without intent - unlocking your phone and mindlessly clicking the YouTube Icon or opening your fridge and catching your eye on the KitKat wrapper.</p> <h3>The Behavior Function</h3> <p>Behavior is a function of a person in their environment:</p> <p>B = f(P,E)</p> <p>Put more precisely, our behavior results from the combination of our personal traits and the environment we inhabit. We react differently than others to the same environment depending on our relationships with that environment. The living room couch could be where one person relaxes or where another does their most intellectually demanding work. When you try to develop a new habit in a familiar environment, you have to contend with the habits that you've already developed in that environment. It's more difficult to develop a habit of reading in the area where you play video games than it is in a new environment that you've yet to connotate with any habits. You can't easily modify the P in f(P,E), but you can change the E. Build new habits in new environments, and whenever possible, avoid mixing the environment of one habit with another.</p> <h4>The Influence of Vision</h4> <p>The average human has 11 million sensory receptors in their body, 10 million of which are dedicated to sight. Because of this, visual cues in our environment are the most effective at influencing our behavior. When designing your environment, make the cues for good habits highly visible and the cues for bad habits invisible. Supermarkets already leverage the influence of vision by placing brand name products with the highest margins at eye level, while hiding away their lower margin alternatives in hard to reach places. A small shift in what you see can lead to a big shift in what you do.</p> <h4>Environment &gt; Discipline</h4> <p>During the Vietnam War it was discovered that 35% of US soldiers stationed in Vietnam had tried heroin and that 20% were addicted. However, upon returning to the US, only 5% became re-addicted within a year and only 12% relapsed within three years. This contradicted the commonly held belief at the time that heroin addiction was permanent and irreversible. State-side, 90% of heroin users become re-addicted when they return home from rehab. In the presence of a radically new environment devoid of the cues associated with their addiction, US soldiers quit heroin almost overnight.</p> <p>"Disciplined" people do not possess heroic willpower and self-control. Instead, they structure their lives in a way that reduces their exposure to negative temptations. It's easier to practice self-restraint when you don't have to use it very often. Anyone can become "Disciplined" if they architect their environment in such a way that the cues associated with bad habits are invisible and the cues associated with good habits are clear and abundant. We all have a finite amount of willpower. The more you have to practice self-restraint, the less you will be able to resist the next temptation. Each act of self-restraint drains your tank of willpower. Instead of spending energy on self-restraint, invest in eliminating temptations from your environment. Self-control is a short-term strategy, environment design is a long-term one.</p> <h3>Make it Attractive</h3> <p>The second law of behavior change is "Make it attractive". The more attractive an opportunity is, the more likely it will become habit-forming.</p> <h4>The Outdated Utility of Dopamine</h4> <p>Dopamine is the chemical in our brain that causes the feeling of pleasure. The pleasure from eating calorically dense foods and having sex motivated our ancestors to take actions that were essential for the survival of the human race. But we no longer live in a resource-scarce environment. We have the brains of our ancestors, but temptations that they could have never imagined. It's no surprise that over 40% of adults in the US are obese; food is abundant, yet our brains continue to crave it like it's scarce.</p> <h4>Exaggerated Reality</h4> <p>All animals, humans included, have certain behaviors wired into their brains through evolution. Baby herring gulls, for example, peck at the red spot on their parent's beak when they're hungry, signaling the parent to throw up food for the infant gull. Researchers found that exposing baby herring gulls to artificial beaks with extremely large red dots caused the infant gulls to peck like crazy. This is known as a supernormal stimuli, an exaggerated cue that elicits a strong reaction from the target creature. Modern industry is filled with supernormal stimuli that overwhelm our primitive instincts and motivate us to consume. Junk food has far more sugar than the naturally occurring foods that our brain's reward system evolved on. Video games have more visual and auditory stimulus than one would ever encounter in the wild. The modern environment is filled with exaggerated versions of reality that overwhelm our senses.</p> <h4>Anticipation Drives Action</h4> <p>Habits are dopamine-driven feedback loops. During the four steps involved with habit formation (cue, craving, response, reward), we experience two dopamine spikes, one during the craving phase and one during the reward phase. It's the dopamine spike that occurs during the craving phase that motivates us to act. The anticipation of the reward drives us to action, not the reward itself. Additionally, our brains have more neural circuitry allocated for wanting rewards than actually liking rewards. This is why the thought of Christmas morning as a child is more enjoyable than the morning itself, and why the research you perform on a product you want to buy brings you more enjoyment than the product itself.</p> <h4>Temptation Bundling</h4> <p>Temptation bundling is a strategy that leverages the motivating powers of anticipation in order to maximize the likelihood of performing good habits. Some habits, like eating healthy or reading, have lackluster immediate rewards. With temptation bundling, you follow-up a good habit with a habit that you enjoy, causing your brain to associate the good habit with the reward of the enjoyable habit. When performed enough times, you'll experience a dopamine spike in the craving phase of your good habit in anticipation of the reward from the enjoyable habit that follows. For example, only listening to podcasts at the gym will associate the good habit of exercise with the enjoyable habit of listening to your favorite podcast. Temptation bundling generally has the form of:</p> <p>After [HABIT I NEED], I will [HABIT I WANT].</p> <ul> <li>After I get on the spin bike, I will start listening to my favorite podcast.</li> <li>After I finish all my homework, I will play my favorite video game.</li> <li>After I open the book I'm reading, I will drink a cup of coffee.</li> </ul> <h4>Leverage the desire to belong</h4> <p>Humans are social creatures with an innate desire for belonging. This innate desire is an artifact of our evolution as a species; early humans that belonged to a group had a far greater chance of survival than the lone wolf. We communicate this desire for belonging through imitation. In particular, we imitate three groups of people</p> <ol> <li>The close (friends and family)</li> <li>The many (people in our society)</li> <li>The powerful (people with status and prestige)</li> </ol> <p>The first group we imitate are those closest to us. As an infant, you copy the sounds your parents make, as a child, you copy how your peers play with one another, and as an adult, you copy how your colleagues work. The closer you are to someone, the more likely you will be to imitate their behaviors. As Jim Rohn says "You are the average of the five people you spend the most time with."</p> <p>One study that tracked 12 thousand people for 32 years found that the chances of someone becoming obese increased by 57% if they had a friend who become obese. On the other end of the spectrum, groups of exceptional people often achieve disproportionately exceptional results. A few such examples include the friendship of Warren Buffet and Charlie Munger, brothers Patrick and John Collison, and the Paypal Mafia. One of the most effective things you can do to build better habits is to join a culture where your desired behavior is the normal behavior. Surround yourself with people who already have the habits that you're trying to develop.</p> <p>The second group we imitate is society as a whole. This large-scale imitation can be observed through the fashion trends or popular music of a certain era. Humans have a strong urge to conform to the behaviors and preferences of the masses. A famous experiment that illustrates the pressures of social conformity was performed by psychologist Solomon Asch. In the experiment, a test subject would unknowingly be placed in a room full of actors, where they would be given 2 cards with lines of various lengths. The subjects were then asked to identify which line on the right card had the same length as the line on the left card.</p> <p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/Asch_experiment.svg/270px-Asch_experiment.svg.png" alt=""></p> <p>The actors would purposely select the wrong answer (B). When the test subject was in the room with one other actor, they would select the correct answer, even after seeing the actor's incorrect selection. But as the number of actors increased, all of whom selected the same wrong answer, the subject would begin to also select the wrong answer in order to conform. The lesson learned from this experiment is that the normal behavior of the tribe often overpowers the desired behavior of the individual. We will do or say something we know to be wrong in order to fit in. We do this because the reward to fit in is often greater than the reward of being right.</p> <p>The last group of people we imitate is the powerful. We do so in hopes that our imitation will bring about the prestige and power that this group possesses. The desire for prestige and power is again rooted in our evolution as a species; individuals with more power and status have historically had greater access to resources, and thus had a greater chance of survival and finding a mate. If a behavior can get us approval, respect, and praise, we find it attractive.</p> <h4>Underlying Motives</h4> <p>Every behavior has a surface-level craving and a deeper underlying motive. For example, a surface-level craving could be "I want a piece of pizza". The underlying motive for this craving is the desire to obtain food and water. Other underlying motives of human behavior include:</p> <ul> <li>Conserve energy</li> <li>Find love and reproduce</li> <li>Connect and bond with others</li> <li>Win social acceptance and approval</li> <li>Reduce uncertainty</li> <li>Achieve status and prestige</li> </ul> <p>Our cravings are just arbitrary manifestations of a deeper underlying motive. Our brains didn't evolve with a desire to smoke cigarettes or to check Instagram or to play video games. These are simply manifestations of our underlying motives to reduce uncertainty, win social acceptance and approval, or achieve status and prestige. Modern-day products don't create new motivations, they latch onto the underlying motives of human behavior.</p> <ul> <li>Find love and reproduce = Using Tinder</li> <li>Connect and bond with others = Browsing Facebook</li> <li>Win social acceptance and approval = Posting on Instagram</li> <li>Reduce uncertainty = Searching on google</li> <li>Achieve status and prestige = Playing video games</li> </ul> <p>Our habits are modern solutions to ancient desires.</p> <h4>Predictive, Not Reactive</h4> <p>Our behavior is heavily dependent on how we interpret the events that happen to us, not necessarily the objective reality of the events themselves. Two people may experience the exact same event but react in entirely different ways. This is because right before reacting to the event, the two individuals make different predictions on the outcome of their actions. These predictions are based on their past experiences and world-views. In response to being offered a cigarette, one person may predict that smoking it will relieve their anxiety, while the other may predict that it will create a foul odor and increase their odds of getting lung cancer. The difference in these two predictions is what causes one individual to smoke and the other not to.</p> <h4>Make it Unattractive</h4> <p>The inversion of the second law of behavior change is to "make it unattractive". To break a bad habit, you need to eliminate the craving for that bad habit. This can be done in one of two ways:</p> <ol> <li>Exposing how the bad habit inadequately addresses the underlying motive that caused its formation.</li> <li>Replacing the bad habit with a good habit that more effectively addresses the underlying motive.</li> </ol> <p>As mentioned previously, habits are just attempts to address the underlying motives of human behavior. To break a bad habit, analyze how the behavior associated with a bad habit inadequately addresses the underlying motive. For example, someone who smokes to calm their nerves may analyze their habit and realize that smoking does not relieve their nerves, it destroys them. Clearly describe the negative consequences of your bad habits to make them as unattractive as possible. In addition, find a good habit that addresses the same underlying motive as the bad habit. In the case of smoking to calm one's nerves, a healthy alternative could be running, which increases one's physical fitness and cardiovascular health.</p> <h4>How To Enjoy Hard Habits</h4> <p>Simple shifts in mindset can have profound effects on how we interpret the world. Instead of telling yourself that you "have to" do a hard habit, tell yourself that you "get to" do a hard habit. This simple shift in words changes your interpretation of the habit, highlighting how the difficult behavior is an opportunity instead of a burden. Telling yourself "I have to go to the gym" subtly implies that going to the gym is an inconvenient and painful experience. But telling yourself "I get to go to the gym" implies that going to the gym is a positive opportunity to improve one's own health and physique.</p> <p>"I have to" → "I get to"</p> <h3>Make it Easy</h3> <p>The third law of behavior change is "Make it easy." Easy behaviors require less motivation to act and are more likely to be performed consistently.</p> <h4>Quantity Over Quality</h4> <p>A professor at the University of Florida divided his photography class into two groups: the quality group and the quantity group. The quantity group was told that they would be graded solely on the amount of work they produced. The more photos they took, the higher the grade they would receive. The quality group was told that they would be graded solely on the excellence of their work. This group only needed to produce a single image for the entire semester. At the end of the semester, the professor was surprised to find that all of the highest quality photos had been submitted by the quantity group. Throughout the semester, the quantity group was constantly experimenting with different techniques, incrementally improving the quality of the photos they captured. Meanwhile, the quality group sat around speculating about what the perfect photo might look like. The iterative technique of the quantity group produced far greater results than the speculation and theorizing of the quality group. If you want to master a habit, the key is to start with repetition, not perfection.</p> <p>"The best is the enemy of the good." - Voltaire</p> <h4>Motion vs Action</h4> <p>Motion is when you're planning, strategizing, or learning. Action is when you're performing a behavior that actually delivers results. Reading a book on programming is motion. Actually programming is action. Researching effective core workouts is motion. Working out is action. We're attracted to motion because it gives us the illusion of progress without the risk of failure. Action is more difficult than motion, but it's also more rewarding.</p> <h4>Neuroplasticity</h4> <p>Neurons that fire together wire together - Hebb's Law</p> <p>The more you repeat an activity, the more the structure of your brain changes to become efficient at that activity. This is because of a phenomenon known as neuroplasticity: the brain's ability to modify and adapt both structure and function in response to experiences. As you repeat some behavior, the connections between the neurons associated with that behavior strengthen. This repetition leads to clear physical changes in the brain. For example, the cerebellum (a component of the brain that's critical for precise muscle movements) is much larger in musicians than in non-musicians.</p> <h4>How long does it take to build a new habit?</h4> <p>Habits are formed through repetitions, not time. The amount of time you've been performing a habit is less important than how many times you've performed the habit. There will come a point where you've performed so many repetitions of a particular behavior that you no longer have to think about each step involved. This is known as the point of automaticity. Reaching the point of automaticity, and thus forming a habit, is dependent on how many repetitions you've performed, not how long you've been performing them.</p> <h4>The Law of Least Effort</h4> <p>Our brains are energy conservation machines, constantly searching for ways to minimize the amount of effort they expend. When deciding between two similar options, people will naturally gravitate towards the easier one. Our brains have evolved to preserve energy and avoid expenditure whenever possible. This is why telling yourself to just "try harder" is rarely effective for developing new habits; It goes against the grain of human nature.</p> <h4>Eliminate Friction</h4> <p>Every habit is an obstacle to getting what you really want. Exercising is an obstacle to getting fit. Writing is an obstacle to thinking clearly. You don't want the habit itself, you want what the habit delivers. The greater the obstacle, the more friction there is between you and your desired state. The key to building new habits is reducing this friction as much as possible. Trying to perform hard habits by increasing your motivation is like trying to force more water through a bent hose. Instead, design your environment to reduce the friction associated with performing your habits, unbending the hose. You're more likely to start working out if you join a gym that's on the way to work. You're more likely to eat healthier if your fridge is filled with meat and vegetables, whilst being void of sugar. Removing the friction associated with your habits allows you to do more with less time and effort. Think addition by subtraction.</p> <h4>Prime for next use</h4> <p>Prepare your environment for performing a particular habit ahead of time. If you want to start going to the gym every morning, pack your bag the night before. If you want to eat healthier, wash and pre-cut your fruits and vegetables. If you want to read before bed, place a book on your pillowcase every morning. Preparing your environment for the performance of a habit serves as both a reminder and a mechanism for making the habit as convenient as possible.</p> <h4>Decisive Moments</h4> <p>Every day we encounter decisive moments that deliver an outsized impact. Choosing to order takeout or to cook dinner. Choosing to go to the gym or to stay home and watch tv. Choosing to study or to play video games. These split-second decisions result in outcomes that consume hours of our time and drastically change the course of our days. The difference between a good day and a bad day is the result of the choices we make during decisive moments.</p> <h4>The 2 Minute Rule</h4> <p>When you start a new habit, it should take 2 minutes or less. We often overcommit to new habits from the get-go. We promise ourselves that we're going to run 5k a day, or read for an hour. But when the time rolls around to perform our new habit, we feel overwhelmed by the lofty goals we've set for ourselves, and end up doing nothing at all. The 2 minute rule solves this problem by leveraging the fact that it's easier to continue what you're doing than to start something new. The hardest part of performing a new behavior is starting it. By making your new habit as small as possible, you increase the likelihood of performing the habit in the first place. Instead of committing to an hour of reading each day, commit to a page. You'll be more likely to start reading, and will inevitably read more. Remember that something is better than nothing. Doing 10 pushups is better than doing none at all. Reading 1 page is better than reading 0. Action, no matter how small, is infinitely superior to inaction.</p> <h4>Ritualize The Process</h4> <p>The more you ritualize the beginning of a process, the more likely you are to slip into a state of focus for performing the process. Examples of this include basketball players who perform the same dribbling ritual before every free throw.</p> <h4>Make it Difficult</h4> <p>"Sometimes success is less about making good habits easy and more about making bad habits hard." - James Clear</p> <p>The inversion of the third law of behavior change is to "Make it Difficult". Make your bad habits difficult by using commitment devices: actions you take in the present that control how you behave in the future. A good example of a commitment device is to pay in advance, incentivizing you to follow through with whatever service you paid for since not doing so has an associated cost.</p> <h4>Onetime actions with longtime rewards</h4> <p>There exists some onetime actions you can take that produce consistent long-term rewards. Here are a few examples:</p> <ul> <li>Buying smaller plates to reduce caloric intake</li> <li>Purchasing a comfortable mattress to increase sleep quality</li> <li>Installing blackout curtains to increase sleep quality</li> <li>Turning off notifications to reduce distractions</li> <li>Getting a dog to increase companionship in your life</li> <li>Asking service providers to lower your bills to save money</li> <li>Buying a high-quality chair or standing desk to improve your posture</li> </ul> <p>Taking as many of these actions as possible is an easy way to positively impact your future self.</p> <h3>Make it Satisfying</h3> <p>The fourth law of behavior change is to "Make it Satisfying". You're more likely to repeat a behavior if the experience is pleasurable. The previous three laws, "Make it obvious", "Make it attractive" and "Make it easy" increase the likelihood of a behavior being performed. The fourth law increases the likelihood of a behavior being repeated.</p> <h4>Immediate-return environment to delayed-return environment</h4> <p>Humans have been around for approximately 200 thousand years. And for the first 195 thousand of those years (97.5% of our existence), we inhabited an immediate-return environment. Our thoughts and actions were focused exclusively on the present. Our primary concerns revolved around what to eat, where to sleep, and how to avoid predators. Our wild environment necessitated an intense focus on the immediate results of our actions. But in the last 5 thousand years, civilizations were born and we transitioned to a delayed-return environment. An environment where the rewards of our actions may not be delivered for days, weeks, months, or even years. We work today but don't get paid for 2 weeks. We study now but don't graduate for 4 years. Innovation in science and technology has minimized our risk of immediate peril and made acting on behalf of our future selves a necessity. The problem is that this environmental change from an immediate-return environment to a delayed-return environment has happened faster than our brains can adapt. We have evolved to prefer rewards that are guaranteed in the present over possible rewards in the future. This is why relying on willpower alone to make the right decision doesn't work. It's human nature to gravitate towards the choice that delivers immediate satisfaction.</p> <h4>The Paradox of Instant Gratification</h4> <p>Bad habits almost always feel good in the moment but bad in the future. Conversely, good habits often feel difficult in the moment but good in the future. It feels good to get drunk on Friday night, but the morning that follows is absolutely dreadful. A hard workout may be difficult in the moment, but it leaves you feeling calm and revitalized afterwards. Always be suspicious of immediately pleasurable behaviors, for they are likely to be detrimental in the long term. From the wise words of Frédéric Bastiat "It almost always happens that when the immediate consequence is favorable, the later consequences are disastrous, and vice versa.."</p> <h4>The Road Less Traveled</h4> <p>The abundance of resources in modern society coupled with the advent of the smartphone has turned most of us into hopeless dopamine fiends. We constantly seek instant gratification in the form of sugary snacks or likes on social media and our environment is capable of providing both indefinitely. But the road less traveled, and far more rewarding, is the road of delayed gratification. Individuals who can tolerate delayed gratification are better at responding to stress, have lower levels of substance abuse, and are less likely to become obese. The road less traveled is the one that requires persevering through the struggles of the present, knowing that they'll make you better off in the future.</p> <h4>Treat Yourself</h4> <p>Some of the most powerful habits, like reading and exercise, can take months to deliver noticeable results. To combat demotivation and prolong your patience, end your difficult habits with a behavior you find satisfying. An example could be rewarding yourself with a delicious smoothie after a hard workout. The only thing to keep in mind is that the reward should reinforce the identity that is associated with the habit. A healthy smoothie reinforces the identity created by working out: a healthy person. But if you were to end your hard workouts with a Big Mac, you'd be conflicting your desired identities. This self-rewarding will only be necessary until you start to reap the long-term rewards of your good habits. The improved aesthetic and increased wellbeing that comes after months of exercising will be satisfying enough to maintain the habit, without the reward of a smoothie.</p> <h4>Track Your Progress</h4> <p>Progress is the most effective form of motivation. Use a habit tracking app to track your progress and maintain motivation for your habits. The visual indicators in habit tracking applications make progress more enjoyable. Whether it be the flaming streak in Duolingo or the checkmarks in your habit tracker, these subtle visual cues make the act of performing and maintaining a habit more satisfying. Habit tracking applications also act as motivation for days where you don't feel like performing the habit since doing so would end your streak or pause your progress. Lastly, habit tracking shifts your focus from the end goal to the process that will get you there.</p> <h4>Never Miss Twice</h4> <p>A simple mental trick for maintaining habits is to tell yourself that you'll never miss twice. Life is full of surprises and we will inevitably miss performing our habits one day. The trick is to not let this slip up get you down. Habit building is not an all-or-nothing endeavor. When you miss a habit, focus on rebounding as soon as possible. It's okay to miss a workout every once and a while, just don't miss two in a row.</p> <h4>Bad &gt; None</h4> <p>A bad workout is infinitely better than no workout at all. In fact, the days where you perform a habit when you don't feel like it are the most valuable. By doing so you are identifying as the type of person who performs habit X through thick and thin. Anyone can work out when they're well-rested and feeling energized. But it takes a special commitment to hit the gym on 2 hours of sleep. Cherish the days where you don't feel like performing a habit, for they provide an opportunity to strengthen your desired identity like no other.</p> <h4>Naïve Metrics</h4> <p>"When a measure becomes a target, it ceases to be a good measure" - Charles Goodhart. When it comes to tracking habits, keep the bigger picture in mind. If you become too obsessed with the metric you've chosen to track your progress, you might lose sight of why you're doing the habit in the first place. Choosing weight as a measurement of health could have inadvertent effects, particularly if you engage in unhealthy behaviors like taking fat-loss pills or starving yourself to lose weight. Remember why you're doing performing a habit in the first place and don't get too obsessed with the metrics.</p> <h4>Make it Unsatisfying</h4> <p>The inversion of the fourth law of behavior change is to make it unsatisfying. The more painful a behavior the more likely we are to avoid it. The trick to stopping our bad habits is to make their consequences occur as soon as possible, be clearly visible, and be more painful than their worth. Imagine if you were instantly issued a speeding ticket every time you sped. This would surely squash your habit of joyriding, as the cost of the ticket would far outweigh the momentary thrill of speeding.</p> <h4>Accountability Partners</h4> <p>Create clear and immediate consequences for your bad habits through the use of a habit contract. A habit contract is a written or verbal agreement in which you state your commitment to a particular habit and the punishment that will occur if you don't follow through with it. Get a friend or family member to sign off on the contract with you and hold you accountable. If the habit contract seems overly formal, try to at least find an accountability partner who will challenge you to stick to your habits. Knowing that you're being monitored and judged can be a powerful motivator.</p> <h4>Listen to Yourself</h4> <p>We all have different natural abilities and inclinations. The most important piece of advice when building new habits is to pick habits that you want to develop, not one's that you're told you should develop. Almost every goal has different variations of habits that lead to it. You will be most likely to develop and stick to a habit if you shape it to match your inclinations. For example, say you want to get fit and you enjoy rock climbing but hate lifting weights. You'll be much more likely to achieve and maintain the habit you enjoy, rock climbing, than the one you dislike, weightlifting. Pick the right habit and progress is easy, pick the wrong habit and life is a struggle.</p> <h4>How to find your habits</h4> <p>Habits that are aligned with your natural inclinations will be easier to develop and more enjoyable to practice. To find these habits, try answering the following questions:</p> <ol> <li> <p>What feels like fun to me but like work to others?<br> Enjoying a habit is the biggest indicator that it aligns with your natural inclinations. If you enjoy a habit that feels like work to others, you'll likely excel in the domain associated with that habit. It's extremely difficult to compete with someone who's having fun.</p> </li> <li> <p>What makes me lose track of time?<br> Losing track of time when engaged in a behavior is an indication that you've entered a state of flow. It's nearly impossible to enter this state if you don't find the task at least moderately satisfying.</p> </li> <li>Where do I experience greater returns than the average person?<br> We constantly compare and contrast our outcomes with those around us. If your progress in a domain is greater than that of your peers, you're going to feel motivated and be more likely to stick with and excel in that domain.</li> <li>What comes natural to me?<br> What's the first thought that came to your mind after reading the above question?</li> </ol> <h4>Escape competition by combining your interests</h4> <p>If you're unable to find a domain that caters to your natural inclinations, create one. Scott Adams, the creator of the comic series Dilbert, combined his above-average drawing skills with his above-average comedian skills to develop an exceptional skill of creating funny comics. Adams attributes the profound success of Dilbert to the unique intersection of comedy and art.</p> <p>The modern environment is incredibly competitive. Almost every mainstream domain is populated with millions of competitors fighting for supremacy. Instead of trying to win at one of these ultra-competitive domains, escape competition by combining your skills to create a new domain. You are far more likely to succeed at the intersection of your two favorite hobbies than either one alone. And the rarer the combination, the greater the odds your work will stand out.</p> <h4>The Goldilocks Zone</h4> <p>We experience maximum motivation when the task we are performing is just on the outskirts of what we're capable of doing. If the task is too easy, well quickly become bored and demotivated. If the task is too difficult, we'll feel overwhelmed and quit. The key to sustaining motivation is to take on tasks that lie in the middle, in the "Goldilocks Zone". As you practice and improve you'll need to constantly adjust the goldilocks zone to sustain the challenge and prevent things from becoming too easy, whilst also not over-adjusting to make them too hard.</p> <h4>Embrace boredom</h4> <p>The greatest threat to success is boredom, not failure. To fail you have to be acting and thus, making progress. But what follows boredom is inaction, and with inaction progress is impossible. No matter how much you enjoy a particular habit there will come days where you don't feel like doing it. One day-off quickly turns into a week, which then cascades into a month. The difference between professionals and amateurs is that professionals show up every day, even when they don't feel like it, even when they're bored. To become truly great at something you have to embrace boredom.</p> <h4>Avoiding Complacency</h4> <p>It's easy to become complacent once you've reached the point of automaticity with a habit. But if you wish to attain and maintain mastery in any domain, you'll need to continually improve. Studies have shown that once an individual masters a skill, their performance declines over time. This is because they can now perform the skill on autopilot and are no longer cognoscente of whether or not they are actually performing the skill optimally. For example, you may have mastered the skill of dribbling a basketball and can do so without thinking about it, but over time you may begin to use your palm instead of the pads of your fingers, limiting your overall control of the ball. To master a domain and avoid complacency, you need to reflect on and review your performance, and to reflect and review, you need to measure your outcomes. It takes effort to record your performance and periodically reflect on it, but it is often the distinguishing factor between those who continually develop and those who plateau.</p> <h4>Keep Your Identity Small</h4> <p>The more you let your beliefs define you, the less adaptive you'll be to change. Life is full of surprises and you never know when unforeseen circumstances alter what you can and can't do. If your identity is brittle, changes beyond your control may break you. If your main identity is being a programmer and new developments in artificial intelligence render human programmers obsolete, you'll have an identity crisis. Instead of identifying with your particular role, identify with the fundamental aspects of your role. "I'm a programmer" becomes "I'm a problem solver". The more fundamental and wide-reaching your identity is, the less susceptible it is to change.</p> <p>"Men are born soft and supple; dead, they are stiff and hard. Plants are born tender and pliant; dead, they are brittle and dry. Thus whoever is stiff and inflexible is a disciple of death. Whoever is soft and yielding is a disciple of life. The hard and stiff will be broken. The soft and supple will prevail." --Lao Tzu</p> https://tedneward.github.io/Research/thinking/crucial-conversations/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/crucial-conversations/index.html <p><em>(by Kerry Patterson, Joseph Grenny, Ron McMillan, and Al Switzler)</em></p> <h3>Chapter 1: What is a Crucial Conversation?</h3> <ul> <li>Crucial conversations are when opinions vary, stakes are high, and emotions run strong.</li> <li>Such conversations have results that can have a huge impact on your quality of life.</li> <li>When we face crucial conversations, we can avoid them, or we can face them and handle them poorly, or we can face them and handle them well.</li> <li>Countless generations of genetic shaping drive us to handle crucial conversations with flying fists and fleet feet, not intelligent persuasion and gentle attentiveness.</li> <li>Such conversations are extraordinarily difficult also because they are spontaneous, are difficult to rehearse for, and cause us to act in self-defeating ways.</li> <li>The Law of Crucial Conversations says that the key skill of effective leaders, teammates, parents, and loved ones is the capacity to skillfully address emotionally and politically risky issues.</li> <li>In the workplace, the individuals who are the most influential, or who can get things done and at the same time build on relationships, are those who master their crucial conversations.</li> <li>Most leaders think organizational productivity and performance are simply about policies, processes, structures, or systems. But really it's about employee behavior, where crucial conversations beget accountability.</li> <li>In every relationship, the partners argue about important issues; but not everyone breaks up. It's how you argue that matters.</li> <li>People fall into three categories: Those who digress into threats and name-calling, those who revert to silent fuming, and those who speak openly, honestly, and effectively.</li> <li>The negative feelings we hold in, the emotional pain we suffer, and the constant battering we endure as we stumble our way through unhealthy conversations slowly eat away at our health.</li> </ul> <h3>Chapter 2: Mastering Crucial Conversations</h3> <ul> <li>Those who master crucial conversations avoid the Fool's Choice, where we think we must choose between telling the truth and keeping a friend, or between candor and kindness.</li> <li>When it comes to risky, controversial, and emotional conversations, skilled people find a way to get all relevant information out into the open. We call this free flow of meaning dialogue.</li> <li>Our opinions, feelings, theories, and experiences of a topic fill our personal pool of meaning. People who are skilled at dialogue do their best to make it safe for everyone to add their meaning to the shared pool.</li> <li>As the pool of shared meaning grows, it exposes individuals to more accurate and relevant information, and so they make better choices. It is a measure of the group's intelligence.</li> <li>The pool of shared meaning is also the birthplace of synergy. From a free flow of meaning, we can create a whole that is truly greater than the sum of its original parts.</li> <li>Because the meaning is shared, and everyone takes part in the free flow of meaning, people act on whatever decisions they make with both unity and conviction.</li> <li>The time you spend up front establishing a shared pool of meaning is more than paid for by faster, more unified, and more committed action later on.</li> <li>Whenever we find ourselves arguing, debating, running away, or otherwise acting in an ineffective way, it's because we don't know how to share meaning.</li> </ul> <h3>Chapter 3: Start with Heart</h3> <ul> <li>The first step of achieving the results that we really want is to fix the problem of believing that others are the source of all that ails us.</li> <li>People who are best at dialogue realize that the only person we can continually inspire, prod, and shape with any degree of success is the person in the mirror.</li> <li>Skilled people Start with Heart; that is, they begin high-risk discussions with the right motives, and they stay focused no matter what happens.</li> <li>These people maintain focus by: sticking with their goals and what they want; and not making Fool's Choices, and so they believe that dialogue is an option regardless of the circumstances.</li> <li>When faced with pressure and strong opinions, we often stop worrying about the goal of adding to the pool of meaning and start looking for ways to win, punish, or keep the peace.</li> <li>Our motives usually change without any conscious thought on our part. To move back to motives that allow for dialogue, we must step away from the interaction and look at ourself.</li> <li>Ask what you want for yourself, what you want for others, and what you want for the relationship. Then ask how you would behave if you really wanted those results.</li> <li>These questions remind us of our goal and original purpose, and also affect our physiology by sending blood to the brain which in turn keeps us focused.</li> <li>Restructure the Fool's Choice: Clarify what you don't want, add it to it what you do want, and then begin searching for healthy options to bring you to dialogue.</li> </ul> <h3>Chapter 4: Learn to Look</h3> <ul> <li>We have trouble watching both content and conditions in a crucial conversation: We are so caught up in what we're trying to say that we don't see what's happening to ourselves and others.</li> <li>In a conversation, look for the moment when it turns crucial, signs that people don't feel safe (silence or violence), and your own Style under Stress.</li> <li>To spot when a conversation turns crucial, look to yourself for a physical signal, a change in emotions, or a behavioral cue.</li> <li>People who are gifted at dialogue keep a constant vigil on safety. They pay attention to the content and they watch for signs that people are becoming fearful.</li> <li>Dialogue calls for the free flow of meaning, and nothing stops the flow of meaning like fear. Fear can cause you to both push too hard and to withdraw.</li> <li>People rarely become defensive because of what you're saying, but moreso because they no longer feel safe. The problem is not the content of your message, but the condition of the conversation.</li> <li>Watching for safety violations allows you to restore the peripheral vision that narrows when you feel genuinely threatened, and to reengage your brain and its centers of higher reasoning.</li> <li>Do not allow the actions of those who feel unsafe to beget fight or flight. Instead, recode the signs of violence or silence that people feel unsafe, and do something to create safety.</li> <li>Silence is characterized by <em>masking</em> (such as sarcasm, sugarcoating, or couching), <em>avoiding</em> sensitive subjects, or <em>withdrawing</em> from the conversation altogether.</li> <li>Violence consists of <em>controlling</em> (such as cutting others off, overstating facts, speaking in absolutes, etc.), <em>labeling</em> to dismiss others under a stereotype or category, or <em>attacking</em> through belittling or threatening.</li> <li>You must also self-monitor. Pay close attention to what you're doing and the impact it's having, and then alter your strategy if necessary. Check whether you're having a good or bad impact on safety.</li> </ul> <h3>Chapter 5: Make It Safe</h3> <ul> <li>The <em>good</em> fix safety issues by sugarcoating their message, but this avoids the real problem. The <em>best</em> don't play games; they step out of the content of the conversation, make it safe, and then step back in.</li> <li>Crucial conversations often go awry not because others dislike the content of the message, but because they believe the content suggests a malicious intent, thereby subverting safety.</li> <li>Mutual Purpose is required to begin dialogue: Where others perceive that you're working toward a common outcome, and that you care about their goals, interests, and values. And vice versa.</li> <li>When Mutual Purpose is at risk, we end up in debate. Other signs include defensiveness, hidden agendas, accusations, and circling back to the same topic.</li> <li>If you enter a conversation to get what you want, you will appear critical and selfish. Instead, find the Mutual Purpose: to draw someone willingly into a crucial conversation, see their point point of view.</li> <li>Mutual Respect is required to continue dialogue: If someone perceives disrespect in the conversation, it is no longer about its original purpose, but about defending dignity.</li> <li>When Mutual Purpose is at risk, emotions become charged and fear turns to anger. Then people resort to name-calling, yelling, and making threats.</li> <li>Recognize that we all have weaknesses; this creates a kinship and connection to others, which creates a Mutual Respect and eventually enables us to stay in dialogue with virtually anyone.</li> <li>When you've made a mistake that hurts others, start with a sincere apology. You must give up saving face, being right, or winning in order to achieve healthy dialogue and better results.</li> <li>Contrasting is a skill that first addresses others' concerns that you don't respect them or that you have a malicious purpose, and then confirms your respect or clarifies your real purpose.</li> <li>Contrasting is not apologizing. It is not a way of taking back something we've said that hurt others' feelings, but a way of ensuring that what we said didn't hurt more than it should have.</li> <li>Use contrast to provide context and proportion for your words, and to also bolster safety when you are about to drop into the pool of meaning something that could cause a splash of defensiveness.</li> <li>If you are in the middle of a debate because each side has a different purpose, seek to create Mutual Purpose using the four skills in the acronym CRIB.</li> <li><em>Commit</em> to seek a mutual purpose: Commit to stay in the conversation until we invent a solution that serves a purpose we both share. Verbalize this even if the other person seems committed to winning.</li> <li><em>Recognize</em> the purpose behind the strategy: Separate strategies, or what you're asking for, from the purpose, or what you actually want. This can create new options that can serve both of your interests.</li> <li><em>Invent</em> a mutual purpose: If you cannot discover a Mutual Purpose for dialogue, invent one by moving to more encompassing goals. Find an objective that is more meaningful or more rewarding.</li> <li><em>Brainstorm</em> new strategies: Once committed to finding something that everyone can support and surfaced what you really want, you'll no longer be spending your time on unproductive conflict.</li> <li>Before a crucial conversation begins, think about which skills will help you must. A little progress can produce a lot of benefit.</li> </ul> <h3>Chapter 6: Master My Stories</h3> <ul> <li>Others don't create emotions for you; you create your own emotions. And then, when you create strong emotions, you either find a way to master them or fall hostage to them.</li> <li>The worst at dialogue treat their emotions as the only valid response, falling hostage to their emotions without even realizing it.</li> <li>The best at dialogue act on their emotions, influencing and often changing their emotions by thinking them out, thereby making it possible to choose behaviors that create better results.</li> <li>After we observe what others do and before we feel an emotion, we tell ourselves a story, adding meaning, guessing at motive, and adding judgment before responding with emotion.</li> <li>Stories are our interpretations of the facts, providing a rationale for what's going on. We use them to explain the <em>why</em>, <em>how</em>, and <em>what</em>.</li> <li>Any set of facts can be used to tell an infinite number of stories; by taking control of our stories, they won't control us.</li> <li>To take control, first take an honest look at your behavior. If an unhelpful story is driving you to silence or violence, stop and consider how others would see your actions.</li> <li>Next, stop and think about your feelings. Most people are emotionally illiterate; expand your vocabulary to better know how you feel, and more understand what is going on and why.</li> <li>Then, question your feelings, thereby allowing you to question your stories. This will allow you to develop new stories, and thereby develop new feelings.</li> <li>Separate stories from facts. Test whether you can see or hear what you're calling a "fact," and look in stories for words that express judgments and attributions that create strong emotions.</li> <li>When we feel a need to justify our ineffective behavior or disconnect ourselves in bad results, we tend to tell ourselves one of three "clever stories."</li> <li>"Victim stories" make us innocent while ignoring our own role in the problem; "villain stories" assume the worst motives or grossest incompetence for others while ignoring any possible good intentions or skills.</li> <li>These stories form a double standard: When we make mistakes, we tell a victim story; when others make mistakes, we tell a villain story.</li> <li>"Helpless stories" make us out to be powerless to do anything healthy or helpful, thereby justifying the action we're about to take.</li> <li>Villain stories and victim stories look back to explain the situation we're in, while helpless stories look forward to explain why we can't do anything to change our situation.</li> <li>We often tell clever stories to excuse ourselves of any responsibility, or to justify our actions when we consciously act against our own sense of what's right.</li> <li>Clever stories are incomplete and omit information about us, others, and our options. To add details, turn victims into actors, villains into humans, and the helpless into the able.</li> <li>These details create a <em>useful</em> story, which creates emotions that lead to healthy action, such as dialogue.</li> <li>When humanizing "villains," we relax our absolute certainty long enough to allow for dialogue, which is the only reliable way to discover others' genuine emotions.</li> </ul> <h3>Chapter 7: STATE My Path</h3> <ul> <li>The best at dialogue speak their minds completely and do it in a way that makes it safe for others to hear what they have to say and respond to it as well.</li> <li>To maintain safety in dialogue, be confident in what you say and your ability to say it without causing offense, and be humble to encourage others' input.</li> <li>Rely on STATE to discuss sensitive topics: State your facts, Tell your story, Ask for others' paths, Talk tentatively, and Encourage testing.</li> <li>Starting with facts shows that you are reasonable and rational; starting with a story could easily surprise and insult others, thereby killing safety and muting the facts.</li> <li>Lead others down your Path to Action, starting with the facts and then moving to your story. The story conveys the severity of the implications.</li> <li>Encourage others to share their facts, stories, and feelings, and be willing to abandon or shape your story as more information pours into the pool of shared meaning.</li> <li>Speaking in absolute and overstated terms decreases your influence, while the more tentatively you speak, the more open people become to your opinions.</li> <li>When you begin your story with a complete disclaimer and do use a tone that suggests you're consumed with doubt, you do the message a disservice.</li> <li>Invite opposing views. If no one speaks up, play devil's advocate and model disagreeing by disagreeing with your own view.</li> <li>You can argue as vigorously as you want for your point of view, provided you are even more vigorous at encouraging, or even pleading with, others to disprove it.</li> <li>When you find yourself being forceful in your views, stop and think about what you really want for yourself, others, and the relationship, and ask if your behavior is congruent with that.</li> </ul> <h3>Chapter 8: Explore Others' Paths</h3> <ul> <li>When someone retreats to the security of silence or violence, you can't force others to dialogue, but you can take steps to make it safer for them to do so.</li> <li>The cure to silence or violence is to get at the underlying source; this calls for genuine curiosity at a time when you're likely feeling frustrated or angry.</li> <li>To avoid overreacting to others' stories, ask yourself "Why would a reasonable, rational, and decent person say this?"</li> <li>When others are in silence or violence, we're joining their Path to Action <em>already in progress</em>. We must encourage them to share not their conclusions, but their observations.</li> <li>Helping others retrace their Path to Action helps us curb or emotion, and returns us to the facts and story behind the emotion, where feelings can be resolved.</li> <li>To encourage others to share, we must listen. We must employ the power listening tools of AMPP: Ask, Mirror, Paraphrase, and Prime.</li> <li>Ask: Stop filling the pool with your meaning, step back, and invite the other person to talk about his or her view.</li> <li>Mirror: Describe how the other person looks or acts, creating safety by using a tone that says that you're okay with them feeling how they're feeling.</li> <li>Paraphrase: Once you get a clue as to why the person is feeling the way he or she does, you can paraphrase to build additional safety.</li> <li>Instead of pushing too hard, ask what the other person wants to see happen. This moves his or her brain to problem solving and away from attacking or avoiding.</li> <li>Prime: When there is still no safety, offer your best guess as to what the other person is thinking or feeling. Only do this as a last resort.</li> <li>Remember that you are trying to understand the other person's point of view; not necessarily agree with it or support it.</li> <li>If you completely agree with the other person's path, say so and move on; don't turn an agreement into an argument.</li> <li>If you agree with what has been said but the information is incomplete build. Point out areas of agreement, and then add elements that were left out of the discussion.</li> <li>If you disagree, do not suggest that he or she is wrong, but suggest that you differ. Then share your path using the STATE skills.</li> </ul> <h3>Chapter 9: Move to Action</h3> <ul> <li>The end of a crucial conversation is risky because if you aren't careful about how you classify the conclusion and decisions, you can violate expectations.</li> <li>This manifests by people not understanding how decisions are going to be made, or by no decision ever being made.</li> <li>When the line of authority for making decisions is clear, it is that authority who decides what method of decision-making to employ.</li> <li>When the line of authority for making decisions is unclear, use your best dialogue skills to get meaning into the pool. Jointly decide how to decide.</li> <li>Make decisions by command, consult, vote, and consensus. These represent increased involvement, which increases commitment but decreases efficiency.</li> <li>With command decisions, either outside forces place demands on us, or we turn decisions over to others and then follow their lead.</li> <li>In the case of external forces, it's not our job to decide what to do; it's our job to decide how to make it work.</li> <li>Consulting is where decision makers invite others to influence before they make a choice. It permits gaining ideas and support without bogging down the decision-making process.</li> <li>Voting is best suited to situations where efficiency is the highest value, and you're selecting from a number of good options.</li> <li>Consensus can either produce tremendous unity and high-quality decisions, or can be a horrible waste of time.</li> <li>Reserve consensus for high-stakes and complex issues, or issues where everyone must absolutely support the final choice.</li> <li>Don't involve people who don't care, omit people who contribute no new information, but involve those whose authority or influence you need.</li> <li>Create assignments by specifying who, does what, by when, and how will you follow up?</li> <li>When passing out assignments, "we" can lead someone to believe that others are taking on the responsibility. Assign a name to every responsibility.</li> <li>To help clarify deliverables, use Contrasting to explain what you don't want. Or make things concrete by pointing to a prototype or sample.</li> <li>Assignments without deadlines are far better at producing guilt than stimulating action.</li> <li>If you want people to feel accountable, then you must give them opportunity to account. Build expectation for follow-up into every assignment.</li> </ul> <h3>Chapter 10: Yeah, But</h3> <ul> <li>With sexual or other harassment, own up to tolerating the behavior for too long, and then establish Mutual Purpose and then STATE your path.</li> <li>With a sensitive spouse, use contrasting, tentatively explain consequences, and use testing. Look for signs that safety is at risk, then make it safe.</li> <li>When a teammate fails to live up to agreements, speak up and hold each other accountable.</li> <li>When experiencing deference to authority, either you are creating fear or you are living with the ghosts of past bosses, or both.</li> <li>If you are the problem, solicit peer feedback. If it stems from ghosts, go public and reward risk-takers, encourage testing, and thank people for contrary opinions.</li> <li>With lack of trust, bring up your concerns, and STATE what you see happening. Do not tell a Villain Story that exaggerates others' untrustworthiness.</li> <li>With someone who won't talk about anything serious, Make It Safe, use tentative language, Explore Others' Paths, and be patient.</li> <li>When you see signs of improvement, invite talking about how you talk or don't talk about important issues, building safety by establishing Mutual Purpose.</li> <li>When faced with subtle and unacceptable actions, retrace your Path to Action to its source, and if the behaviors are worthy of dialogue STATE Your Path.</li> <li>With someone shows no initiative, establish new and higher expectations. Deal with the overall pattern, not a specific instance.</li> <li>With a problem shows a pattern, STATE Your Path. Talk about the pattern; talking about any one instance will make your concern seem trivial.</li> <li>When you need to calm down, suggest that you need some time alone and that you'd like to pick up the conversation later.</li> <li>Suggesting someone else needs to calm down is patronizing. To get back to the source of their anger, retrace their Path to Action.</li> <li>When faced with endless excuses, gain commitment to solving the overall problem, not simply the stated cause.</li> <li>Insubordination catches most leaders by surprise. They buy time to figure out what to do, which lets the person get away with it.</li> <li>When you catch insubordination, speak up immediately but respectfully. Change topics from the issue at hand to how the person is currently acting.</li> <li>Stories left unattended don't get better with time, they ferment. Then when we can't take it anymore, we say something that we regret.</li> <li>Use your STATE skills before a story turns too ugly, or if you've let the problem build, use your STATE skills and tell the most simple and least offensive story.</li> <li>With touchy and personal issues, use contrasting, establish Mutual Purpose to show honorable intentions, and tentatively describe the problem.</li> <li>When faced with word games, STATE the pattern of playing games, discuss both behaviors and outcomes, and then hold them accountable to results.</li> <li>With someone who delivers surprises, clarify that one needs to complete the assignment as planned or immediately inform you if they run into a problem.</li> <li>With someone who violates all the dialogue principles, choose your targets carefully: Focus on what bothers you the most, and what is easiest to work on.</li> </ul> <h3>Chapter 11: Putting It All Together</h3> <ul> <li>The first lever for getting to dialogue is asking whether you're playing games or in dialogue. If you've moved away, don't be afraid to express that sentiment.</li> <li>The second lever is making it safe. Asking a question, showing interest, an appropriate touch, an apology, a smile, or a request for a "time out" all build safety.</li> <li>For "Start with Heart", focus on what you really want, and refuse the Sucker's Choice.</li> <li>For "Learn to Look," look for when the conversation becomes crucial, and look for safety problems.</li> <li>For "Make It Safe," apologize when appropriate, contrast to fix misunderstanding, and CRIB to get Mutual Purpose.</li> <li>For "Master My Stories," retrace your path to action, separate fact from story, and watch for the Three Clever Stories.</li> <li>For "STATE My Path," Share your facts, Tell your story, Ask for others' paths, Talk tentatively, and Encourage testing.</li> <li>For "Explore Others' Paths," use AMPP, or Ask, Mirror, Paraphrase, and Prime, and then Agree, Build, and Compare.</li> <li>For "Move to Action," decide how you will decide, and then document decisions and follow up.</li> <li>The current quality of your leadership and your life is fundamentally a function of how you are presently handling crucial conversations.</li> </ul> https://tedneward.github.io/Research/thinking/50-philosophy-ideas/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/50-philosophy-ideas/index.html <h3>The Brain in a vat</h3> <p><em>(are you an envatted brain?)</em><br> scepticism: how do we know what we know?</p> <p>the philosophical skeptic doesn't claim that we know nothing--not least because to do so would be obviously self-defeating (one thing we could not know is that we know nothing). Rather the skeptic's position is to challenge our right to make claims to knowledge. We think we know lots of things, but how can we defend those claims? What grounds can we produce to justify any particular claim of knowledge? Our supposed knowledge of the world is based on perceptions gained via our senses, usually mediated by our use of reason. But are not such perceptions always open to error? Can we ever be sure we're not hallucinating or dreaing, or that our memory isn't playing tricks? [Epistemology]</p> <h3>Plato's cave</h3> <p><em>(earthly knowledge's but a shadow)</em></p> <p>the cave represents 'the realm of becoming'--the visible world of our everyday experience, where everything is imperfect and constantly changing. The chained captives (us) live in a world of conjecture and illusion, while the former prisoner, free to roam within the cave, attains the most accurate view of reality possible, within the ever-changing world of perception and experience. By contrast, the world outside the cave represents the 'realm of being'--the intelligible world of truth populated by the objects of knowledge, which are perfect, eternal and unchanging.</p> <p>what is known must not only be true but also perfect and unchanging. Nothing in the empirical world (represented by life within the cave) fits this description: a tall person is short next to a tree; an apple that appears red at noon looks black at dusk; and so on. As nothing in the empirical world is an object of knowledge, Plato proposed that there must be another ream (the world outside the cave) of perfect and unchanging ideas which he called 'Forms' or Ideas.</p> <p>the problem of universals: realists/Platonists believe that universals such as redness and tallness exist independently of particular red and tall things; and the anti-realists/nominalists hold that they are mere names or labels that are attached to objects to highlight particular similarities between them. Modern debate continues: realists hold that there are entities 'out there' in the world--physical things or ethical facts or mathematical properties--that exist independently of our knowing/experiencing them. Anti-realists there must be a necessary and internal link or relation between what is known and our knowledge of it.</p> <h3>The veil of perception</h3> <p><em>(what lies beyond the veil?)</em></p> <p>most of us uncritically suppose that physical objects around us are more or less what we perceive them to be, but some question that--in their view we have direct access to inner 'ideas', 'impressions' or 'sense data'; human understanding is like a closet wholly shut from light, with only some little openings left, to let in external visible resemblances, or ideas of things without. Our ideas, which are all that we have direct access to, form an impenetrable 'veil of perception' between us and the outside world. 'Representational' models of perception: any such model that involves intermediate ideas or sense data drives a wedge between us and the external world, and it is in the fissure so formed that skepticism about our claims to knowledge really exist. It is only by re-establishing a direct link between observer and external object that the veil can be torn and the skeptic vanquished. So, given that the model causes such problems, why adopt it in the first place? Locke hoped that by distinguishing between 'primary' and 'secondary' characteristics (one being intrinsic, the other perceptionally-influenced) he could disarm the skeptic.</p> <p>Berkeley refuted it by saying that Locke could never actually check whether his supposed resemblances actually resembled the external things themselves, then took it to its conclusion and said that reality consists in the 'ideas' or sensations themselves; with these, we are already fully and properly connected, so the dangers of skepticism are evaded, but only at the price of the denial of an external, physical world. Berkeley's idealist (or immaterialist) theory states "to exist is to be perceived", and since God perceives all things, it remains in existence even when passing out of human perception.</p> <h3><em>cogito ergo sum</em></h3> <p><em>(I am thinking therefore I exist)</em></p> <h3>Reason &amp; experience</h3> <p><em>(How do we know?)</em></p> <h3>The tripartite theory of knowledge</h3> <p><em>(When do we really know?)</em></p> <h3>The mind-body problem</h3> <p><em>(Mind boggles)</em></p> <h3>What is it like to be a bat?</h3> <p><em>(Inside a bat's mind?)</em></p> <h3>The Turing test</h3> <p><em>("Did you ever take that test yourself?")</em></p> <h3>The ship of Theseus</h3> <p><em>(What makes you, you?)</em></p> <h3>Other minds</h3> <p><em>(Is there anybody there?)</em></p> <h3>Hume's guillotine</h3> <p><em>(The is-ought gap)</em></p> <h3>One man's meat</h3> <p><em>(Is it all relative?)</em></p> <h3>The divine command theory</h3> <p><em>(Because God says so)</em></p> <h3>The boo/hooray theory</h3> <p><em>(Expressing moral judgements)</em></p> <h3>Ends &amp; means</h3> <p><em>(The least bad option)</em></p> <h3>The experience machine</h3> <p><em>(Is happiness enough?)</em></p> <h3>The categorical imperative</h3> <p><em>(Duty at any cost)</em></p> <h3>The Golden Rule</h3> <p><em>(Do as you would be done by)</em></p> <h3>Acts &amp; omissions</h3> <p><em>(To do or not to do?)</em></p> <h3>Slippery slopes</h3> <p><em>(If you give an inch...)</em></p> <h3>Beyond the call of duty</h3> <p><em>(Should we all be heroes?)</em></p> <h3>Is it bad to be unlucky?</h3> <p><em>(Does Fortune favor the good?)</em></p> <h3>Virtue ethics</h3> <p><em>(Who you are, not what you do)</em></p> <h3>Do animals feel pain?</h3> <p><em>(Animal cruelty)</em></p> <h3>Do animals have rights?</h3> <p><em>(Human wrongs?)</em></p> <h3>Forms of argument</h3> <p><em>(Infallible reasoning?)</em></p> <h3>The barber paradox</h3> <p><em>(If it is, it isn't)</em></p> <h3>The gambler's fallacy</h3> <p><em>(Against the odds)</em></p> <h3>The Sorites Paradox</h3> <p><em>(How many grains make a heap?)</em></p> <h3>The King of France is bald</h3> <p><em>(Language and logic)</em></p> <h3>The beetle in the box</h3> <p><em>(Language games)</em></p> <h3>Science &amp; pseudoscience</h3> <p><em>(Evidence falsifying hypothesis)</em></p> <h3>Paradigm shifts</h3> <p><em>(science--evolution and revolution)</em></p> <h3>Occam's Razor</h3> <p><em>(Keep it simple)</em></p> <h3>What is art?</h3> <p><em>(Aesthetic values)</em></p> <h3>The intentional fallacy</h3> <p><em>(Meanings in art)</em></p> <h3>The argument from design</h3> <p><em>(The divine watchmaker)</em></p> <h3>The cosmological argument</h3> <p><em>(The first and uncaused cause)</em></p> <h3>The ontological argument</h3> <p><em>(The greatest imaginable being)</em></p> <h3>The problem of evil</h3> <p><em>(Why does God let bad things happen?)</em></p> <h3>The freewill defense</h3> <p><em>(Freedom to do wrong)</em></p> <h3>Faith &amp; reason</h3> <p><em>(The leap of faith)</em></p> <h3>Positive &amp; negative freedom</h3> <p><em>(Divided loyalties)</em></p> <h3>The difference principle</h3> <p><em>(Justice as fairness)</em></p> <h3>Leviathan</h3> <p><em>(The social contract)</em></p> <h3>The prisoner's dilemma</h3> <p><em>(Playing the game)</em></p> <h3>Theories of punishment</h3> <p><em>(Does the punishment fit the crime?)</em></p> <h3>Lifeboat Earth</h3> <p><em>(Is there more room in the boat?)</em></p> <h3>Just war</h3> <p><em>(Fight the good fight)</em></p> https://tedneward.github.io/Research/thinking/50-psychology-ideas/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/50-psychology-ideas/index.html <p><em>(Furnham, Adrian (2009-03-02). 50 Psychology Ideas You Really Need to Know (50 Ideas You Really Need to Know Series) )</em></p> <h3>Abnormal behavior</h3> <p>"For clinical psychology, the issue is not so much whether the behaviour is abnormal, as whether it is maladaptive, causing a person distress and social impairment. If a person’s behaviour seems irrational or potentially harmful to themselves and others, we tend to think of that as abnormal. For the psychologist it is called psychopathology."<br> "Today, psychological definitions of abnormality revolve around a handful of generally agreed-upon criteria. These have been classified as the 4Ds: distress, deviance, dysfunction, danger. … A very common criterion is irrationality -- bizarre, illogical beliefs about the physical or social world as well as, very often, the spiritual world."<br> * Subjective: Uses ourselves, our behavior, our values as the criteria of normality<br> * Normative: There is an ideal, desirable state of how one should think and behave<br> * Clinical: "Social scientists and medical clinicians attempt to assess the effectiveness, organization and adaptiveness of a person’s functioning. Much depends on which dimension is being assessed. Clinicians also accept that the normal–abnormal distinctions are grey and somewhat subjective, though they strive for reliable diagnosis. Abnormality is usually associated with poor adaptations, pain or bizarre behaviours."<br> * Cultural: "Culture dictates trends in everything from dress to demeanour, language to love. Culture prescribes and proscribes behaviours. Certain things are taboo, others are illegal."<br> * Statistical: "All statisticians know the concept of the bell curve or the normal distribution. It has particular properties and is best known in the world of intelligence. Thus a score of 100 is average and 66 per cent of the population score between 85 and 115, and around 97 per cent between 70 and 130. Thus if you score below 70 and over 130 you are unusual, though the word ‘abnormal’ would not be applied. This model has drawbacks in the fact that behaviour that occurs frequently does not necessarily make it healthy or desirable."</p> <h3>Placebo effect</h3> <p>"A placebo is simply defined as a preparation with no medicinal value and no pharmacological effects. An active placebo is one that mimics the side-effects of the drug under investigation but lacks its specific, assumed therapeutic affect. … Modern research in the area is usually attributed to a paper written in the American Dental Association Journal over 50 years ago. Henry Beecher shocked the medical world by claiming that just placebo procedures like giving sugar pills or even sympathetically physically examining the patient would lead to an improvement in 30 per cent of patients. Today that estimate has increased to between a half to three-quarters of patients, with all sorts of problems from asthma to Parkinson’s showing real lasting improvements from a range of treatments."</p> <p>"More serious, ‘major’ or invasive procedures do appear to have stronger placebo effects. Injections per se appear to have a greater impact than pills, and even placebo surgery (where people are cut open and sewn up with little or nothing done) has yielded high positive response rates. The style of treatment administration and other qualities of the therapist appear to contribute substantially to the impact of the treatment itself. Those therapists who also exhibit greater interest in their patients, greater confidence in their treatments, and higher professional status, all appear to promote stronger placebo effects in their patients."</p> <h3>Kicking the habit</h3> <p><em>(addiction)</em><br> "Addiction involves the exposure to something and then the behaviour seeking to repeat the experience very often. Over time the addiction becomes established. There is regular and increasing consumption, with the takers knowing their habit is expensive, unhealthy and possibly illegal but seemingly being unable to give it up."</p> <p>"With regard to drugs, the psychiatric literature distinguishes between substance dependence and abuse. Both have technical meaning. Dependence has very specific characteristics like tolerance (people take more and more for limited effect); withdrawal symptoms (on not taking the drug); obsessions with trying to get hold of the drug; a deterioration in all social, occupational and recreational activities; and continued use with full knowledge of all the damage that is being done. … Abuse means using the drug despite the need to fulfil various school, home and work obligations; use in dangerous situations (driving, at work); use despite illegal behaviour; use despite persistent negative side-effects."</p> <h3>Lost touch</h3> <p><em>(schizophrenia)</em><br> "Schizophrenia is a psychotic illness characterized by a disorder of thoughts and perceptions, behaviours and moods. … They tend to have various manifestations of thought disorders (disorganized, irrational thinking), delusions and hallucinations. They tend to lack energy, initiative and social contacts. They are emotionally very flat, have few pleasures and are withdrawn."</p> <p>"The classification of schizophrenia remains complex because of the diversity of symptoms. These include delusions; hallucinations; disorganized speech (incoherence, loose association, use of nonsense words); disorganized behaviour (dress, body posture, personal hygiene); negative, flat emotions; poor insight into their problems; and depression. Because of complications with the diagnosis, various subtypes have been named. Thus there is paranoid and catatonic schizophrenia. Catatonics (from the Greek ‘to stretch or draw tight’) often adopt odd, stationary poses for long periods of time. Paranoid schizophrenics have delusions of control, grandeur and persecution and are consistently suspicious of all around them. Disorganized schizophrenics manifest bizarre thoughts and language, with sudden inappropriate emotional outbursts. Some psychiatrists mention simple or undifferentiated schizophrenia. Others have distinguished between acute (sudden, severe onset) and chronic (prolonged, gradual onset). Another distinction is between Type I (mostly positive symptoms) and Type II (mostly negative symptoms)."</p> <p>"There is still no complete agreement about the subtypes or the precise ‘deficits’ in functioning, though these usually come under four headings: cognitive or thinking; perceptual or seeing; motor or moving; emotional or feeling."</p> <h3>Not neurotic, just different</h3> <p><em>(anti-psychiatry)</em><br> "Anti-psychiatry critics questioned three things: the medicalization of madness; the existence of mental illness; and the power of psychiatrists to diagnose and treat certain individuals with compulsion. Anti-psychiatry was more than anti-custodial: it was often anti-state, almost anarchic. It saw many state institutions, particularly mental hospitals, as distorting and repressing the human spirit and potential in various groups."</p> <p>"One of the most famous anti-psychiatry studies was done in the early 1970s. Eight ‘normal’, mentally healthy researchers tried to gain admission, through diagnosis, to a number of American mental hospitals. The only symptom they reported was hearing voices. Seven were diagnosed as schizophrenic and admitted. Once in the hospital they behaved normally and were ignored when they politely asked for information. They later reported that their diagnostic label of schizophrenia meant they had low status and power in the hospital. Then they ‘came clean’ and admitted they had no symptoms and felt fine. But it took nearly three weeks before they were discharged, often with the diagnosis ‘schizophrenia in remission’. So normal, healthy people could easily be diagnosable as ‘abnormal’. But could the reverse happen? The same researchers told psychiatric hospital staff that fake or pseudopatients pretending to be schizophrenics may try to gain access to their hospital. They then found that 19 genuine patients were suspected as frauds by two or more members of staff, including a psychiatrist."</p> <h3>Seem sane</h3> <p><em>(psychopathy/sociopathy)</em><br> "Psychopathy is a personality disorder characterized by people who have no conscience and are incapable of empathy, guilt or loyalty to anyone but themselves. Sociopathy is a non-psychiatric condition and refers to those who are anti-social and criminal and follow the norms of a particular subculture."</p> <p>"Overall psychopaths tend to be impulsive and irresponsible, with few clear life goals. They have a history of problems with authority and poor behavioural controls. They lack empathy and remorse and never accept responsibility for their actions. … The first response to being found out is to escape, leaving colleagues, family or debtors to pick up the pieces. They do so without a qualm. The next response is to lie with apparent candour and sincerity even under oath and even to parents and loved ones. They behave as if social rules and regulations do not really apply to them. They have no respect for authorities and institutions, families and traditions."</p> <ol> <li>Psychopaths show a disregard for, and violation of, the rights of others. They often have a history of being difficult, delinquent or dangerous.</li> <li>They fail to conform to social norms with respect to lawful behaviours (repeatedly performing acts that are grounds for arrest, imprisonment and serious detention). This includes, lying, stealing and cheating.</li> <li>They are always deceitful, as indicated by repeated lying, use of aliases, or conning others for personal profit or pleasure. They are nasty, aggressive, con artists – the sort who often get profiled on business crime programmes.</li> <li>They are massively impulsive and fail to plan ahead. They live only in, and for, the present.</li> <li>They show irritability and aggressiveness, as indicated by repeated physical fights or assaults. They can’t seem to keep still – ever.</li> <li>They manifest a reckless disregard for the physical and psychological safety of others.</li> <li>They are consistently irresponsible. Repeated failure to sustain consistent work behaviour or to honour financial obligations are their hallmark.</li> <li>They show a lack of remorse. They are indifferent to, or rationalize, having hurt, mistreated or stolen from another. They never learn from their mistakes. It can seem that labelling them as anti-social is a serious understatement."</li> </ol> <p>"They avoid group/committee meetings because they say very different things to different people and can’t present a single façade or voice that is coherent. Co-workers, colleagues and reports are frequently abandoned when their usefulness is at an end. They deliberately create conflict between individuals to try to prevent them sharing information about them. All detractors are ‘neutralized’ not so much by violence or threats but by raising doubts about their integrity and loyalty as well as their competence. Psychopaths seek out organizations in flux or change as well as those with poor monitoring systems so that they are rarely threatened or challenged."</p> <h3>Stress</h3> <p>Challenge and support at work:</p> <ul> <li>Much support, little challenge People in this situation are in the fortunate position of good technical and social support, but the fact they are under-challenged probably means that they underperform. They may actually be stressed by boredom and monotony.</li> <li>Much support, much challenge This combination tends to get the most out of people as they are challenged by superiors, subordinates, shareholders and customers to ‘work smarter’ but are given the appropriate support to succeed.</li> <li>Little support, much challenge This unfortunate, but very common, situation is a major cause of stress for any manager because he or she is challenged to work consistently hard but only offered minimal emotional, informational (feedback) and physical (equipment) support.</li> <li>Little support, little challenge People in some bureaucracies lead a quiet and unstressed life because they are neither challenged nor supported, which usually means neither they nor their organization benefit.</li> </ul> <p>"Three components: Firstly, stress can be a function of the make-up of the individual, particularly their personality, ability and biography. Secondly, there are features about the environment (job, family, organization), usually but not exclusively considered in terms of the work environment. Thirdly, there is how the individual and the environment perceive, define but more importantly try to cope with stress, strains and pressures."</p> <p>"Coping One distinction that has been made is between problem-focused coping (aimed at problem-solving or doing something to alter the source of stress) and emotion-focused coping (aimed at reducing or managing the emotional distress that is associated with, or cued by, a particular set of circumstances). Emotion-focused responses can involve denial; others involve positive reinterpretation of events; and still others involve the seeking out of social support. Similarly, problem-focused coping can potentially involve several distinct activities, such as planning, taking direct action, seeking assistance, screening out particular activities, and sometimes stopping acting for an extended period.<br> * Optimism: a buffer against stress Optimists are hopeful in their outlook on life, interpret a wide range of situations in a positive light, and tend to expect favourable outcomes and results. Pessimists, by contrast, interpret many situations negatively, and expect unfavourable outcomes and results. Optimists concentrate on problem-focused coping – making and enacting specific plans for dealing with sources of stress. In addition, they seek social support – the advice and help of friends and others – and refrain from engaging in other activities until current problems are solved and stress is reduced.<br> * Hardiness: viewing stress as a challenge Hardy people appear to differ from others in three respects. They show a higher level of commitment – deeper involvement in their jobs and other life activities; control – the belief that they can, in fact, influence important events in their lives and the outcomes they experience; and challenge – they perceive change as a challenge and an opportunity to grow rather than as a threat to their security."</p> <h3>Visual illusions</h3> <p>"It has been suggested that all illusions fall into one of four groups: ambiguities, distortions, paradoxes and factions. Of course illusions are of particular interest to visual scientists and cognitive psychologists because they give an important insight into the process of perception.</p> <h3>Psychophysics (physics of the mind)</h3> <p>"A simple psychophysics question becomes ‘What is the chain of events that begins with a stimulus and leads up to reports such as “a bright red”, or “a loud noise”?’ The details of this sequence obviously differ for each sense but there are always three basic steps: a stimulus to a sense receptor; a neural chain of events caused by this stimulus – it is changed into an electric signal and then into a nerve impulse; a psychological response to the message (sensation)."</p> <h3>Hallucinations</h3> <h3>Delusions</h3> <h3>Are you conscious?</h3> <h3>Positive psychology</h3> <h3>Emotional intelligence</h3> <h3>What are emotions for?</h3> <h3>Cognitive therapy</h3> <h3>IQ and you</h3> <h3>Flynn effect</h3> <h3>Multiple intelligences</h3> <h3>Cognitive differences</h3> <h3>Rorschach inkblot test</h3> <h3>Detecting lies</h3> <h3>Authoritarian personality</h3> <h3>Obedience to authority</h3> <h3>Fitting in</h3> <h3>Self-sacrifice or selfishness</h3> <h3>Cognitive dissonance</h3> <h3>Gambler's fallacy</h3> <h3>Judgement and problem-solving</h3> <h3>Too much invested to quit</h3> <h3>Rational decision-making</h3> <h3>Remembrance of things past</h3> <h3>What the witness saw</h3> <h3>Artificial intelligence</h3> <h3>Perchance to dream</h3> <h3>Try to forget</h3> <h3>Tip-of-the-tongue phenomenon</h3> <h3>Psychosexual stages</h3> <h3>Cognitive stages</h3> <h3>Ducks in a row</h3> <h3>Tabula rasa</h3> <h3>Stay hungry</h3> <h3>Behaviorism</h3> <h3>Reinforcement schedules</h3> <h3>Mastering complexity</h3> <h3>Phrenology</h3> <h3>Breaking up is hard to do</h3> <h3>Aphasia</h3> <h3>Dyslexia</h3> <h3>Who's that?</h3> https://tedneward.github.io/Research/thinking/8-pillars-of-greek-philosophy/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/8-pillars-of-greek-philosophy/index.html <ol> <li>Humanism <ol> <li>The Challenges of Humanness <ol> <li>Proper posture for humanity is upright and tall</li> <li>Man as potentially superior to the gods; it is man who must die, it’s only man who can most truly live. Gods are static and unchanging, but humanity possesses the capacity for growth.</li> <li>Achilles chose the short life of unending fame. Only in that way could he achieve the immortality he had been denied by birth. We all have a weakness we cannot see, but without which we would cease to be human.</li> <li>Our lives exist in order that we may compete an unfinished task, and it is by that task, that our lives are given meaning whatever the peri. Ulysses could not stay with Calypso because to do would have been to hide from himself, from the self he could be</li> </ol> </li> <li>In Praise of Life: Humanism is the proud affirmation of both our promise and our duty. Greeks celebrated the human enterprise with all of their talent and might in pictures and stories. They expended great time and energy in endowing their pottery and coins with symmetrical form and harmonious decoration.</li> <li>The Test of Humanism: The modern world provides its own share of challenges; we must decide how we respond to these challenges. Anything humanly conceived, the Greeks are quick to remind us, can be humanly changed.</li> <li>Practicing Humanism: <ol> <li>Putting humanism into practice requires we define the term accurately. Implicit in "humane" is the notion that there are latent sensibilities in our character that must be cultivated if we are to attain our fullest stature. That task is the moral imperative of our existence.</li> <li>Humanism thus implies a commitment to the special abilities and talents we possess as human beings. To practical humanism in a personal way means first to look into ourselves to discover the particular abilities and talents that are ours. The second step is to actively apply them to our everyday lives.</li> <li>In taking the self-assessment, we come to see that our fulfillment as individuals depends, at least in part, on our engagement with the outside world. Becoming fully humane thus means opening ourselves to the needs of other human beings.</li> <li>"Freedom and life are only earned by those who conquer them each day anew." (Geothe, end of Faust). Until we recognize our own intrinsic worth as human beings we will not make such an effort.</li> </ol> </li> <li>Humanism is a philosophy of limitless possibilities tempered by the fact that we must someday die. It is marked as the tension between these two forces, passionate delight in life, and clear apprehension of its unalterable framework.</li> </ol> </li> <li>Pursuit of Excellence <ol> <li>In the grammar of the Greek universe, humanity was not a noun, but a verb: moving, changing, evolving toward a fuller realization of its inner potential. Humanism implied a dynamic and progressive process of spiritual growth: the pursuit of excellence.</li> <li>Heroic Code: the compulsion to be the best</li> <li>From Olympus to Olympia: the Olympic games were really all about showing the gods what human beings were capable of doing if they bent mind and muscle to the task. The Games were meant to honor Zeus by human achievement, energized by the principle that man could best serve the gods by developing his own innate talents, in the process expanding the definition of what it means to be human. This is the inner justification for the Greek pursuit of excellence: the belief that within the human soul is a divine spark, and it is the duty of each of us to fan that spark into a flame.</li> <li>Wine and Inspiration: the Greeks also revered intellectual competition. Like the drinking of too much wine, the pursuit of excellence does not guarantee a happy outcome. Competitive striving also implied the risk of loss, but the ancient Greeks were congenial risk-takers. Heroism often courts tragedy. This is because the hero's life is a life lived at the outer edge of experience. At such an extremity, success becomes an intoxicant that can cause the individual to lose perspective on his own human limitations and in the end pay a higher price than he would ever have imagined. But to dare less would be to know less and to be less, an alternative the Greek culture rejected.</li> <li>The Meaning of Excellence: We cannot all be winners, but neither were the Greeks. Winning is not what life asks of us; it asks us to discover those things we are capable of doing well, and then to do them with all our heart and soul. The ancient Greeks never mistook winning for heroism. To be a hero, to pursue excellence, is to be a loving mother, to be a compassionate husband, to do our job well, with honor, integrity, and passion. To do so is to fulfill our finest nature; to do less is to lose our chance to experience the full meaning of being alive.</li> <li>The Price of Excellence: To pursue excellence, however, inevitably means to encounter frustration (the cruelest punishment of all, to the Greeks). Greek Hell was about being frustrated, not physical torture, because they were a race of achievers. Just being in the land of the dead constituted punishment because nothing could be accomplished there. In its dark and dank environment, the dead dwelt as disembodied ghosts, devoid of substance and purpose. If the price of excellence was frustration in the here and now, it was a price worth paying--without the savor of success the rest would be empty indeed. It is the choice of Achilles, but it is our choice, too, individually and collectively, to wallow in the warm mud of mediocrity, or climb to the mountain's cold and rocky heights.</li> <li>The Cost of Success: Achievement installs pride, but pride goeth before a fall. The myths of Niobe, Arachne; the arrogance of Agamemnon.</li> </ol> </li> <li> <p>Practice of Moderation</p> </li> <li> <p>Self-Knowledge</p> </li> <li> <p>Rationalism</p> </li> <li> <p>Restless Curiosity</p> </li> <li> <p>Love of Freedom</p> </li> <li> <p>Individualism</p> </li> </ol> https://tedneward.github.io/Research/thinking/50-big-ideas/index.html Mon, 13 Apr 2026 14:04:56 +0000 thinking/50-big-ideas/index.html <h3>Platonism</h3> <p><em>(towards transcendent reality)</em></p> <h3>Aristotelianism</h3> <p><em>("The Master of Those Who Know")</em></p> <h3>The Golden Rule</h3> <p><em>(Do as you would be done by)</em></p> <h3>Altruism</h3> <p><em>(Selfless or selfish?)</em></p> <h3>Liberty</h3> <p><em>(something worth dying for?)</em></p> <h3>Tolerance</h3> <p><em>(a troubled and paradoxical virtue)</em></p> <h3>Scepticism</h3> <p><em>(the scourge of drama)</em></p> <h3>Reason</h3> <p><em>(Master or slave of the passions?)</em></p> <h3>Punishment</h3> <p><em>(a necessary evil?)</em></p> <h3>Materialism</h3> <p><em>(Matter over mind)</em></p> <h3>Relativism</h3> <p><em>(anything goes?)</em></p> <h3>Utilitarianism</h3> <p><em>(The greatest happiness principle)</em></p> <h3>Existentialism</h3> <p><em>(Condemned to be free)</em></p> <h3>Evil</h3> <p><em>(Is evil good for us?)</em></p> <h3>Fate</h3> <p><em>(The iron hand of destiny)</em></p> <h3>Soul</h3> <p><em>(A mystery no mind can hope to unravel)</em></p> <h3>Faith</h3> <p><em>(belief that is blind to reason)</em></p> <h3>Fundamentalism</h3> <p><em>(when faith becomes fanatic)</em></p> <h3>Atheism</h3> <p><em>(beyond belief)</em></p> <h3>Secularism</h3> <p><em>(not doing God)</em></p> <h3>Creationism</h3> <p><em>(when world-views collide)</em></p> <h3>War</h3> <p><em>(politics by other means)</em></p> <h3>Duty</h3> <p><em>(thou shalt not... come what may?)</em></p> <h3>Utopia</h3> <p><em>(Heaven or hell on earth)</em></p> <h3>Liberalism</h3> <p><em>(Social progress and individual freedom)</em></p> <h3>Democracy</h3> <p><em>(Bludgeoning of the people by the people?)</em></p> <h3>Conservatism</h3> <p><em>(adherence to the old and tried)</em></p> <h3>Imperialism</h3> <p><em>(to enslave or to set free?)</em></p> <h3>Nationalism</h3> <p><em>(the measles of the human race)</em></p> <h3>Multiculturalism</h3> <p><em>(melting pot or salad bowl?)</em></p> <h3>The social contract</h3> <p><em>(society by consent)</em></p> <h3>Republicanism</h3> <p><em>(Government of laws, not of men)</em></p> <h3>Communism</h3> <p>*("Workers of the world, unite!")<br> *</p> <h3>Fascism</h3> <p><em>(a toxic synthesis of left &amp; right)</em></p> <h3>Racism</h3> <p><em>(the starless midnight of racism)</em></p> <h3>Feminism</h3> <p><em>(equal, not the same)</em></p> <h3>Islamism</h3> <p><em>(a clash of civilizations)</em></p> <h3>Capitalism</h3> <p><em>(the unequal sharing of blessings)</em></p> <h3>Globalization</h3> <p><em>(life in the global village)</em></p> <h3>Classicism</h3> <p><em>(Rome's ancient genius)</em></p> <h3>Romanticism</h3> <p><em>(The addition of strangeness to beauty)</em></p> <h3>Modernism</h3> <p><em>(the shock of the new)</em></p> <h3>Surrealism</h3> <p><em>(the omnipotence of dreams)</em></p> <h3>Censorship</h3> <p><em>(protecting the servants)</em></p> <h3>Evolution</h3> <p><em>(survival of the fittest)</em></p> <h3>Gaia</h3> <p><em>(the quest for mother earth)</em></p> <h3>Chaos</h3> <p><em>(the butterfly effect)</em></p> <h3>Relativity</h3> <p><em>(the meaning of space &amp; time)</em></p> <h3>Quantum Mechanics</h3> <p><em>(the strange poetry of atoms)</em></p> <h3>The Big Bang</h3> <p><em>(the beginning of space &amp; time)</em></p> https://tedneward.github.io/Research/teaching/datasets/index.html Mon, 13 Apr 2026 14:04:56 +0000 teaching/datasets/index.html <p>From <a href="https://www.linkedin.com/posts/ravena-o-04b39b68_10-free-datasets-to-start-building-your-activity-7072183693087322112-Azwy/">Ravenna O</a>:</p> <ol> <li> <p>Supermarket Sales: Historical record of sales data in 3 different supermarkets <a href="https://lnkd.in/e86UpCMv">https://lnkd.in/e86UpCMv</a></p> </li> <li> <p>Credit Card Fraud Detection: Anonymized credit card transactions labeled as fraudulent or genuine. <a href="https://lnkd.in/eFTsZDCW">https://lnkd.in/eFTsZDCW</a></p> </li> <li> <p>FIFA 22 complete player dataset: The datasets provided include the players data for the Career Mode from FIFA 15 to FIFA 22. The data allows multiple comparisons for the same players across the last 8 version of the videogame. <a href="https://lnkd.in/eDScdUUM">https://lnkd.in/eDScdUUM</a></p> </li> <li> <p>Walmart Store Sales Forecasting: Use historical markdown data to predict store sales. <a href="https://lnkd.in/eVT6h-CT">https://lnkd.in/eVT6h-CT</a></p> </li> <li> <p>Netflix Movies and TV Shows: Listings of movies and tv shows on Netflix - Regularly Updated <a href="https://lnkd.in/eZ3cduwK">https://lnkd.in/eZ3cduwK</a></p> </li> <li> <p>LinkedIn Data Analyst jobs listings: More than 8400 rows of data analyst jobs from USA, Canada and Africa. <a href="https://lnkd.in/ezqxcmrE">https://lnkd.in/ezqxcmrE</a></p> </li> <li> <p>Top 50 Fast-Food Chains in USA: The key features of this Dataset are - Fast-Food Chains, U.S. Systemwide Sales (Millions - U.S Dollars), Average Sales per Unit (Thousands - U.S Dollars), Franchised Stores, Company Stores, 2021 Total Units, Total Change in Units from 2020. <a href="https://lnkd.in/esBjf5u4">https://lnkd.in/esBjf5u4</a></p> </li> <li> <p>Amazon and Best Buy Electronics: This is a list of over 7,000 online reviews for 50 electronic products from websites like Amazon and Best Buy provided by Datafiniti's Product Database. The dataset includes the review date, source, rating, title, reviewer metadata, and more. <a href="https://lnkd.in/e4fBZvJ3">https://lnkd.in/e4fBZvJ3</a></p> </li> <li> <p>Forecasting Book Sales: Forecast sales of 8 book titles in 2418 locations <a href="https://lnkd.in/eXHN2XsQ">https://lnkd.in/eXHN2XsQ</a></p> </li> <li> <p>Real / Fake Job Posting Prediction: This dataset contains 18K job descriptions out of which about 800 are fake. The data consists of both textual information and meta-information about the jobs. The dataset can be used to create classification models which can learn the job descriptions which are fraudulent. <a href="https://lnkd.in/e5SDDW9G">https://lnkd.in/e5SDDW9G</a></p> </li> </ol> https://tedneward.github.io/Research/storage/scrollsets/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/scrollsets/index.html <p><a href="https://breckyunits.com/scrollsets.html">Website</a></p> https://tedneward.github.io/Research/storage/spicedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/spicedb/index.html <p><a href="https://authzed.com/docs">Website</a> | <a href="https://github.com/authzed/spicedb">Source</a></p> https://tedneward.github.io/Research/storage/storj/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/storj/index.html <p><a href="https://www.storj.io/">Website</a> | <a href="https://github.com/storj/storj">Source</a></p> https://tedneward.github.io/Research/storage/tdengine/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/tdengine/index.html <p><a href="https://tdengine.com/?en">Website</a> | <a href="https://github.com/taosdata/TDengine">Source</a></p> https://tedneward.github.io/Research/storage/tingodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/tingodb/index.html <p><a href="http://www.tingodb.com/">Website</a> | <a href="https://github.com/sergeyksv/tingodb">Source</a></p> https://tedneward.github.io/Research/storage/typedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/typedb/index.html <p><a href="https://vaticle.com/">Website</a> | <a href="https://github.com/vaticle/typedb">Source</a></p> <p><a href="https://github.com/vaticle/typeql">TypeQL Source</a></p> <p>Example schema for a social network graph:</p> <pre><code>define title sub attribute, value string; event-date sub attribute, abstract, value datetime; approved-date sub event-date; ## an abstract relation, only to be subtyped by other relations request sub relation, abstract, owns approved-date, relates subject, relates requester, relates respondent; friendship sub relation, relates friend, plays friend-request:friendship, plays friendship-list:listed; ## an example of subtyping in TypeDB friend-request sub request, relates friendship as subject, relates friend-requester as requester, relates friend-respondent as respondent; friendship-list sub relation, owns title, relates owner, relates listed; person sub entity, plays friendship:friend, plays friend-request:friend-requester, plays friend-request:friend-respondent, plays friendship-list:owner; </code></pre> <p>Java client:</p> <pre><code>package com.vaticle.doc.examples; import com.vaticle.typedb.client.TypeDB; import com.vaticle.typedb.client.api.connection.TypeDBClient; import com.vaticle.typedb.client.api.connection.TypeDBOptions; import com.vaticle.typedb.client.api.connection.TypeDBSession; import com.vaticle.typedb.client.api.connection.TypeDBTransaction; import static com.vaticle.typeql.lang.TypeQL.*; import com.vaticle.typeql.lang.query.*; import com.vaticle.typedb.client.api.answer.ConceptMap; import java.util.stream.Stream; import java.util.List; public class SocialNetworkQuickstartQuery { public static void main(String[] args) { TypeDBClient client = TypeDB.coreClient("localhost:1729"); try (TypeDBSession session = client.session("social_network", TypeDBSession.Type.DATA)) { TypeDBOptions options = TypeDBOptions.core().infer(true); // enable reasoning try (TypeDBTransaction transaction = session.transaction(TypeDBTransaction.Type.WRITE, options)) { TypeQLMatch query = match( var().rel("employer", var("org")).rel("employee", var("per")).isa("employment"), var("per").has("full-name", var("per-fn")), var("org").has("name", var("org-n")) ); Stream&lt;ConceptMap&gt; answers = transaction.query().match(query); answers.forEach(answer -&gt; { System.out.println(answer.get("per-fn").asAttribute().getValue()); System.out.println(answer.get("org-n").asAttribute().getValue()); System.out.println(" - - - - - - - - "); }); } } } } </code></pre> https://tedneward.github.io/Research/storage/velocitydb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/velocitydb/index.html <p><a href="http://velocitydb.com/">Website</a></p> <p>Commercial with a free trial.</p> https://tedneward.github.io/Research/storage/weaver/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/weaver/index.html <p><a href="https://github.com/dubey/weaver">Source</a></p> https://tedneward.github.io/Research/storage/yugabytedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/yugabytedb/index.html <p><a href="https://www.yugabyte.com/">Website</a> | <a href="https://github.com/yugabyte">Source</a></p> <p>It allows you to host your database on multiple clouds at the same time thereby giving you more control over your database system and eliminating vendor lock-in. This also enables you to deploy your cloud instance in different regions which in turn provides you with a higher degree of fail tolerance for your database. YugaByte supports both SQL and NoSQL APIs. This gives developers the flexibility and freedom to choose the type of API that works best with respect to their familiarity and use case. Besides ensuring high performance and scalability, it also provides high availability.</p> https://tedneward.github.io/Research/storage/sharedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/sharedb/index.html <p>"ShareDB is a realtime database backend based on Operational Transformation (OT) of JSON documents. It is the realtime backend for the DerbyJS web application framework.</p> <p>"Features:</p> <ul> <li>Realtime synchronization of any JSON document</li> <li>Concurrent multi-user collaboration</li> <li>Synchronous editing API with asynchronous eventual consistency</li> <li>Realtime query subscriptions</li> <li>Simple integration with any database - MongoDB, PostgresQL (experimental)</li> <li>Horizontally scalable with pub/sub integration</li> <li>Projections to select desired fields from documents and operations</li> <li>Middleware for implementing access control and custom extensions</li> <li>Ideal for use in browsers or on the server</li> <li>Offline change syncing upon reconnection</li> <li>In-memory implementations of database and pub/sub for unit testing"</li> </ul> <p><a href="https://github.com/share/sharedb">Source</a></p> https://tedneward.github.io/Research/storage/sqlite/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/sqlite/index.html <p><a href="https://sqlite.org/index.html">Website</a> | <a href="https://sqlite.org/c3ref/intro.html">C/C++ Interface</a> | <a href="https://sqlite.org/tclsqlite.html">TCL Interface</a> | <a href="https://sqlite.org/src">Source</a> | <a href="https://www.sqlite.org/fileformat.html">File format for SQLite databases</a> and <a href="https://sqlite.org/fileformat2.html">here?</a> | <a href="https://sqlite.org/appfileformat.html">SQLite as an "application format"</a></p> <p><a href="https://www.tutorialspoint.com/sqlite">SQLite Tutorial</a> | <a href="https://sqlite-internal.pages.dev/">SQLite File Format Viewer</a> (<a href="https://github.com/invisal/sqlite-internal">Source</a>)</p> <p><a href="https://github.com/proofrock/ws4sqlite">WS4SQLite</a>: Query sqlite via http</p> <h2>Articles</h2> <p>How does SQLite work? <a href="https://jvns.ca/blog/2014/09/27/how-does-sqlite-work-part-1-pages/">Part 1</a> | <a href="https://jvns.ca/blog/2014/10/02/how-does-sqlite-work-part-2-btrees/">Part 2</a></p> <p><a href="https://fly.io/blog/sqlite-virtual-machine/">How the SQLite VM works</a></p> <p><a href="https://jlongster.com/future-sql-web">"A future for SQL on the Web"</a>: Using <a href="/storage/indexeddb">IndexedDB</a> in the browser as a "file system" underneath a SQLite implementation compiled to WASM. Resulting project is <a href="https://github.com/jlongster/absurd-sql">absurd-sql</a>. Makes use of <a href="/storage/sqljs">sql.js</a> and a few source changes to it.</p> <p><a href="https://christine.website/blog/sqlite-json-munge-2022-01-04">"Bashing JSON into Shape with SQLite"</a>: "SQLIte added JSON functions to allow you to munge and modify JSON data in whatever creative ways you want."</p> <p><a href="https://tirkarthi.github.io/programming/2022/02/26/sqlite-json-improvements.html">"JSON improvements in SQLite 3.38.0"</a></p> <p><a href="https://medium.com/capital-one-tech/experimenting-with-sqlite-in-ios-ae9dec92dbaf">"Experimenting With SQLite in iOS"</a></p> <h2>Plugins/additions/enhancements</h2> <p><a href="https://github.com/vlcn-io/cr-sqlite">cr-sqlite</a>: a loadable SQLite extension by Matt Wonlaw that "allows merging different SQLite databases together that have taken independent writes".</p> <ul> <li><a href="https://til.simonwillison.net/sqlite/cr-sqlite-macos">Trying cr-sqlite out on macOS</a></li> </ul> <p><a href="https://github.com/maxpert/marmot">Marmot</a>: A distributed SQLite replicator built on top of NATS.</p> <p><a href="https://www.zetetic.net/sqlcipher/">SQLCypher</a> is an extension to SQLite adding encryption. Open-source and commercial.</p> <p><a href="https://dirtylittlesql.com/">A fast SQLite PWA notebook for CSV files</a> (<a href="https://github.com/mwenge/dirtylittlesql.com">Source</a>, uses <a href="https://github.com/sql-js/sql.js">sql.js</a> (The database is a web-assembly version of SQLite called sql.js that runs in your browser.), <a href="http://www.dessus.com/files/vsv.c">VSV</a> (It also uses a SQLite extension called vsv to load delimited text files quickly.), <a href="https://codemirror.net/">Codemirror</a> (Syntax highlighting), <a href="https://sheetjs.com/">sheet.js</a> (Excel and Openoffice support.), <a href="https://chartjs.org/">chart.js</a> (Charts), and <a href="https://github.com/localForage/localForage">LocalForage</a> (IndexedDB for efficient local storage.) under the hood.)</p> <p><a href="https://sqlitestudio.pl/">SQLStudio</a> (<a href="https://github.com/pawelsalawa/sqlitestudio">Source</a>): A free, open source, multi-platform SQLite database manager.</p> <h2>Access</h2> <h3>Typescript</h3> <ul> <li><a href="https://kysely.dev/">Kysely</a> (<a href="https://github.com/kysely-org/kysely">Source</a>): The type-safe SQL query builder for TypeScript</li> </ul> <h3>Swift/iOS</h3> <p>By default, SQLite from Swift/iOS is an exercise in C integration. Numerous wrappers are available.</p> <ul> <li><a href="https://github.com/stephencelis/SQLite.swift">SQLite.swift</a>: Type-safe Swift wrapper over sqlite3 C API</li> <li><a href="https://github.com/wildthink/SQift">Sqift</a>: Powerful Swift wrapper for SQLite</li> </ul> <h3>Java</h3> <ul> <li><a href="https://github.com/xerial/sqlite-jdbc">SQLite-jdbc</a>: Straight JDBC driver for SQLite</li> </ul> <h3>C#/.NET</h3> <p><a href="https://docs.microsoft.com/en-us/dotnet/standard/data/sqlite/">Microsoft.Data.Sqlite</a>: <code>dotnet add package Microsoft.Data.Sqlite</code></p> <pre><code class="language-cs">using (var connection = new SqliteConnection("Data Source=hello.db")) { connection.Open(); var command = connection.CreateCommand(); command.CommandText = @" SELECT name FROM user WHERE id = $id "; command.Parameters.AddWithValue("$id", id); using (var reader = command.ExecuteReader()) { while (reader.Read()) { var name = reader.GetString(0); Console.WriteLine($"Hello, {name}!"); } } } </code></pre> <h3>Native</h3> <p><a href="https://cgsql.dev/">CG/SQL</a> is a code generation system for the popular SQLite library that allows developers to write stored procedures in a variant of Transact-SQL (T-SQL) and compile them into C code that uses SQLite’s C API to do the coded operations. CG/SQL enables engineers to create highly complex stored procedures with very large queries, without the manual code checking that existing methods require. (Produces C source that can be compiled, and the language is flexible enough to do a fair amount of typical C dev without leaving CQL.)</p> <ul> <li><a href="https://cgsql.dev/docs/introduction">Docs</a></li> <li><a href="https://cgsql.dev/cql-guide/ch01">Language Guide</a></li> <li>Internals <a href="https://cgsql.dev/cql-guide/int01">Part 1</a>, <a href="https://cgsql.dev/cql-guide/int02">Part 2</a>: An interesting look into how it works</li> <li><a href="https://github.com/facebookincubator/CG-SQL">Source</a></li> </ul> https://tedneward.github.io/Research/storage/superdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/superdb/index.html <p><a href="https://superdb.org/docs/">Website</a> | <a href="https://github.com/brimdata/super">Source</a></p> <hr> <p>Used to be the <a href="https://zed.brimdata.io/">Zed Project and language</a> <a href="https://github.com/brimdata">Source</a></p> <p>"Zed" is an umbrella term that describes a number of different elements of the system:</p> <ul> <li>The Zed data model is the abstract definition of the data types and semantics that underlie the Zed formats.</li> <li>The Zed formats are a family of sequential (ZNG), columnar (VNG), and human-readable (ZSON) formats that all adhere to the same abstract Zed data model.</li> <li>A Zed lake is a collection of Zed data stored across one or more data pools with ACID commit semantics and accessed via a Git-like API.</li> <li>The Zed language is the system's pipeline language for performing queries, searches, analytics, transformations, or any of the above combined together.</li> <li>A Zed query is a Zed script that performs search and/or analytics.</li> <li>A Zed shaper is a Zed script that performs data transformation to shape the input data into the desired set of organizing Zed data types called "shapes", which are traditionally called schemas in relational systems but are much more flexible in the Zed system.</li> </ul> <p>--</p> <p>Also related: The <a href="https://zui.brimdata.io/docs">Zui Desktop Application</a>, a graphical user interface for exploring data in Zed lakes.</p> https://tedneward.github.io/Research/storage/terminusdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/terminusdb/index.html <p><a href="https://terminusdb.com/">Website</a> | <a href="https://github.com/terminusdb/terminusdb">Source</a></p> https://tedneward.github.io/Research/storage/tinkerpop/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/tinkerpop/index.html <p>"Apache TinkerPop™ is an open source, vendor-agnostic, graph computing framework distributed under the commercial friendly Apache2 license. When a data system is TinkerPop-enabled, its users are able to model their domain as a graph and analyze that graph using the Gremlin graph traversal language. Furthermore, all TinkerPop-enabled systems integrate with one another allowing them to easily expand their offerings as well as allowing users to choose the appropriate graph technology for their application. Sometimes an application is best served by an in-memory, transactional graph database. Sometimes a multi-machine distributed graph database will do the job. Or perhaps the application requires both a distributed graph database for real-time queries and, in parallel, a Big(Graph)Data processor for batch analytics. Whatever the application's requirements, there exists a TinkerPop-enabled graph system out there to meet its needs."</p> <p><a href="http://tinkerpop.apache.org/">Website</a> | <a href="https://www.apache.org/dyn/closer.lua/tinkerpop/">Source</a></p> https://tedneward.github.io/Research/storage/unqlite/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/unqlite/index.html <p>"UnQLite is a in-process software library which implements a self-contained, serverless, zero-configuration, transactional NoSQL database engine. UnQLite is a document store database similar to MongoDB, Redis, CouchDB etc. as well a standard Key/Value store similar to BerkeleyDB, LevelDB, etc."</p> <p><a href="https://unqlite.org/">Website</a> | <a href="https://github.com/symisc/unqlite">Github</a></p> <p>"Jx9 is the embedded scripting language which power the document-store interface to UnQLite. Jx9 is a Turing complete programming language based on JSON and implemented as a library in the UnQLite core." (<a href="https://unqlite.org/jx9.html">Website</a>)</p> https://tedneward.github.io/Research/storage/veloxdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/veloxdb/index.html <p><a href="https://www.vlxdb.com/index.html">Website</a> | <a href="https://www.vlxdb.com/guide/introduction.html">Docs</a> | <a href="https://github.com/VeloxDB/VeloxDB">Source</a></p> <p><a href="https://dev.to/defufna/introducing-veloxdb-26j1">"Introducing VeloxDB"</a></p> https://tedneward.github.io/Research/storage/webstorage/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/webstorage/index.html <p>"The two mechanisms within Web Storage are as follows:<br> * sessionStorage maintains a separate storage area for each given origin that's available for the duration of the page session (as long as the browser is open, including page reloads and restores)<br> * Stores data only for a session, meaning that the data is stored until the browser (or tab) is closed.<br> * Data is never transferred to the server.<br> * Storage limit is larger than a cookie (at most 5MB).</p> <ul> <li>localStorage does the same thing, but persists even when the browser is closed and reopened.</li> <li>Stores data with no expiration date, and gets cleared only through JavaScript, or clearing the Browser cache / Locally Stored Data.</li> <li>Storage limit is the maximum amongst the three.</li> </ul> <p>These mechanisms are available via the <code>Window.sessionStorage</code> and <code>Window.localStorage</code> properties (to be more precise, in supporting browsers the Window object implements the <code>WindowLocalStorage</code> and <code>WindowSessionStorage</code> objects, which the <code>localStorage</code> and <code>sessionStorage</code> properties hang off) — invoking one of these will create an instance of the <code>Storage</code> object, through which data items can be set, retrieved and removed. A different <code>Storage</code> object is used for the <code>sessionStorage</code> and <code>localStorage</code> for each origin — they function and are controlled separately."</p> <p><a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API">Website</a></p> https://tedneward.github.io/Research/storage/zvec/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/zvec/index.html <p><a href="https://zvec.org/en/">Website</a> | <a href="https://github.com/alibaba/zvec">Source</a></p> <p align="center"> <a href="https://zvec.org/en/docs/quickstart/">🚀 <strong>Quickstart</strong> </a> | <a href="https://zvec.org/en/">🏠 <strong>Home</strong> </a> | <a href="https://zvec.org/en/docs/">📚 <strong>Docs</strong> </a> | <a href="https://zvec.org/en/docs/benchmarks/">📊 <strong>Benchmarks</strong> </a> | <a href="https://discord.gg/rKddFBBu9z">🎮 <strong>Discord</strong> </a> </p> <p><strong>Zvec</strong> is an open-source, in-process vector database — lightweight, lightning-fast, and designed to embed directly into applications. Built on <strong>Proxima</strong> (Alibaba's battle-tested vector search engine), it delivers production-grade, low-latency, scalable similarity search with minimal setup.</p> <h2>💫 Features</h2> <ul> <li><strong>Blazing Fast</strong>: Searches billions of vectors in milliseconds.</li> <li><strong>Simple, Just Works</strong>: <a href="#-installation">Install</a> and start searching in seconds. No servers, no config, no fuss.</li> <li><strong>Dense + Sparse Vectors</strong>: Work with both dense and sparse embeddings, with native support for multi-vector queries in a single call.</li> <li><strong>Hybrid Search</strong>: Combine semantic similarity with structured filters for precise results.</li> <li><strong>Runs Anywhere</strong>: As an in-process library, Zvec runs wherever your code runs — notebooks, servers, CLI tools, or even edge devices.</li> </ul> <h2>📦 Installation</h2> <h3><a href="https://pypi.org/project/zvec/">Python</a></h3> <p><strong>Requirements</strong>: Python 3.10 - 3.12</p> <pre><code class="language-bash">pip install zvec </code></pre> <h3><a href="https://www.npmjs.com/package/@zvec/zvec">Node.js</a></h3> <pre><code class="language-bash">npm install @zvec/zvec </code></pre> <h3>✅ Supported Platforms</h3> <ul> <li>Linux (x86_64, ARM64)</li> <li>macOS (ARM64)</li> </ul> <h3>🛠️ Building from Source</h3> <p>If you prefer to build Zvec from source, please check the <a href="https://zvec.org/en/docs/build/">Building from Source</a> guide.</p> <h2>⚡ One-Minute Example</h2> <pre><code class="language-python">import zvec # Define collection schema schema = zvec.CollectionSchema( name="example", vectors=zvec.VectorSchema("embedding", zvec.DataType.VECTOR_FP32, 4), ) # Create collection collection = zvec.create_and_open(path="./zvec_example", schema=schema) # Insert documents collection.insert([ zvec.Doc(id="doc_1", vectors={"embedding": [0.1, 0.2, 0.3, 0.4]}), zvec.Doc(id="doc_2", vectors={"embedding": [0.2, 0.3, 0.4, 0.1]}), ]) # Search by vector similarity results = collection.query( zvec.VectorQuery("embedding", vector=[0.4, 0.3, 0.3, 0.1]), topk=10 ) # Results: list of {'id': str, 'score': float, ...}, sorted by relevance print(results) </code></pre> <h2>📈 Performance at Scale</h2> <p>Zvec delivers exceptional speed and efficiency, making it ideal for demanding production workloads.</p> <p><img src="https://zvec.oss-cn-hongkong.aliyuncs.com/qps_10M.svg" width="800" alt="Zvec Performance Benchmarks"></p> <p>For detailed benchmark methodology, configurations, and complete results, please see our <a href="https://zvec.org/en/docs/benchmarks/">Benchmarks documentation</a>.</p> https://tedneward.github.io/Research/storage/simpledb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/simpledb/index.html <p><a href="http://www.cs.bc.edu/~sciore/simpledb/">Website</a> | <a href="http://www.cs.bc.edu/%7Esciore/simpledb/SimpleDB_3.4.zip">Code</a> (v3.4)</p> <blockquote> <p>SimpleDB is a multi-user transactional database server written in Java, which interacts with Java client programs via JDBC. The system is intended for pedagogical use only. The code is clean and compact. The APIs are straightforward. The learning curve is relatively small. Everything about it is geared towards improving the experience of a database system internals course. Consequently, the system is intentionally bare-bones. It implements only a small fraction of SQL and JDBC, and does little or no error checking. Although it is a great teaching tool, I can't imagine that anyone would want to use it for anything else.</p> <p>The SimpleDB code is an integral part of my textbook <a href="https://www.amazon.com/dp/3030338355/">Database Design and Implementation</a>, published by Springer. If you are interested in understanding how (and why) the software works, your best bet is to take a look at the text.</p> <p>I teach a course at Boston College (CSCI3357: Database Systems and Implementation), which does not require previous database knowledge. ... Here are the <a href="http://www.cs.bc.edu/~sciore/simpledb/hwassignments.zip">homework assignments</a> that I used in my most recent offering of the course.</p> </blockquote> https://tedneward.github.io/Research/storage/sqljs/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/sqljs/index.html <p><a href="http://sql.js.org/">Website</a> | <a href="https://github.com/sql-js/sql.js">Source</a></p> <p>"sql.js is a port of <a href="sqlite.html">SQLite</a> to Webassembly, by compiling the SQLite C code with Emscripten, with contributed math and string extension functions included. It uses a virtual database file stored in memory, and thus doesn't persist the changes made to the database. However, it allows you to import any existing sqlite file, and to export the created database as a JavaScript typed array.</p> <p>"There are no C bindings or node-gyp compilation here, sql.js is a simple JavaScript file, that can be used like any traditional JavaScript library. If you are building a native application in JavaScript (using Electron for instance), or are working in node.js, you will likely prefer to use a native binding of SQLite to JavaScript."</p> https://tedneward.github.io/Research/storage/surrealdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/surrealdb/index.html <p><a href="https://surrealdb.com">Website</a> |</p> <p>When it comes to data modeling it’s extremely flexible, making it an ideal choice for recommendation engines, social networks, etc. In this regard, it’s similar to the popular graph database Neo4J.</p> <p>It’s also very simple to use with any SQL type query language and also provides the ability to perform real-time queries efficiently.</p> <p>For instance, here’s an insertion query in SurrealQL that looks almost identical to an SQL insert query:</p> <pre><code>INSERT INTO company { name: 'Siddhant', age: 24, hobbies: ['technical writing', 'basketball'] }; </code></pre> <p>Moreover, you can use SurrealQL to directly connect your Surreal DB to your client applications.</p> https://tedneward.github.io/Research/storage/tiledb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/tiledb/index.html <p><a href="https://www.tiledb.com/">Website</a> | <a href="https://github.com/TileDB-Inc/TileDB">Source</a> |</p> <p>APIs:</p> <ul> <li>C (examples, API docs)</li> <li>C++ (examples, API docs)</li> <li>Python</li> <li>R</li> <li>Java</li> <li>Go</li> <li>C#</li> </ul> https://tedneward.github.io/Research/storage/titan/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/titan/index.html <p><a href="http://titan.thinkaurelius.com/">Website</a> | <a href="https://github.com/thinkaurelius/titan/">Source</a></p> <p>Titan is a transactional database that can support thousands of concurrent users executing complex graph traversals in real time.</p> <p>In addition, Titan provides the following features:</p> <ul> <li>Elastic and linear scalability for a growing data and user base.</li> <li>Data distribution and replication for performance and fault tolerance.</li> <li>Multi-datacenter high availability and hot backups.</li> <li>Support for ACID and eventual consistency.</li> <li>Support for various storage backends: <ul> <li>Apache <a href="../cassandra">Cassandra</a></li> <li>Apache <a href="../hbase">HBase</a></li> <li>Oracle <a href="../berkeleydb">BerkeleyDB</a></li> </ul> </li> <li>Support for global graph data analytics, reporting, and ETL through integration with big data platforms: <ul> <li>Apache Spark</li> <li>Apache Giraph</li> <li>Apache Hadoop</li> </ul> </li> <li>Support for geo, numeric range, and full-text search via: <ul> <li>ElasticSearch</li> <li>Solr</li> <li>Lucene</li> </ul> </li> <li>Native integration with the TinkerPop graph stack: <ul> <li>Gremlin graph query language</li> <li>Gremlin graph server</li> <li>Gremlin applications</li> </ul> </li> </ul> <h3>Quick start</h3> <pre><code>&lt;dependency&gt; &lt;groupId&gt;com.thinkaurelius.titan&lt;/groupId&gt; &lt;artifactId&gt;titan-core&lt;/artifactId&gt; &lt;version&gt;1.0.0&lt;/version&gt; &lt;/dependency&gt; &lt;!-- core, all, cassandra, hbase, berkeleyje, es, lucene --&gt; </code></pre> <p>then</p> <pre><code>// who is hercules' grandfather? g.V.has('name','hercules').out('father').out('father').name </code></pre> https://tedneward.github.io/Research/storage/uor/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/uor/index.html <p><a href="https://www.uor.foundation/">Website</a> | <a href="https://github.com/UOR-Foundation">Source</a> | <a href="https://uor-foundation.discourse.group/">Discourse Group</a></p> <h2>Universal Object Reference (UOR)</h2> <p>What if every piece of data on the internet, including every website, API, AI model, or configuration file, could be referenced, trusted, and composed like code?</p> <p>Universal Object Reference (UOR) makes this possible.</p> <p>UOR is an open source data system that reimagines the internet as a single, structured and composable dataset. It treats everything as an object. Each is uniquely identified by a cryptographic hash and described by its intrinsic attributes, such as size, media type and content digest, rather than by location.</p> <p>This Rosetta Stone for data changes the paradigm.</p> <p>Instead of chasing data by where it lives, you access it by what it is. The result is secure, deterministic, and interoperable across systems.</p> <p>The outcome is a universal layer for decentralized, serverless and composable infrastructure. It invites developers, researchers and architects to help build the next foundation of the internet.</p> https://tedneward.github.io/Research/storage/vitess/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/vitess/index.html <p><a href="https://vitess.io/">Website</a></p> https://tedneward.github.io/Research/storage/whitedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/whitedb/index.html <p><a href="http://whitedb.org/">Website</a> | <a href="https://github.com/priitj/whitedb">Source</a></p> <h3>Project goals</h3> <ul> <li>speed</li> <li>portability</li> <li>simplicity and small footprint</li> <li>low memory usage</li> <li>easy to use in embedded systems</li> <li>graph database applications</li> <li>extended rdf database applications</li> <li>fast interprocess communication<br> For the next version we're working on JSON support and a REST tool.</li> </ul> <h3>Data storage</h3> <p>Data is kept in shared memory by default, making all the data accessible to separate processes.</p> <p>Each database record is a tuple of N elements, encoded in WhiteDB-s simple compact format. You can store both conventional datatypes and direct pointers to records: the latter enables highly efficient traversal of complex data.</p> <h3>Supported features</h3> <ul> <li>indexes (T-tree)</li> <li>persistence through logging and memory dumps</li> <li>concurrency through locking</li> <li>limited queries (conjunctive only)</li> <li>CSV and RDF support</li> <li>Linux and Windows</li> <li>Python bindings</li> <li>command line utility tools</li> </ul> https://tedneward.github.io/Research/storage/sones/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/sones/index.html <p>No updates for 8 years (2013?).</p> <p><a href="https://github.com/sones/sones">Source</a></p> https://tedneward.github.io/Research/storage/sqlserver/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/sqlserver/index.html <h3>Tools</h3> <ul> <li><a href="https://github.com/tsqllint/tsqllint">tsqllint</a>: Configurable linting for TSQL. Install with <code>$ dotnet tool install --global TSQLLint</code> or <code>brew tap tsqllint/homebrew-tsqllint; brew install tsqllint</code> for macOS/Linux or <code>npm install tsqllint -t</code> for npm. Supports the creation of custom plugins (implemenent an interface in CLR code).</li> <li><a href="https://github.com/davebally/TSQL-Smells">TSQL-Smells</a>: "TSQL Smells Finder". Last meaningful activity in 2016/2018.</li> </ul> <h3>Reading</h3> <ul> <li><a href="http://www.red-gate.com/community/books/ssc-7">Best of SQLServerCentral.com Vol 7</a> *(RedGate, By SQLServerCentral Authors)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-maintenance-plans">Brad's Sure Guide to SQL Server Maintenance Plans</a> - Brad McGehee (PDF) (email address <em>requested</em>)</li> <li><a href="https://www.red-gate.com/library/defensive-database-programming">Defensive Database Programming</a> - Alex Kuznetsov (PDF)</li> <li><a href="https://www.red-gate.com/library/fundamentals-of-sql-server-2012-replication">Fundamentals Of SQL Server 2012 Replication</a> - Sebastian Meine (PDF) (email address <em>requested</em>)</li> <li><a href="http://www.red-gate.com/community/books/exceptional-dba-book">How to Become an Exceptional DBA, Second edition</a> - Brad McGehee (PDF)</li> <li><a href="http://www.red-gate.com/products/sql-development/sql-prompt/entrypage/sql-query-optimizer-ebook3">Inside the SQL Server Query Optimizer</a> - Benjamin Nevarez (PDF) (email address <em>requested</em>)</li> <li><a href="http://social.technet.microsoft.com/wiki/contents/articles/11608.e-book-gallery-for-microsoft-technologies-en.aspx#IntroducingMicrosoftSQLServer2008R2">Introducing Microsoft SQL Server 2008 R2</a></li> <li><a href="http://social.technet.microsoft.com/wiki/contents/articles/11608.e-book-gallery-for-microsoft-technologies-en.aspx#IntroducingMicrosoftSQLServer2012">Introducing Microsoft SQL Server 2012</a></li> <li><a href="http://blogs.msdn.com/b/microsoft_press/archive/2014/04/02/free-ebook-introducing-microsoft-sql-server-2014.aspx">Introducing Microsoft SQL Server 2014</a></li> <li><a href="http://www.red-gate.com/community/books/mastering-sql-server-profiler">Mastering SQL Server Profiler</a> - Brad McGehee (PDF)</li> <li><a href="http://goalkicker.com/MicrosoftSQLServerBook/">Microsoft SQL Server Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="http://www.red-gate.com/community/books/dynamic-management-views">Performance Tuning with SQL Server Dynamic Management Views</a> - Tim Ford, Louis Davidson (PDF)</li> <li><a href="http://www.red-gate.com/community/books/protecting-data">Protecting SQL Server Data</a> - John Magnabosco (PDF)</li> <li><a href="http://social.technet.microsoft.com/wiki/contents/articles/11608.e-book-gallery-for-microsoft-technologies-en.aspx#SQLServer2012Tutorials%3AReportingServices">SQL Server 2012 Tutorials: Reporting Services</a></li> <li><a href="https://www.packtpub.com/free-ebooks/sql-server-2017-administrators-guide">SQL Server 2017 Administrator's Guide</a> - Marek Chmel, Vladimír Mužný (Packt account <em>required</em>)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-backup-and-restore">SQL Server Backup and Restore</a> - Shawn McGehee (PDF) (email address <em>requested</em>)</li> <li><a href="https://assets.red-gate.com/community/books/sql-server-execution-plans-3rd-edition.pdf">SQL Server Execution Plans, Third Edition</a> - Grant Fritchey (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/sql_server_for_c_sharp_developers_succinctly">SQL Server for C# Developers Succinctly</a> - Sander Rossel</li> <li><a href="http://www.red-gate.com/community/books/sql-server-hardware">SQL Server Hardware</a> - Glenn Berry (PDF)</li> <li><a href="http://www.red-gate.com/library/sql-server-internals-in-memory-oltp">SQL Server Internals: In-Memory OLTP</a> - Kalen Delaney (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/sql-server-metadata-succinctly">SQL Server Metadata Succinctly</a> - Joseph D. Booth</li> <li><a href="https://www.red-gate.com/products/sql-development/sql-source-control/entrypage/sql-server-source-control-basics">SQL Server Source Control Basics</a> - Rob Sheldon, Rob Richardson, Tony Davis (PDF)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-statistics">SQL Server Statistics</a> - Holger Schmeling (PDF)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-stumpers-v5">SQL Server Stumpers Vol.5</a> (PDF)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-tacklebox">SQL Server Tacklebox</a> - Rodney Landrum (PDF)</li> <li><a href="http://www.red-gate.com/community/books/sql-server-transaction-log-management">SQL Server Transaction Log Management</a> - Tony Davis, Gail Shaw (PDF)</li> <li><a href="http://www.red-gate.com/community/books/art-of-filestream">The Art of SQL Server FILESTREAM</a> - Jacob Sebastian, Sven Aelterman (PDF)</li> <li><a href="https://www.red-gate.com/library/the-art-of-xsd">The Art of XSD</a> - Jacob Sebastian (PDF)</li> <li><a href="https://www.red-gate.com/library/the-best-of-sqlservercentral-com-vol-7">The Best of SQLServerCentral.com Vol 7</a> (PDF)</li> <li><a href="https://www.red-gate.com/library/the-redgate-guide-to-sql-server-team-based-development">The Redgate Guide to SQL Server Team-based Development</a> - Phil Factor, Grant Fritchey, Alex Kuznetsov, Mladen Prajdić (PDF)</li> <li><a href="http://www.red-gate.com/community/books/accidental-dba">Troubleshooting SQL Server: A Guide for the Accidental DBA</a> - Jonathan Kehayias, Ted Krueger (PDF)</li> </ul> https://tedneward.github.io/Research/storage/taffydb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/taffydb/index.html <p><a href="http://taffydb.com/">Website</a> | <a href="https://github.com/typicaljoe/taffydb">Source</a></p> https://tedneward.github.io/Research/storage/timescaledb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/timescaledb/index.html <p><a href="https://www.timescale.com/">Website</a> | <a href="https://github.com/timescale/timescaledb">Source</a></p> https://tedneward.github.io/Research/storage/trino/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/trino/index.html <p><a href="https://trino.io/">Website</a> | <a href="https://github.com/trinodb/trino">Source</a></p> https://tedneward.github.io/Research/storage/vedis/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/vedis/index.html <p><a href="https://vedis.symisc.net/">Website</a></p> <p>C/C++ API</p> https://tedneward.github.io/Research/storage/voltdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/voltdb/index.html <p><a href="https://voltactivedata.com/">Website</a> | <a href="https://github.com/voltdb/voltdb/">Source</a></p> <p>From the GitHub repo:</p> <blockquote> <p>This version is provided for students, hobbyists, and researchers interested in testing an ultra-fast, scalable ACID database. But all new and production-ready features are available through the commercial product from <a href="https://www.voltactivedata.com">https://www.voltactivedata.com</a>.</p> <p>Volt Active Data offers the fully open source, AGPL3-licensed Community Edition of Volt Active Data through GitHub here: <a href="https://github.com/voltdb/voltdb/">https://github.com/voltdb/voltdb/</a> . The Community Edition has full application compatibility and provides everything needed to run a real-time, in-memory SQL database with datacenter-local redundancy and snapshot-based disk persistence.</p> <p>Trials of the enterprise edition of Volt Active Data are available from the Volt Active Data website at the following URL: <a href="https://www.voltactivedata.com">https://www.voltactivedata.com</a> . The commercial editions add operational features to support industrial-strength durability, manageability, and availability, including per-transaction disk-based persistence, multi-datacenter replication, elastic online expansion, live online upgrade, etc.</p> </blockquote> https://tedneward.github.io/Research/storage/xata/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/xata/index.html <p><a href="https://xata.io/">Website</a> | Commercial</p> <p>Treats your data as a spreadsheet making it very easy for developers to build applications that require a flexible schema. You can also imagine it as a developer-friendly alternative to AirTable or Notion.</p> <p>It comes with a full-text search out of the box and can also help you visualize the relationships in real time between your tables using the schema editor it provides. It also supports branching like Dolt and is primarily used for processing large volumes of data in real-time. It also supports some advanced querying capabilities like aggregations, filters, joins, etc.</p> https://tedneward.github.io/Research/storage/orioledb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/orioledb/index.html <p><a href="https://github.com/orioledb/orioledb">Source</a></p> <p>OrioleDB consists of an extension, building on the innovative table access method framework and other standard Postgres extension interfaces. By extending and enhancing the current table access methods, OrioleDB opens the door to a future of more powerful storage models that are optimized for cloud and modern hardware architectures.</p> <p>OrioleDB is currently distributed under the standard PostgreSQL license.</p> <ol> <li> <p>Designed for modern hardware. OrioleDB design avoids legacy CPU bottlenecks on modern servers containing dozens and hundreds CPU cores, providing optimized usage of modern storage technologies such as SSD and NVRAM.</p> </li> <li> <p>Reduced maintenance needs. OrioleDB implements the concepts of undo log and page-mergins, eliminating the need for dedicated garbage collection processes. Additionally, OrioleDB implements default 64-bit transaction identifiers, thus eliminating the well-known and painful wraparound problem.</p> </li> <li> <p>Designed to be distributed. OrioleDB implements a row-level write-ahead log with support for parallel apply. This log architecture is optimized for raft consensus-based replication allowing the implementation of active-active multimaster.</p> </li> </ol> <p>The key technical differentiations of OrioleDB are as follows:</p> <ol> <li> <p>No buffer mapping and lock-less page reading. In-memory pages in OrioleDB are connected with direct links to the storage pages. This eliminates the need for in-buffer mapping along with its related bottlenecks. Additionally, in OrioleDB in-memory page reading doesn't involve atomic operations. Together, these design decisions bring vertical scalability for Postgres to the whole new level.</p> </li> <li> <p>MVCC is based on the UNDO log concept. In OrioleDB, old versions of tuples do not cause bloat in the main storage system, but eviction into the undo log comprising undo chains. Page-level undo records allow the system to easily reclaim space occupied by deleted tuples as soon as possible. Together with page-mergins, these mechanisms eliminate bloat in the majority of cases. Dedicated VACUUMing of tables is not needed as well, removing a significant and common cause of system performance deterioration and database outages.</p> </li> <li> <p>Copy-on-write checkpoints and row-level WAL. OrioleDB utilizes copy-on-write checkpoints, which provides a structurally consistent snapshot of data every moment of time. This is friendly for modern SSDs and allows row-level WAL logging. In turn, row-level WAL logging is easy to parallelize (done), compact and suitable for active-active multimaster (planned).</p> </li> </ol> https://tedneward.github.io/Research/storage/perst/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/perst/index.html <p><a href="https://www.mcobject.com/perst/">Website</a></p> https://tedneward.github.io/Research/storage/postgresql/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/postgresql/index.html <p><a href="https://www.postgresql.org/">Website</a> | <a href="https://git.postgresql.org/gitweb/?p=postgresql.git">Source</a> | <a href="https://wiki.postgresql.org/wiki/Main_Page">Wiki</a></p> <p><a href="https://www.postgresql.org/download/">Download</a> | Also installable via Homebrew (macOS): <code>brew install postgresql</code></p> <p><a href="https://en.wikipedia.org/wiki/PostgreSQL">Wikipedia</a></p> <h2>Implementations</h2> <h3>Distributions</h3> <ul> <li><a href="https://postgresapp.com/">Postgres.app</a> - The Easiest Way to Get Started with PostgreSQL on macOS.</li> <li><a href="https://github.com/mihasic/PostgreSql.Binaries.Lite">PostgreSql.Binaries.Lite</a> - Minimum set of Windows binaries of the PostgreSQL database. Also made available through NuGet.</li> <li><a href="https://github.com/snaplet/postgres-wasm">Postgres-wasm</a>: A PostgreSQL server instance running in a virtual machine running in the browser</li> </ul> <h3>Optimizations</h3> <ul> <li><a href="https://philbooth.me/blog/nine-ways-to-shoot-yourself-in-the-foot-with-postgresql">9 ways to shoot yourself in the foot</a></li> </ul> <h3>PaaS <em>(PostgreSQL as a Service)</em></h3> <ul> <li><a href="https://aiven.io/postgresql">Aiven PostgreSQL</a> - PostgreSQL as a service in AWS, Azure, DigitalOcean, Google Cloud and UpCloud; plans range from $19/month single node instances to large highly-available setups, free trial for two weeks.</li> <li><a href="https://aws.amazon.com/rds/postgresql/">Amazon RDS for PostgreSQL</a> - Amazon Relational Database Service (RDS) for PostgreSQL</li> <li><a href="https://azure.microsoft.com/en-us/services/postgresql/">Azure Database for PostgreSQL</a> - Azure Database for PostgreSQL provides fully managed, enterprise-ready community PostgreSQL database as a service. It provides builtin HA, elastic scaling and native integration with Azure ecosystem.</li> <li><a href="https://www.crunchydata.com/products/crunchy-bridge/">Crunchy Bridge</a> - Fully managed Postgres from the Postgres experts. Available across all major cloud providers: Amazon AWS, Google GCP, Microsoft Azure. No lock-in with full super-user support.</li> <li><a href="https://www.databaselabs.io">Database Labs</a> - Get a production-ready cloud PostgreSQL server in minutes, from $20 a month Backups, monitoring, patches, and 24/7 tech support all included.</li> <li><a href="https://www.digitalocean.com/products/managed-databases/">DigitalOcean Managed Databases</a> - Fully managed PostgreSQL databases. No free plan. Starting at $15/mo. Daily backups with point-in-time recovery. Standby nodes with auto-failover.</li> <li><a href="https://www.elephantsql.com/">ElephantSQL</a> - Offers databases ranging from shared servers for smaller projects and proof of concepts, up to enterprise grade multi server setups. Has free plan for up to 5 DBs, 20 MB each.</li> <li><a href="https://cloud.google.com/sql/docs/postgres/">Google Cloud SQL for PostgreSQL</a> - Fully-managed database service that makes it easy to set up, maintain, manage, and administer your PostgreSQL relational databases on Google Cloud Platform.</li> <li><a href="https://elements.heroku.com/addons/heroku-postgresql">Heroku Postgres</a> - Plans from free to huge, operated by PostgreSQL experts. Does not require running your app on Heroku. Free plan includes 10,000 rows, 20 connections, up to two backups, and has PostGIS support.</li> <li><a href="https://render.com/docs/databases">Render Managed PostgreSQL</a> - Secure, reliable, and completely hands-off managed PostgreSQL. Encryption at rest, automated backups, and expandable SSD storage included in all plans. Plans start at $7 per month for 256MB RAM and 1GB storage (free for first 90 days).</li> <li><a href="https://www.scaleway.com/en/database/">Scaleway Managed Database</a> - Fully managed PostgreSQL databases with HA, scaling, and automated backups, hosted in the EU. Starting at €10 per month.</li> </ul> <h3>Server</h3> <ul> <li><a href="https://www.postgres-xl.org/">Postgres-XL</a> - Scalable Open Source PostgreSQL-based Database Cluster.</li> <li><a href="https://bitnine.net/">AgensGraph</a> - Powerful graph database based on the PostgreSQL.</li> <li><a href="https://github.com/greenplum-db/gpdb">Greenplum Database</a> - Open source fork of PostgreSQL for large data volumes.</li> </ul> <h3>Docker images</h3> <ul> <li><a href="https://hub.docker.com/_/postgres/">postgres</a> - Official postgres container (from Docker)</li> <li><a href="https://hub.docker.com/r/citusdata/citus/">citusdata/citus</a> - Citus official images with citus extensions. Based on the official Postgres container.</li> <li><a href="https://hub.docker.com/r/mdillon/postgis/">mdillon/postgis</a> - PostGIS 2.3 on Postgres 9. Based on the official Postgres container.</li> </ul> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="https://edu.postgrespro.com/postgresql_internals-14_en.pdf">PostgreSQL 14 Internals</a></li> </ul> <h3>Articles</h3> <ul> <li><a href="https://dev.to/shayy/postgres-is-too-good-and-why-thats-actually-a-problem-4imc">"Postgres is Too Good (And Why That's Actually a Problem)"</a></li> <li><a href="https://pierce.dev/notes/go-ahead-self-host-postgres">Go ahead, self-host Postgres</a></li> <li><a href="https://www.interdb.jp/pg/">The Internals of PostgreSQL</a></li> </ul> <h2>Access Tools</h2> <h3>Language bindings</h3> <ul> <li>API: <a href="https://github.com/PostgREST/postgrest">PostgREST</a></li> <li>Common Lisp: <a href="https://github.com/marijnh/Postmodern">Postmodern</a></li> <li>Clojure: <a href="https://github.com/remodoy/clj-postgresql">clj-postgresql</a></li> <li>Elixir: <a href="https://github.com/elixir-ecto/postgrex">postgrex</a></li> <li>Go: <a href="https://github.com/lib/pq">pq</a>, <a href="https://github.com/jackc/pgx">pgx</a></li> <li>Haskell: <a href="http://hackage.haskell.org/package/postgresql-simple">postgresql-simple</a></li> <li>Java: <a href="https://jdbc.postgresql.org/">PostgreSQL JDBC Driver</a></li> <li>.NET/.NET Core: <a href="https://github.com/npgsql/npgsql">Npgsql</a></li> <li>Node: <a href="https://github.com/brianc/node-postgres">node-postgres</a>, <a href="https://github.com/vitaly-t/pg-promise">pg-promise</a>, <a href="https://github.com/holdfenytolvaj/pogi">pogi</a>, <a href="https://github.com/gajus/slonik">slonik</a>, <a href="https://github.com/porsager/postgres">postgres</a></li> <li>Perl: <a href="https://metacpan.org/pod/distribution/DBD-Pg/Pg.pm">DBD-Pg</a></li> <li>PHP: <a href="http://www.pomm-project.org">Pomm</a>, <a href="https://github.com/m6w6/ext-pq">pecl/pq</a></li> <li>Python: <a href="https://pypi.org/project/psycopg2/">psycopg2</a>, <a href="https://pypi.org/project/asyncpg/">asyncpg</a></li> <li>R: <a href="https://github.com/tomoakin/RPostgreSQL">RPostgreSQL</a></li> <li>Ruby: <a href="https://github.com/ged/ruby-pg">pg</a></li> <li>Rust: <a href="https://github.com/sfackler/rust-postgres">rust-postgresql</a>, <a href="https://github.com/tcdi/pgx">pgx</a></li> <li>Lua: <a href="https://github.com/arcapos/luapgsql">luapgsql</a></li> </ul> <h3>CLI</h3> <ul> <li><a href="https://github.com/dbcli/pgcli">pgcli</a> - Postgres CLI with autocompletion and syntax highlighting</li> <li><a href="https://github.com/sastraxi/pgsh">pgsh</a> - Branch your PostgreSQL Database like Git</li> <li><a href="https://www.postgresql.org/docs/current/static/app-psql.html">psql</a> - The built-in PostgreSQL CLI client</li> <li><a href="https://github.com/fphilipe/psql2csv">psql2csv</a> - Run a query in psql and output the result as CSV</li> <li><a href="https://gitlab.com/postgres-ai/nancy">nancy</a> - The Nancy CLI is a unified way to manage automated database experiments either in clouds or on-premise</li> <li><a href="https://github.com/schemaspy/schemaspy">schemaspy</a> - SchemaSpy is a JAVA JDBC-compliant tool for generating your database to HTML documentation, including Entity Relationship diagrams</li> </ul> <h3>GUI</h3> <ul> <li><a href="https://www.adminer.org/">Adminer</a> - Full-featured database management tool written in PHP.</li> <li><a href="https://www.beekeeperstudio.io">Beekeeper Studio</a> - Free and open source SQL client with a modern UI and great Postgres support. Cross platform.</li> <li><a href="https://chartbrew.com">Chartbrew</a> - Create live dashboards, charts, and client reports from PostgreSQL data. Features a query tool that works with SQL.</li> <li><a href="https://count.co/">Count</a> - Web-based analytics platform with a notebook interface which connects to PostgreSQL (Commercial Software).</li> <li><a href="https://www.jetbrains.com/datagrip/">DataGrip</a> - IDE with advanced tool sets and good cross-platform experience (Commercial Software).</li> <li><a href="https://datazenit.com/">Datazenit</a> - Web-based PostgreSQL GUI (Commercial Software).</li> <li><a href="https://www.datarow.com/">DataRow</a> - Cross-platform SQL Client for Amazon Redshift: Simple, Effortless, Extensible.</li> <li><a href="https://dbeaver.io/">DBeaver</a> - Universal Database Manager with excellent support for PostgreSQL.</li> <li><a href="http://dbglass.web-pal.com">dbglass</a> - Cross-platform desktop client for PostgreSQL, built with Electron.</li> <li><a href="https://www.holistics.io/">Holistics</a> - Online cross platform database management tool and SQL query reporting GUI with strong PostgreSQL support (Commercial Software).</li> <li><a href="https://www.jackdb.com/">JackDB</a> - Web-based SQL query interface (Commercial Software).</li> <li><a href="https://www.metabase.com/">Metabase</a> - Simple dashboards, charts and query tool for PostgreSQL.</li> <li><a href="https://numeracy.co/">Numeracy</a> - Fast SQL editor with charts and dashboards for PostgreSQL (Commercial Software).</li> <li><a href="https://omnidb.org/en/">OmniDB</a> - Open Source Collaborative Environment<br> For Database Management</li> <li><a href="https://www.pgadmin.org/">pgAdmin</a> - PostgreSQL Administration and Management GUI.</li> <li><a href="https://pgmodeler.io/">pgModeler</a> - pgModeler is an open-source PostgreSQL Database Modeler.</li> <li><a href="https://github.com/sosedoff/pgweb">pgweb</a> - Web-based PostgreSQL database browser written in Go.</li> <li><a href="https://github.com/phppgadmin/phppgadmin">phpPgAdmin</a> - The Premier Web Based Administration Tool for PostgreSQL.</li> <li><a href="https://github.com/Paxa/postbird">Postbird</a> - PostgreSQL Client for macOS.</li> <li><a href="https://www.postgrescompare.com">PostgresCompare</a> - Cross-platform database comparison and deployment tool (Commercial Software).</li> <li><a href="https://eggerapps.at/postico/">Postico</a> - Modern PostgreSQL Client for macOS (Commercial Software).</li> <li><a href="http://www.psequel.com/">PSequel</a> - Clean and simple interface to perform common PostgreSQL tasks quickly (Commercial Software).</li> <li><a href="http://www.sqltabs.com/">SQL Tabs</a> - Cross Platform Desktop Client for PostgreSQL written in JS.</li> <li><a href="http://macpostgresclient.com/">SQLPro for Postgres</a> - Simple, powerful PostgreSQL manager for macOS (Commercial Software).</li> <li><a href="https://github.com/dalibo/temboard">temBoard</a> - Web-based PostgreSQL GUI and monitoring.</li> <li><a href="https://tableplus.com/">TablePlus</a> - Native App which let you edit database and structure. High-end security ensured (Commercial Software).</li> <li><a href="https://www.valentina-db.com/en/valentina-studio-overview">Valentina Studio</a> - Cross-platform database administration tool (Free/Commercial)</li> <li><a href="https://dbgate.org">DbGate</a> - The Smartest (no)SQL Database Client</li> </ul> <h2>Extensions</h2> <ul> <li><a href="https://github.com/citusdata/citus">Citus</a> - Scalable PostgreSQL cluster for real-time workloads.</li> <li><a href="https://github.com/citusdata/cstore_fdw">cstore_fdw</a> - Columnar store for analytics with PostgreSQL.</li> <li><a href="https://pgxn.org/dist/cyanaudit/">cyanaudit</a> - Cyan Audit provides in-database logging of all DML activity on a column-by-column basis.</li> <li><a href="https://github.com/citusdata/pg_cron">pg_cron</a> - Run periodic jobs in PostgreSQL.</li> <li><a href="https://github.com/2ndQuadrant/pglogical">pglogical</a> - Extension that provides logical streaming replication.</li> <li><a href="https://github.com/pgpartman/pg_partman">pg_partman</a> - Partition management extension for PostgreSQL.</li> <li><a href="https://github.com/citusdata/pg_paxos/">pg_paxos</a> - Basic implementation of Paxos and Paxos-based table replication for a cluster of PostgreSQL nodes.</li> <li><a href="https://github.com/citusdata/pg_shard">pg_shard</a> - Extension to scale out real-time reads and writes.</li> <li><a href="https://wiki.postgresql.org/wiki/PGStrom">PGStrom</a> - Extension to offload CPU intensive workloads to GPU.</li> <li><a href="https://pgxn.org/">pgxn</a> PostgreSQL Extension Network - central distribution point for many open-source PostgreSQL extensions</li> <li><a href="https://www.confluent.io/blog/pipelinedb-team-joins-confluent/">PipelineDB</a> - A PostgreSQL extension that runs SQL queries continuously on streams, incrementally storing results in tables.</li> <li><a href="https://github.com/okbob/plpgsql_check">plpgsql_check</a> - Extension that allows to check plpgsql source code.</li> <li><a href="http://postgis.net/">PostGIS</a> - Spatial and Geographic objects for PostgreSQL.</li> <li><a href="https://github.com/cossacklabs/pg_themis">PG_Themis</a> - Postgres binding as extension for crypto library Themis, providing various security services on PgSQL's side.</li> <li><a href="https://github.com/zombodb/zombodb">zomboDB</a> - Extension that enables efficient full-text searching via the use of indexes backed by Elasticsearch.</li> <li><a href="https://github.com/pgMemento/pgMemento">pgMemento</a> - Provides an audit trail for your data inside a PostgreSQL database using triggers and server-side functions written in PL/pgSQL.</li> <li><a href="https://www.timescale.com/">TimescaleDB</a> - Open-source time-series database fully compatible with Postgres, distributed as extension</li> <li><a href="https://pgtap.org/">pgTAP</a> - Database testing framework for Postgres</li> <li><a href="https://github.com/HypoPG/hypopg">HypoPG</a> - HypoPG provides hypothetical/virtual indexes feature.</li> <li><a href="https://github.com/pgRouting/pgrouting">pgRouting</a> - pgRouting extends the PostGIS/PostgreSQL geospatial database to provide geospatial routing and other network analysis functionality.</li> <li><a href="https://pgroonga.github.io/">PGroonga</a> - PGroonga provides a new index access method that uses Groonga allowing super fast full text search feature against all languages.</li> <li><a href="https://www.pgaudit.org/">PGAudit</a> - The PostgreSQL Audit Extension (or pgaudit) provides detailed session and/or object audit logging via the standard logging facility provided by PostgreSQL.</li> </ul> <h3>Backups</h3> <ul> <li><a href="https://www.pgbarman.org/index.html">Barman</a> - Backup and Recovery Manager for PostgreSQL by 2ndQuadrant.</li> <li><a href="https://github.com/omniti-labs/omnipitr">OmniPITR</a> - Advanced WAL File Management Tools for PostgreSQL.</li> <li><a href="https://github.com/postgrespro/pg_probackup">pg_probackup</a> – A fork of pg_arman, improved by @PostgresPro, supports incremental backups, backups from replica, multithreaded backup and restore, and anonymous backup without archive command.</li> <li><a href="https://pgbackrest.org/">pgBackRest</a> - Reliable PostgreSQL Backup &amp; Restore.</li> <li><a href="https://github.com/orgrim/pg_back/">pg_back</a> - pg_back is a simple backup script</li> <li><a href="https://github.com/aiven/pghoard">pghoard</a> - Backup and restore tool for cloud object stores (AWS S3, Azure, Google Cloud, OpenStack Swift).</li> <li><a href="https://github.com/wal-e/wal-e">wal-e</a> (obsolete) - Simple Continuous Archiving for PostgreSQL to S3, Azure, or Swift by Heroku.</li> <li><a href="https://github.com/wal-g/wal-g">wal-g</a> - The successor of WAL-E rewritten in Go. Currently supports cloud object storage services by AWS (S3), Google Cloud (GCS), Azure, as well as OpenStack Swift, MinIO, and file system storages. Supports block-level incremental backups, offloading backup tasks to a standby server, provides parallelization and throttling options. In addition to Postgres, WAL-G can be used for MySQL and MongoDB databases.</li> <li><a href="https://dalibo.github.io/pitrery/">pitrery</a> - pitrery is a set of Bash scripts to manage Point In Time Recovery (PITR) backups for PostgreSQL.</li> </ul> <h3>High-Availability</h3> <ul> <li><a href="https://github.com/2ndQuadrant/bdr">BDR</a> - BiDirectional Replication - a multimaster replication system for PostgreSQL</li> <li><a href="https://github.com/zalando/patroni">Patroni</a> - Template for PostgreSQL HA with ZooKeeper or etcd.</li> <li><a href="https://github.com/sorintlab/stolon">Stolon</a> - PostgreSQL HA based on Consul or etcd, with Kubernetes integration.</li> <li><a href="https://github.com/aiven/pglookout">pglookout</a> - Replication monitoring and failover daemon.</li> <li><a href="https://github.com/2ndQuadrant/repmgr">repmgr</a> - Open-source tool suite to manage replication and failover in a cluster of PostgreSQL servers.</li> <li><a href="https://slony.info/">Slony-I</a> - "Master to multiple slaves" replication system with cascading and failover.</li> <li><a href="https://github.com/ClusterLabs/PAF">PAF</a> - PostgreSQL Automatic Failover: High-Availibility for Postgres, based on Pacemaker and Corosync.</li> <li><a href="https://github.com/pgq/skytools-legacy">SkyTools</a> - Replication tools, including PgQ, a queuing system, and Londiste, a replication system a bit simpler to manage than Slony.</li> </ul> <h3>Monitoring</h3> <ul> <li><a href="https://github.com/OPMDG/check_pgactivity">check_pgactivity</a> - check_pgactivity is designed to monitor PostgreSQL clusters from Nagios. It offers many options to measure and monitor useful performance metrics.</li> <li><a href="https://github.com/bucardo/check_postgres">Check_postgres</a> - Nagios check_postgres plugin for checking status of PostgreSQL databases.</li> <li><a href="https://github.com/Instrumental/instrumentald">Instrumental</a> - Real-time performance monitoring, including <a href="https://instrumentalapp.com/docs/instrumentald/postgresql#suggested-graphs">pre-made graphs</a> for ease of setup (Commercial Software)</li> <li><a href="https://github.com/cavaliercoder/libzbxpgsql">libzbxpgsql</a> - Comprehensive PostgreSQL monitoring module for Zabbix.</li> <li><a href="https://github.com/percona/pmm">PMM</a> - Percona Monitoring and Management (PMM) is a Free and Open Source platform for monitoring and managing PostgreSQL, MySQL, and MongoDB.</li> <li><a href="https://github.com/rach/pome">Pome</a> - Pome stands for PostgreSQL Metrics. Pome is a PostgreSQL Metrics Dashboard to keep track of the health of your database.</li> <li><a href="https://pgmetrics.io/">pgmetrics</a> - pgmetrics is an open-source, zero-dependency, single-binary tool that can collect a lot of information and statistics from a running PostgreSQL server and display it in easy-to-read text format or export it as JSON and CSV for scripting.</li> <li><a href="https://github.com/zalando/pg_view">pg_view</a> - Open-source command-line tool that shows global system stats, per-partition information, memory stats and other information.</li> <li><a href="https://github.com/cybertec-postgresql/pgwatch2">pgwatch2</a> - Flexible and easy to get started PostgreSQL metrics monitor focusing on Grafana dashboards.</li> <li><a href="https://www.postgresql.org/docs/devel/static/pgbench.html">pgbench</a> - Run a benchmark test on PostgreSQL.</li> <li><a href="http://opm.io">opm.io</a> - Open PostgreSQL Monitoring is a free software suite designed to help you manage your PostgreSQL servers. It can gather stats, display dashboards and send warnings when something goes wrong.</li> <li><a href="https://okmeter.io/pg">okmeter.io</a> - Commercial SaaS agent-based monitoring with a very detailed PostgreSQL plugin. It automatically gathers 100s of stats, displays dashboards on every aspect and sends alerts when something goes wrong (Commercial Software).</li> <li><a href="https://github.com/ankane/dexter">dexter</a> - The automatic indexer for Postgres. Detects slow queries and creates indexes if configured to do so.</li> </ul> <h3>Optimization</h3> <ul> <li><a href="https://github.com/mgartner/pg_flame">pg_flame</a> - A flamegraph generator for query plans.</li> <li><a href="https://github.com/ankane/pghero">PgHero</a> - PostgreSQL insights made easy.</li> <li><a href="https://www.pgmustard.com/">pgMustard</a> - A modern user interface<br> for <code>EXPLAIN</code>, that also provides performance tips (Commercial Software).</li> <li><a href="https://github.com/gregs1104/pgtune/">pgtune</a> - PostgreSQL configuration wizard.</li> <li><a href="https://github.com/le0pard/pgtune">pgtune</a> - Online version of PostgreSQL configuration wizard.</li> <li><a href="https://github.com/sebastianwebber/pgconfig">pgconfig.org</a> - PostgreSQL Online Configuration Tool (also based on pgtune).</li> <li><a href="https://powa.readthedocs.io/en/latest/">PoWA</a> - PostgreSQL Workload Analyzer gathers performance stats and provides real-time charts and graphs to help monitor and tune your PostgreSQL servers.</li> <li><a href="https://github.com/kirs/pg_web_stats">pg_web_stats</a> - Web UI to view pg_stat_statements.</li> <li><a href="https://github.com/timescale/timescaledb-tune">TimescaleDB Tune</a> - a program for tuning a TimescaleDB database to perform its best based on the host's resources such as memory and number of CPUs.</li> </ul> <h2>Utilities</h2> <ul> <li><a href="https://www.apgdiff.com/">apgdiff</a> - Compares two database dump files and creates output with DDL statements that can be used to update old database schema to new one.</li> <li><a href="https://github.com/Alexis-benoist/eralchemy">ERAlchemy</a> - ERAlchemy generates Entity Relation (ER) diagram from databases.</li> <li><a href="https://github.com/dalibo/ldap2pg">ldap2pg</a> - Synchronize roles and privileges from YML and LDAP.</li> <li><a href="https://github.com/lanyrd/mysql-postgresql-converter">mysql-postgresql-converter</a> - Lanyrd's MySQL to PostgreSQL conversion script.</li> <li><a href="http://ora2pg.darold.net">ora2pg</a> - Perl module to export an Oracle database schema to a PostgreSQL compatible schema.</li> <li><a href="https://github.com/dalibo/pg_activity">pg_activity</a> - top like application for PostgreSQL server activity monitoring.</li> <li><a href="https://github.com/gajus/pg-formatter">pg-formatter</a> - A PostgreSQL SQL syntax beautifier (Node.js).</li> <li><a href="https://pganalyze.com">pganalyze</a> - PostgreSQL Performance Monitoring (Commercial Software).</li> <li><a href="https://github.com/darold/pgbadger">pgbadger</a> - Fast PostgreSQL Log Analyzer.</li> <li><a href="http://www.pgbouncer.org/">PgBouncer</a> - Lightweight connection pooler for PostgreSQL.</li> <li><a href="https://github.com/lesovsky/pgcenter">pgCenter</a> - Provides convenient interface to various statistics, management task, reloading services, viewing log files and canceling or terminating database backends.</li> <li><a href="https://github.com/the4thdoctor/pg_chameleon">pg_chameleon</a> - Real time replica from MySQL to PostgreSQL with optional type override migration and migration capabilities.</li> <li><a href="https://github.com/lukasmartinelli/pgclimb">pgclimb</a> - Export data from PostgreSQL into different data formats.</li> <li><a href="https://github.com/mchristofides/pg_docs_bot/">pg_docs_bot</a> - Browser extension to redirect PostgreSQL docs links to the current version.</li> <li><a href="https://github.com/lukasmartinelli/pgfutter">pgfutter</a> - Import CSV and JSON into PostgreSQL the easy way.</li> <li><a href="http://pginsight.io/">PGInsight</a> - CLI tool to easily dig deep inside your PostgreSQL database.</li> <li><a href="https://github.com/lob/pg_insights">pg_insights</a> - Convenient SQL for monitoring Postgres database health.</li> <li><a href="https://github.com/dimitri/pgloader">pgloader</a> - Loads data into PostgreSQL using the COPY streaming protocol, and does so with separate threads for reading and writing data.</li> <li><a href="https://github.com/CrunchyData/pgmonitor">pgMonitor</a> - Postgres metrics collection and visualization that can be deployed to bare metal, virtual machines, or Kubernetes.</li> <li><a href="https://www.pgpool.net/mediawiki/index.php/Main_Page">pgpool-II</a> - Middleware that provides connection pooling, replication, load balancing and limiting exceeding connections.</li> <li><a href="https://github.com/ankane/pgsync">pgsync</a> - Tool to sync PostgreSQL data to your local machine.</li> <li><a href="https://github.com/pgxn/pgxnclient">PGXN client</a> - Command line tool to interact with the PostgreSQL Extension Network</li> <li><a href="https://github.com/spotify/postgresql-metrics">postgresql-metrics</a> - Tool that extracts and provides metrics for your PostgreSQL database.</li> <li><a href="https://github.com/PostgREST/postgrest">PostgREST</a> - Serves a fully RESTful API from any existing PostgreSQL database. <a href="https://docs.postgrest.org/en/v13/index.html">Docs</a></li> <li><a href="https://github.com/prest/prest">pREST</a> - Serve a RESTful API from any PostgreSQL database (Golang)</li> <li><a href="https://github.com/graphile/postgraphile">PostGraphile</a> - Instant GraphQL API or GraphQL schema for your PostgreSQL database</li> <li><a href="https://github.com/nanopack/yoke">yoke</a> - PostgreSQL high-availability cluster with auto-failover and automated cluster recovery.</li> <li><a href="https://github.com/kabirbaidhya/pglistend">pglistend</a> - A lightweight PostgresSQL <code>LISTEN</code>/<code>NOTIFY</code> daemon built on top of <code>node-postgres</code>.</li> <li><a href="https://github.com/postgrespro/zson">ZSON</a> - PostgreSQL extension for transparent JSONB compression</li> <li><a href="http://ossc-db.github.io/pg_bulkload/index.html">pg_bulkload</a> - It's a high speed data loading utility for PostgreSQL.</li> <li><a href="https://github.com/jwdeitch/pg_migrate">pg_migrate</a> - Manage PostgreSQL codebases and make VCS simple.</li> <li><a href="https://github.com/sqitchers/sqitch">sqitch</a> - Tool for managing versioned schema deployment</li> <li><a href="https://github.com/yandex/pgmigrate">pgmigrate</a> - CLI tool to evolve schema migrations, developed by Yandex.</li> <li><a href="https://github.com/cbbrowne/pgcmp">pgcmp</a> - Tool to compare database schemas, with capability to accept some persistent differences</li> <li><a href="https://github.com/multum/pg-differ">pg-differ</a> - Tool for easy initialization / updating of the structure of PostgreSQL tables, migration alternative (Node.js).</li> <li><a href="https://github.com/jarulraj/sqlcheck">sqlcheck</a> - Automatically detects common SQL anti-patterns. Such anti-patterns often slow down queries. Addressing them will, therefore, help accelerate queries.</li> <li><a href="https://gitlab.com/postgres-ai/postgres-checkup">postgres-checkup</a> - a new-generation diagnostics tool that allows users to collect deep analysis of the health of a Postgres database.</li> <li><a href="https://scaffoldhub.io">ScaffoldHub.io</a> - Generate fullstack PostgreSQL apps with Angular, Vue or React (Commercial Software).</li> </ul> <h2>Resources</h2> <h3>Tutorials</h3> <ul> <li><a href="https://coderwall.com/p/cwe2_a/backup-and-recover-a-postgres-db-using-wal-e">Backup and recover a PostgreSQL DB using wal-e</a> - Tutorial about setting up continuous archiving in PostgreSQL using wal-e.</li> <li><a href="https://www.pgcasts.com">PG Casts</a> - Free weekly PostgreSQL screencasts by Hashrocket.</li> <li><a href="http://postgresguide.com/">Postgres Guide</a> - Guide designed as an aid for beginners and experienced users to find specific tips and explore tools available within PostgreSQL.</li> <li><a href="https://pgexercises.com/">PostgreSQL Exercises</a> - Site to make it easy to learn PostgreSQL by doing exercises.</li> <li><a href="http://www.tutorialspoint.com/postgresql/">tutorialspoint PostgreSQL tutorial</a> - Very extensive collection of tutorials on PostgreSQL</li> <li><a href="https://github.com/morenoh149/postgresDBSamples">postgresDBSamples</a> - A collection of sample postgres schemas</li> <li><a href="https://zaiste.net/posts/postgresql-primer-for-busy-people/">PostgreSQL Primer for Busy People</a> - A collection of the most common commands used in PostgreSQL</li> <li><a href="https://github.com/dataegret/pg-utils">pg-utils</a> - Useful DBA tools by Data Egret</li> </ul> <h3>Blogs</h3> <ul> <li><a href="https://planet.postgresql.org/">Planet PostgreSQL</a> - Blog aggregation service for PostgreSQL.</li> <li><a href="http://adpgtech.blogspot.com/search/label/PostgreSQL/">Andrew Dunstan's PostgreSQL and Technical blog</a></li> <li><a href="https://momjian.us/main/blogs/pgblog.html">Bruce Momjian's PostgreSQL blog</a></li> <li><a href="http://www.craigkerstiens.com/categories/postgres/">Craig Kerstiens PostgreSQL posts</a> - Set of posts on PostgreSQL cool features, tips and tricks.</li> <li><a href="http://www.databasesoup.com/search/label/postgresql/">Database Soup</a> - Josh Berkus' blog.</li> <li><a href="https://paquier.xyz/">Michael Paquier's blog</a></li> <li><a href="http://rhaas.blogspot.com/search/label/postgresql/">Robert Haas' blog</a></li> <li><a href="https://www.depesz.com/tag/postgresql/">select * from depesz;</a> - Hubert Lubaczewski's blog.</li> </ul> <h3>Books</h3> <ul> <li><a href="http://www.postgresql.org/docs/">Postgres Official Documentation</a></li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/postgres">Postgres Succinctly</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://goalkicker.com/PostgreSQLBook/">PostgreSQL Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://www.tutorialspoint.com/postgresql">PostgreSQL Tutorial</a> - Tutorials Point</li> <li><a href="http://www.faqs.org/docs/ppbook/book1.htm">Practical PostgreSQL</a></li> <li><a href="http://www.interdb.jp/pg">The Internals of PostgreSQL for database administrators and system developers</a></li> </ul> <h3>Articles</h3> <ul> <li>What PostgreSQL has over other open source SQL databases: <a href="https://www.compose.com/articles/what-postgresql-has-over-other-open-source-sql-databases/">Part I</a> | <a href="https://www.compose.com/articles/what-postgresql-has-over-other-open-source-sql-databases-part-ii/">Part II</a></li> <li><a href="https://di.nmfay.com/postgres-vs-mysql">the ultimate postgres vs mysql blog post</a></li> <li><a href="https://www.justwatch.com/blog/post/debugging-postgresql-performance-the-hard-way/">Debugging PostgreSQL performance, the hard way</a></li> <li><a href="http://www.craigkerstiens.com/2017/04/30/why-postgres-five-years-later/">Why use Postgres?</a></li> <li><a href="https://infinum.com/the-capsized-eight/superfast-csv-imports-using-postgresqls-copy">Superfast CSV imports using PostgreSQL's COPY command</a></li> <li><a href="https://spacelift.io/blog/tricking-postgres-into-using-query-plan">Tricking Postgres into using an insane – but 200x faster – query plan</a></li> <li><a href="https://heap.io/blog/optimizing-postgres-queries-at-scale">"Optimizing Postgres Queries at Scale"</a></li> <li><a href="https://blog.crunchydata.com/blog/five-tips-for-a-healthier-postgres-database-in-the-new-year">5 Tips for a Healthier Postgres Database</a></li> <li><a href="https://webapp.io/blog/postgres-is-the-answer/">Postgres is a great pub/sub &amp; job server</a></li> <li><a href="https://stepzen.com/blog/join-data-postgresql-declarative-graphql-without-sql">"Join Data from PostgreSQL Declaratively in GraphQL w/o writing SQL"</a></li> </ul> <h3>Newsletters</h3> <ul> <li><a href="https://postgresweekly.com/">Postgres Weekly</a> - Weekly newsletter that contains articles, news, and repos relevant to PostgreSQL.</li> </ul> <h3>WTF</h3> <ul> <li><a href="https://github.com/eggyknap/pllolcode">PostgresQL implementation of LOLCode</a>; <a href="https://www.pgcon.org/2009/schedule/track/Tutorial/159.en.html">PGCon slides (PDF)</a></li> </ul> https://tedneward.github.io/Research/storage/questdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/questdb/index.html <p><a href="https://questdb.io/">Website</a> | <a href="https://github.com/questdb/questdb">Source</a></p> <p><a href="https://bjug.jerrinot.info/#1">Slides explaining its engineering</a></p> <blockquote> <p>We achieve high performance via a column-oriented storage model, parallelized vector execution, SIMD instructions, and low-latency techniques. In addition, QuestDB is hardware efficient, with quick setup and operational efficiency.<br> QuestDB implements ANSI SQL with native time-series SQL extensions. These SQL extensions make it simple to analyze, filter and downsample data, or to correlate data from multiple sources using relational and time-series joins.</p> </blockquote> https://tedneward.github.io/Research/storage/redisgraph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/redisgraph/index.html <p>RedisGraph is the first queryable Property Graph database to use sparse matrices to represent the adjacency matrix in graphs and linear algebra to query the graph.</p> <p>Primary features:</p> <ul> <li>Based on the Property Graph Model</li> <li>Nodes (vertices) and Relationships (edges) that may have attributes</li> <li>Nodes that can be labeled</li> <li>Relationships have a relationship type</li> <li>Graphs represented as sparse adjacency matrices</li> <li><a href="/languages/cypher">Cypher</a> as query language</li> <li>Cypher queries translated into linear algebra expressions</li> </ul> <p><a href="https://oss.redislabs.com/redisgraph/">Website</a> | <a href="https://github.com/RedisGraph/RedisGraph/">Source</a></p> <p>To quickly try out RedisGraph, launch an instance using docker: <code>docker run -p 6379:6379 -it --rm redislabs/redisgraph</code></p> <p>After you load RedisGraph, you can interact with it using <code>redis-cli</code>.</p> <p>Here we'll quickly create a small graph representing a subset of motorcycle riders and teams taking part in the MotoGP league. Once created, we'll start querying our data.</p> <p>With redis-cli</p> <pre><code>$ redis-cli 127.0.0.1:6379&gt; GRAPH.QUERY MotoGP "CREATE (:Rider {name:'Valentino Rossi'})-[:rides]-&gt;(:Team {name:'Yamaha'}), (:Rider {name:'Dani Pedrosa'})-[:rides]-&gt;(:Team {name:'Honda'}), (:Rider {name:'Andrea Dovizioso'})-[:rides]-&gt;(:Team {name:'Ducati'})" 1) 1) Labels added: 2 2) Nodes created: 6 3) Properties set: 6 4) Relationships created: 3 5) "Query internal execution time: 0.399000 milliseconds" </code></pre> <p>Now that our MotoGP graph is created, we can start asking questions. For example: Who's riding for team Yamaha?</p> <pre><code>127.0.0.1:6379&gt; GRAPH.QUERY MotoGP "MATCH (r:Rider)-[:rides]-&gt;(t:Team) WHERE t.name = 'Yamaha' RETURN r.name, t.name" 1) 1) "r.name" 2) "t.name" 2) 1) 1) "Valentino Rossi" 2) "Yamaha" 3) 1) "Query internal execution time: 0.625399 milliseconds" </code></pre> <p>How many riders represent team Ducati?</p> <pre><code>127.0.0.1:6379&gt; GRAPH.QUERY MotoGP "MATCH (r:Rider)-[:rides]-&gt;(t:Team {name:'Ducati'}) RETURN count(r)" 1) 1) "count(r)" 2) 1) 1) (integer) 1 3) 1) "Query internal execution time: 0.624435 milliseconds" </code></pre> https://tedneward.github.io/Research/storage/riak/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/riak/index.html <p><a href="https://riak.com/index.html">Website</a> | <a href="https://github.com/basho/riak">Source</a> | <a href="https://docs.riak.com/">Docs</a></p> https://tedneward.github.io/Research/storage/octobase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/octobase/index.html <p><a href="https://octobase.dev/">Website</a> | <a href="https://github.com/toeverything/OctoBase">Source</a></p> <h2>Features</h2> <p>As an offline collaborative data database, OctoBase has the following characteristics:</p> <ul> <li>📚 <strong>Multi-platform available</strong> offline collaboration, Schemaless, structured/unstructured/rich text data storage.</li> <li>🗃️ <strong>Binary storage</strong> that supports data deduplication and rich media editing.</li> <li>🔍 <strong>High-performance real-time full-text indexing</strong> with high-quality multilingual word segmentation support.</li> <li>🌐 <strong>CRDT-driven P2P synchronization</strong> with rich multi-platform native support.</li> <li>🔒 <strong>Fine-grained permission control</strong> with advanced permission management.</li> </ul> <p>OctoBase provides native support for offline collaboration, full-text indexing, and binary storage, making it easy for developers to build secure and high-performance local-first collaborative applications that work seamlessly across multiple platforms.</p> <p>With OctoBase, you will have access to same data abstractions across platform that enable you to maintain consistency and coherence across all your applications, regardless of the devices or platforms used.</p> <p>Additionally, OctoBase can function as a standalone server database, or it can be integrated directly into your application as an embedded database while remaining fully functional.</p> https://tedneward.github.io/Research/storage/orly/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/orly/index.html <p><a href="https://github.com/orlyatomics/orly/wiki">Website</a> | <a href="https://github.com/orlyatomics/orly">Source</a></p> <h2>Orly ("oh-really") features:</h2> <ul> <li><strong>Points of View</strong>: This is our version of optimistic locking or isolation. In traditional databases, clients have to lock the entire database (or at least large swaths of it) before updating it to ensure data remains consistent. In Orly, clients make changes in their own private points of view, which are like small sandboxes. Changes in private points of view eventually propagate into shared points of view and eventually reach the global point of view, which is the whole database. Updates to private points of view don't lock anything: Orly determines later whether, when, and how to reconcile changes from different points of view. We also encourage field calls rather than field changes (e.g., <code>x += 1</code> is better than <code>x = x + 1</code>).</li> <li><strong>Time Travel</strong>: We use something called the <em>Flux Capacitor</em> to keep a history of changes made to the database and to resolve conflicts as they come into shared points of view and the global point of view. This permits us to perform consistent read for any point in time. Orly defines its "time line" by causality rather than clock time. Instead of manipulating timestamps, Orly records an ordering of events (e.g., update A affects update B, so A "happens before" B).)</li> <li><strong>Query Language</strong>: Orly has its own high-level, compiled, type-safe, functional language called <em>Orlyscript</em>. Orlyscript is not just a query language: You can write general-purpose programs in it complete with compile-time unit tests. Orly comes with a compiler that transforms Orlyscript into shared libraries (.so files on Linux), which Orly servers load as packages.</li> <li><strong>Scalability and Availability</strong>: While we eventually plan to develop a sharded Orly machine (and actively design so that we can build such a machine), our current single-node server with fail-over/replication can handle hundreds of thousands of transactions per second. We like to say that Orly will function on a "planetary scale": Your data and computations will not only distribute across a data center, but also across many data centers across the globe. This means that no disaster short of nuking the planet fifty times over or colliding with a gigantic asteroid will destroy your data. (Even those might not be catastrophic: Maybe we'll have data centers running Orly with your replicated data on the moon or Mars.)</li> </ul> https://tedneward.github.io/Research/storage/pingcap/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/pingcap/index.html <p><a href="https://en.pingcap.com/">Website</a> | <a href="https://github.com/pingcap/tidb">Source</a></p> https://tedneward.github.io/Research/storage/pouchdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/pouchdb/index.html <p><a href="http://pouchdb.com">Website</a> | <a href="https://github.com/pouchdb/pouchdb">Source</a></p> <p>PouchDB databases come in two flavors: local and remote.</p> <p>To create a local database, you simply call new PouchDB and give it a name:</p> <pre><code>var db = new PouchDB('kittens'); </code></pre> <p>To create a remote database, you call new PouchDB and give it a path to a database in CouchDB.</p> <pre><code>var db = new PouchDB('http://localhost:5984/kittens'); </code></pre> <p>You can see basic information about the database by using the info() method.</p> <pre><code>db.info().then(function (info) { console.log(info); }) </code></pre> <p>The local database should show something like:</p> <pre><code>{"doc_count":0,"update_seq":0,"db_name":"kittens"} </code></pre> <p>The remote database may have a bit more information:</p> <pre><code>{"db_name":"kittens","doc_count":0,"doc_del_count":0,"update_seq":0,"purge_seq":0,"compact_running":false,"disk_size":79,"data_size":0,"instance_start_time":"1410722558431975","disk_format_version":6,"committed_update_seq":0} </code></pre> <p>You can also enable debug logging by doing:</p> <pre><code>PouchDB.debug.enable('*'); </code></pre> <p>And then disable it by doing:</p> <pre><code>PouchDB.debug.disable(); </code></pre> <p>During development, it's often useful to destroy the local database, so you can see what your users will experience when they visit your site for the first time. A page refresh is not enough, because the data will still be there!</p> <p>In Chrome, you can use the Clear Cache extension, which will add a trashcan icon to your toolbar, which you can click to delete all local data (IndexedDB, WebSQL, LocalStorage, cookies, etc.).</p> <p>In Firefox, you can use the Clear Recent History+ add-on, so when you right-click a page you can quickly clear all data.</p> <p>In Safari, you can simply click Safari → Clear History and Website Data.</p> <p>When you create a local PouchDB database, it uses whatever underlying datastore is available - IndexedDB in most browsers, WebSQL in older browsers, and LevelDB in Node.js.</p> <p>When you create a remote PouchDB database, it communicates directly with the remote database – CouchDB, Cloudant, Couchbase, etc.</p> <p>The goal of PouchDB is to allow you to seamlessly communicate with one or the other. You should not notice many differences between the two, except that of course the local one is much faster!</p> <p>To store a document, you simply put it:</p> <pre><code>var doc = { "_id": "mittens", "name": "Mittens", "occupation": "kitten", "age": 3, "hobbies": [ "playing with balls of yarn", "chasing laser pointers", "lookin' hella cute" ] }; db.put(doc); </code></pre> <p>Whenever you put() a document, it must have an _id field so that you can retrieve it later.</p> <p>So now let's get() the document by using its _id:</p> <pre><code>db.get('mittens').then(function (doc) { console.log(doc); }); </code></pre> <p>You should see:</p> <pre><code>{ "name": "Mittens", "occupation": "kitten", "age": 3, "hobbies": [ "playing with balls of yarn", "chasing laser pointers", "lookin' hella cute" ], "_id": "mittens", "_rev": "1-bea5fa18e06522d12026f4aee6b15ee4" } </code></pre> <p>The document looks exactly the same as when we put it, except... aha! What is this? There is a new field, _rev, the revision marker. It is a randomly-generated ID that changes whenever a document is created or updated.</p> <p>For instance, to increment Mittens' age to 4, we would do:</p> <pre><code>doc.age = 4; doc._rev = "1-bea5fa18e06522d12026f4aee6b15ee4"; db.put(doc); </code></pre> <p>If you fail to include the correct _rev, you will get the following sad error:</p> <pre><code>{ "status": 409, "name": "conflict", "message": "Document update conflict" } </code></pre> <p>HTTP 409 is a standard HTTP error message that indicates a conflict.</p> <p>So to update Mittens' age, we will first need to fetch Mittens from the database, to ensure that we have the correct _rev before we put them back. We don't need to manually assign the _rev value here (like we did above), as it is already in the doc we're fetching.</p> <pre><code>// fetch mittens db.get('mittens').then(function (doc) { // update their age doc.age = 4; // put them back return db.put(doc); }).then(function () { // fetch mittens again return db.get('mittens'); }).then(function (doc) { console.log(doc); }); </code></pre> <p>Now you should see the following:</p> <pre><code>{ "name": "Mittens", "occupation": "kitten", "age": 4, "hobbies": [ "playing with balls of yarn", "chasing laser pointers", "lookin' hella cute" ], "_id": "mittens", "_rev": "2-3e3fd988b331193beeeea2d4221b57e7" } </code></pre> <p>As you can see, we have successfully updated Mittens' age to 4 (they grow up so fast!), and their revision marker has also changed to "2-3e3fd988b331193beeeea2d4221b57e7". If we wanted to increment their age to 5, we would need to supply this new revision marker.</p> <p>PouchDB provides a fully asynchronous API. This ensures that when you talk to PouchDB, the UI doesn't stutter, because the DOM is not being blocked by database operations. To make things as flexible as possible for PouchDB users, the API is provided in both callback format and promise format.</p> <p>The callback format looks like this:</p> <pre><code>db.get('mittens', function (error, doc) { if (error) { // oh noes! we got an error } else { // okay, doc contains our document } }); </code></pre> <p>The promise format looks like this:</p> <pre><code>db.get('mittens').then(function (doc) { // okay, doc contains our document }).catch(function (err) { // oh noes! we got an error }); </code></pre> <p>Basically, if you include a callback as the last argument in a function, then PouchDB assumes you want the callback style. Otherwise it assumes you want the promise style.</p> <p>The big advantage of working with Promises in asynchronous code is that you can always attach a catch function to the end of a big promise chain, and any errors that occur along the way will show up at the end.</p> <p>This avoids endless if (err) {} checking in the callback world:</p> <pre><code>doSomething(function (err, result) { if (err) { // handle error } doSomethingElse(function (err, result) { if (err) { // handle error again... } doSomethingYetAgain(function (err, result) { if (err) { // seriously? okay, handle error again... } }); }); }); </code></pre> <p>Instead, in the promise world, you can have a long chain of asynchronous operations with a single catch at the end. To use PouchDB as an example:</p> <pre><code>db.put({_id: 'charlie', age: 21}).then(function () { return db.get('charlie'); }).then(function (charlie) { // increment Charlie's age charlie.age++; return db.put(charlie); }).then(function () { return db.get('charlie'); }).then(function (charlie) { // increment Charlie's age again charlie.age++; return db.put(charlie); }).then(function () { return db.get('charlie'); }).then(function (charlie) { console.log(charlie); }).catch(function (err) { console.log(err); }); </code></pre> <p>You should see:</p> <pre><code>{"age":23,"_id":"charlie","_rev":"3-e794618b4e39ed566cc68b56f5426e8e"} </code></pre> <p>You can see a live example of this code.</p> <p>In this example, we put/get a document 3 times in a row. At the very end, there is a catch() statement to catch any errors along the way.</p> <p>What kind of errors might we run into? Well, let's imagine that we accidentally misspell the id 'charlie' at some point. In this case, we will gracefully catch the error. Here's another live example.</p> <p>You should see:</p> <pre><code>{"status":404,"name":"not_found","message":"missing"} </code></pre> <p>This is really nice! No matter where the misspelling is, the error can be handled within a single function. That's much nicer than having to do if (err){} an endless number of times!</p> <p>If you've been doing promises for awhile, you might have seen this instead:</p> <pre><code>db.get('charlie').then(function (charlie) { // we got the charlie doc }, function (err) { // we got an error }) </code></pre> <p>This is equivalent to:</p> <pre><code>db.get('charlie').then(function (charlie) { // we got the charlie doc }).catch(function (err) { // we got an error }) </code></pre> <p>The catch() method is just syntactic sugar. You can use either format.</p> <p>Often in our code, we'll want to get() a document, and if it doesn't exist, we want to create some default. For instance, let's say we have a configuration object. We want to provide some reasonable defaults for our config:</p> <pre><code>{ _id: 'config', background: 'blue', foreground: 'white', sparkly: 'false' } </code></pre> <p>This is a pretty good default setting! So let's write the code to set it as our default.</p> <p>Thankfully, promises make this rather easy:</p> <pre><code>db.get('config').catch(function (err) { if (err.name === 'not_found') { return { _id: 'config', background: 'blue', foreground: 'white', sparkly: 'false' }; } else { // hm, some other error throw err; } }).then(function (configDoc) { // sweet, here is our configDoc }).catch(function (err) { // handle any errors }); </code></pre> <p>This code is doing the following:</p> <ul> <li>Try to get() a doc with _id equal to 'config'</li> <li>If it doesn't find it, return the default doc</li> <li>Otherwise, you'll just get back the existing document</li> </ul> <p>A common question from new PouchDB/CouchDB users is: why do we have to deal with _rev at all? Why can't I just put() the document without providing a _rev?</p> <p>The answer is: because _revs are what makes sync work so well. PouchDB asks for a little upfront effort with managing document revisions, so that later on, sync is a breeze.</p> <p>In fact, you are probably already familiar with a system that forces you to go through a similar dance. This system is called Git.</p> <p>PouchDB and CouchDB's document revision structure is very similar to Git's. In fact, each document's revision history is stored as a tree (exactly like Git), which allows you to handle conflicts when any two databases get out of sync.</p> <pre><code>rev 3-a rev 3-b \___/ | rev 2 | rev 1 </code></pre> <p>Conflicts will be discussed later in this guide. For now, you can think of revisions as being a single lineage:</p> <pre><code> rev 4 | rev 3 | rev 2 | rev 1 </code></pre> <p>When you remove() a document, it's not really deleted; it just gets a _deleted attribute added to it.</p> <p>That is, the database saves a tombstone at the end of the revision tree.</p> <pre><code>{_id: 'foo', _rev: '4-z', _deleted: true} | {_id: 'foo', _rev: '3-y'} | {_id: 'foo', _rev: '2-x'} | {_id: 'foo', _rev: '1-w'} </code></pre> <p>There are three ways of deleting a document, which are all equivalent:</p> <p>1) You can call db.remove(doc):</p> <pre><code>db.get('mydoc').then(function (doc) { return db.remove(doc); }); </code></pre> <p>2) You can call db.remove(doc._id, doc._rev):</p> <pre><code>db.get('mydoc').then(function (doc) { return db.remove(doc._id, doc._rev); }); </code></pre> <p>3) You can call db.put(doc) with _deleted set to true:</p> <pre><code>db.get('mydoc').then(function (doc) { doc._deleted = true; return db.put(doc); }); </code></pre> <p>Of course, you will want to add catch() to the end of all these, unless you like to live dangerously.</p> <p>You can get(), put(), and remove() single documents to your heart's content, but a database isn't a database unless it can handle many operations at once!</p> <p>PouchDB provides two methods for bulk operations - bulkDocs() for bulk writes, and allDocs() for bulk reads.</p> <p>Attachments are where PouchDB can get really fun. The big difference between storage engines like WebSQL/IndexedDB and the older localStorage API is that you can stuff a lot more data in it. PouchDB attachments allow you to use that to full advantage to store images, MP3s, zip files, or whatever you want. As their name implies, attachments are attached to documents. You can work with attachments either in base64-encoded format, or as a Blob.</p> <p>For example, here is a very simple document with a plain text attachment, stored as base64.</p> <pre><code>db.put({ _id: 'mydoc', _attachments: { 'myattachment.txt': { content_type: 'text/plain', data: 'aGVsbG8gd29ybGQ=' } } }); </code></pre> <p>Our document has the usual _id field, but it also has a special _attachments field that holds the attachments. Documents can have as many attachments as you want.</p> <p>As it turns out, 'aGVsbG8gd29ybGQ=' is just the string 'hello world' encoded in base64. You can use the atob() and btoa() methods in your browser to verify.</p> <p>btoa('hello world') // "aGVsbG8gd29ybGQ="<br> atob('aGVsbG8gd29ybGQ=') // "hello world"<br> Let's see what happens after we store this document. If you try to get() it normally, you may be surprised to see that the attachment data itself isn't returned:</p> <p>db.get('mydoc').then(function (doc) {<br> console.log(doc);<br> });<br> The returned document will look like this:</p> <p>{<br> "_attachments": {<br> "myattachment.txt": {<br> "content_type": "text/plain",<br> "digest": "md5-XrY7u+Ae7tCTyyK7j1rNww==",<br> "stub": true<br> }<br> },<br> "_id": "mydoc",<br> "_rev": "1-e8a84187bb4e671f27ec11bdf7320aaa"<br> }<br> You can see a live example of this code.</p> <p>By default, PouchDB will only give you an attachment stub, which contains a digest, i.e. the md5sum of the binary attachment.</p> <p>To get the full attachments when using get() or allDocs(), you need to specify {attachments: true}:</p> <p>db.get('mydoc', {attachments: true}).then(function (doc) {<br> console.log(doc);<br> });<br> Then you'll get back the full attachment, base64-encoded:</p> <p>{<br> "_attachments": {<br> "myattachment.txt": {<br> "content_type": "text/plain",<br> "digest": "md5-XrY7u+Ae7tCTyyK7j1rNww==",<br> "data": "aGVsbG8gd29ybGQ="<br> }<br> },<br> "_id": "mydoc",<br> "_rev": "1-e8a84187bb4e671f27ec11bdf7320aaa"<br> }<br> You can see a live example of this code.</p> <p>Plaintext is cool and all, but you know what would be really awesome? Storing images.</p> <p>So let's do it! In this example, we'll put a document with a small icon attachment, represented as a base64-encoded string. Then we'll fetch it and display the icon as a normal <img src="https://tedneward.github.io/Research/storage/"> tag:</p> <pre><code>db.put({ _id: 'meowth', _attachments: { 'meowth.png': { content_type: 'image/png', data: 'iVBORw0KGgoAAAANSUhEUgAAACgAAAAkCAIAAAB0Xu9BAAAABGdBTUEAALGPC/xhBQAAAuNJREFUWEetmD1WHDEQhDdxRMYlnBFyBIccgdQhKVcgJeQMpE5JSTd2uqnvIGpVUqmm9TPrffD0eLMzUn+qVnXPwiFd/PP6eLh47v7EaazbmxsOxjhTT88z9hV7GoNF1cUCvN7TTPv/gf/+uQPm862MWTL6fff4HfDx4S79/oVAlAUwqOmYR0rnazuFnhfOy/ErMKkcBFOr1vOjUi2MFn4nuMil6OPh5eGANLhW3y6u3aH7ijEDCxgCvzFmimvc95TekZLyMSeJC68Bkw0kqUy1K87FlpGZqsGFCyqEtQNDdFUtFctTiuhnPKNysid/WFEFLE2O102XJdEE+8IgeuGsjeJyGHm/xHvQ3JtKVsGGp85g9rK6xMHtvHO9+WACYjk5vkVM6XQ6OZubCJvTfPicYPeHO2AKFl5NuF5UK1VDUbeLxh2BcRGKTQE3irHm3+vPj6cfCod50Eqv5QxtwBQUGhZhbrGVuRia1B4MNp6edwBxld2sl1splfHCwfsvCZfrCQyWmX10djjOlWJSSy3VQlS6LmfrgNvaieRWx1LZ6s9co+P0DLsy3OdLU3lWRclQsVcHJBcUQ0k9/WVVrmpRzYQzpgAdQcAXxZzUnFX3proannrYH+Vq6KkLi+UkarH09mC8YPr2RMWOlEqFkQClsykGEv7CqCUbXcG8+SaGvJ4a8d4y6epND+pEhxoN0vWUu5ntXlFb5/JT7JfJJqoTdy9u9qc7ax3xJRHqJLADWEl23cFWl4K9fvoaCJ2BHpmJ3s3z+O0U/DmzdMjB9alWZtg4e3yxzPa7lUR7nkvxLHO9+tvJX3mtSDpwX8GajB283I8R8a7D2MhUZr1iNWdny256yYLd52DwRYBtRMvE7rsmtxIUE+zLKQCDO4jlxB6CZ8M17GhuY+XTE8vNhQiIiSE82ZsGwk1pht4ZSpT0YVpon6EvevOXXH8JxVR78QzNuamupW/7UB7wO/+7sG5V4ekXb4cL5Lyv+4IAAAAASUVORK5CYII=' } } }).then(function () { return db.getAttachment('meowth', 'meowth.png'); }).then(function (blob) { var url = URL.createObjectURL(blob); var img = document.createElement('img'); img.src = url; document.body.appendChild(img); }).catch(function (err) { console.log(err); }); </code></pre> <p>You should be unsurprised to see a cat smiling back at you. If the kitten theme bothers you, then you haven't been on the Internet very long.</p> <p>How does this code work? First off, we are making use of the URL.createObjectURL() method, which is a standard HTML5 method that converts a Blob to a URL that we can easily use as the src of an img.</p> <p>Second off, we are using the getAttachment() API, which returns a Blob rather than a base64-encoded string. To be clear: we can always convert between base64 and Blobs, but in this case, getAttachment() is just more convenient.</p> <p>Up to now, we've been supplying our attachments as base64-encoded strings. But we can also create the Blobs ourselves and store those directly in PouchDB.</p> <p>Another shortcut we can use is the putAttachment() API, which simply modifies the existing document to hold a new attachment. Or, if the document does not exist, it will create an empty one.</p> <p>In Node.js, PouchDB uses Buffers instead of Blobs. Otherwise, the same rules apply.<br> For instance, we can read the image data from an <img src="https://tedneward.github.io/Research/storage/"> tag using a canvas element, and then directly write that Blob to PouchDB:</p> <pre><code>function convertImgToBlob(img, callback) { var canvas = document.createElement('canvas'); var context = canvas.getContext('2d'); context.drawImage(img, 0, 0); // Warning: toBlob() isn't supported by every browser. // You may want to use blob-util. canvas.toBlob(callback, 'image/png'); } var catImage = document.getElementById('cat'); convertImgToBlob(catImage, function (blob) { db.putAttachment('meowth', 'meowth.png', blob, 'image/png').then(function () { return db.get('meowth', {attachments: true}); }).then(function (doc) { console.log(doc); }); }); </code></pre> <p>This stores exactly the same image content as in the other example, which you can confirm by checking the base64-encoded output.</p> <p>Blobs can be tricky to work with, especially when it comes to cross-browser support. You may find blob-util to be a useful addition to the attachment API. For instance, it has an imgSrcToBlob() method that will work cross-browser.</p> <p>You can also upload a file with the HTML5 File API and store it directly in the database, because the data you get from the <input type="file"> element is already a Blob. (See: Blob API and File API, which inherits properties from the Blob Interface.)</p> <p>Here is an example of allowing a user to choose a file from their filesystem:</p> <pre><code>&lt;input type="file"&gt; </code></pre> <p>And then "uploading" that file directly into PouchDB:</p> <pre><code>var input = document.querySelector('input'); input.addEventListener('change', function () { var file = input.files[0]; // file is a Blob db.put({ _id: 'mydoc', _attachments: { filename: { type: file.type, data: file } } }).catch(function (err) { console.log(err); }); }); </code></pre> <p>Select a file and you will see the stored file, size, and type, which are valid Blob properties. If you choose an image, it will also show the image!</p> <p>Whether you supply attachments as base64-encoded strings or as Blobs/Buffers, PouchDB will try to store them in the most efficient way.</p> <p>So when you insert your attachments, either format is acceptable. For instance, you can put Blobs/Buffers using put():</p> <pre><code>db.put({ _id: 'mydoc', _attachments: { 'myattachment.txt': { content_type: 'text/plain', data: myBlob } } }); </code></pre> <p>And you can also pass base64-encoded strings to putAttachment():</p> <pre><code>db.putAttachment('mydoc', 'myattachment.png', myBase64String, 'image/png'); </code></pre> <p>You can also insert multiple attachments at once using put():</p> <pre><code>db.put({ _id: 'mydoc', _attachments: { 'myattachment1.txt': { content_type: 'text/plain', data: myBlob1 }, 'myattachment2.txt': { content_type: 'text/plain', data: myBlob2 }, 'myattachment3.txt': { content_type: 'text/plain', data: myBlob3 }, // etc. } }); </code></pre> <p>The bulkDocs() and post() APIs also accept attachments in either format.</p> <p>When you fetch attachments, however, getAttachment() will always return Blobs/Buffers.</p> <p>The other "read" APIs, such as get(), allDocs(), changes(), and query() have an {attachments: true} option that returns the attachments base64-encoded strings. If you add {binary: true}, though, they will return Blobs/Buffers.</p> <p>Performance tip: If you can insert and retrieve your attachments using only Blobs/Buffers, then you will typically get better performance, especially when it comes to memory usage. The base64 string format is mostly provided for developer convenience and debugging.</p> <p>Blobs have their own type, but there is also a content_type that you specify when you store it in PouchDB:</p> <pre><code>var myBlob = new Blob(['I am plain text!'], {type: 'text/plain'}); console.log(myBlob.type); // 'text/plain' db.put({ _id: 'mydoc', _attachments: { 'myattachment.txt': { content_type: 'text/plain', data: myBlob } } }); </code></pre> <p>The reason for this redundancy is 1) Buffers in Node do not have a type, and 2) the CouchDB attachment format requires it.</p> <p>So for best results, you should ensure that your Blobs have the same type as the one reported to PouchDB. Otherwise you may see inconsistent behavior (e.g. in IndexedDB, where the Blob is stored as-is on compatible browsers).</p> <p>PouchDB and CouchDB were designed for one main purpose: sync. Jason Smith has a great quote about this:</p> <blockquote> <p>The way I like to think about CouchDB is this: CouchDB is bad at everything, except syncing. And it turns out that's the most important feature you could ever ask for, for many types of software."</p> </blockquote> <p>When you first start using CouchDB, you may become frustrated because it doesn't work quite like other databases. Unlike most databases, CouchDB requires you to manage revisions (_rev), which can be tedious.</p> <p>However, CouchDB was designed with sync in mind, and this is exactly what it excels at. Many of the rough edges of the API serve this larger purpose. For instance, managing your document revisions pays off in the future, when you eventually need to start dealing with conflicts.</p> <p>CouchDB sync has a unique design. Rather than relying on a master/follower architecture, CouchDB supports a multi-master architecture. You can think of this as a system where any node can be written to or read from, and where you don't have to care which one is the "master" and which one is the "follower." In CouchDB's egalitarian world, every citizen is as worthy as another.</p> <p>When you use PouchDB, CouchDB, and other members of the Couch family, you don't have to worry which database is the "single source of truth." They all are. According to the CAP theorem, a database can only have at most 2 of 3 properties: Consistency, Availability, or Partition-Tolerance. Typical relational databases such as MySQL are CP, which means they are consistent and tolerant to node partitions, at the expense of availability. CouchDB is an AP database, meaning that it's Partition-Tolerant, every node is Available at all times, but it's only eventually Consistent.</p> <p>To illustrate, imagine a multi-node architecture with CouchDB servers spread across several continents. As long as you're willing to wait, the data will eventually flow from Australia to Europe to North America to wherever. Users around the world running PouchDB in their browsers or Couchbase Lite/Cloudant Sync in their smartphones experience the same privileges. The data won't show up instantaneously, but depending on the Internet connection speed, it's usually close enough to real-time.</p> <p>In cases of conflict, CouchDB will choose an arbitrary winner that every node can agree upon deterministically. However, conflicts are still stored in the revision tree (similar to a Git history tree), which means that app developers can either surface the conflicts to the user, or just ignore them.</p> <p>In this way, CouchDB replication "just works."</p> <p>As you already know, you can create either local PouchDBs:</p> <pre><code>var localDB = new PouchDB('mylocaldb') </code></pre> <p>or remote PouchDBs:</p> <pre><code>var remoteDB = new PouchDB('http://localhost:5984/myremotedb') </code></pre> <p>This pattern comes in handy when you want to share data between the two.</p> <p>The simplest case is unidirectional replication, meaning you just want one database to mirror its changes to a second one. Writes to the second database, however, will not propagate back to the master database.</p> <p>To perform unidirectional replication, you simply do:</p> <pre><code>localDB.replicate.to(remoteDB).on('complete', function () { // yay, we're done! }).on('error', function (err) { // boo, something went wrong! }); </code></pre> <p>Congratulations, all changes from the localDB have been replicated to the remoteDB.</p> <p>However, what if you want bidirectional replication? You could do:</p> <pre><code>localDB.replicate.to(remoteDB); localDB.replicate.from(remoteDB); </code></pre> <p>However, to make things easier for your poor tired fingers, PouchDB has a shortcut API:</p> <pre><code>localDB.sync(remoteDB); </code></pre> <p>These two code blocks above are equivalent. And the sync API supports all the same events as the replicate API:</p> <pre><code>localDB.sync(remoteDB).on('complete', function () { // yay, we're in sync! }).on('error', function (err) { // boo, we hit an error! }); </code></pre> <p>Live replication (or "continuous" replication) is a separate mode where changes are propagated between the two databases as the changes occur. In other words, normal replication happens once, whereas live replication happens in real time.</p> <p>To enable live replication, you simply specify {live: true}:</p> <pre><code>localDB.sync(remoteDB, { live: true }).on('change', function (change) { // yo, something changed! }).on('error', function (err) { // yo, we got an error! (maybe the user went offline?) }); </code></pre> <p>However, there is one gotcha with live replication: what if the user goes offline? In those cases, an error will be thrown and replication will stop.</p> <p>You can allow PouchDB to automatically handle this error, and retry until the connection is re-established, by using the retry option:</p> <pre><code>localDB.sync(remoteDB, { live: true, retry: true }).on('change', function (change) { // yo, something changed! }).on('paused', function (info) { // replication was paused, usually because of a lost connection }).on('active', function (info) { // replication was resumed }).on('error', function (err) { // totally unhandled error (shouldn't happen) }); </code></pre> <p>This is ideal for scenarios where the user may be flitting in and out of connectivity, such as on mobile devices.</p> <p>Sometimes, you may want to manually cancel replication – for instance, because the user logged out. You can do so by calling cancel() and then waiting for the 'complete' event:</p> <pre><code>var syncHandler = localDB.sync(remoteDB, { live: true, retry: true }); syncHandler.on('complete', function (info) { // replication was canceled! }); syncHandler.cancel(); // &lt;-- this cancels it </code></pre> <p>The replicate API also supports canceling:</p> <pre><code>var replicationHandler = localDB.replicate.to(remoteDB, { live: true, retry: true }); replicationHandler.on('complete', function (info) { // replication was canceled! }); replicationHandler.cancel(); // &lt;-- this cancels it </code></pre> <p>One thing to note about replication is that it tracks the data within a database, not the database itself. If you destroy() a database that is being replicated to, the next time the replication starts it will transfer all of the data again, recreating the database to the state it was before it was destroyed. If you want the data within the database to be deleted you will need to delete via remove() or bulkDocs(). The pouchdb-erase plugin can help you remove the entire contents of a database.</p> <p>Any PouchDB object can replicate to any other PouchDB object. So for instance, you can replicate two remote databases, or two local databases. You can also replicate from multiple databases into a single one, or from a single database into many others.</p> <p>This can be very powerful, because it enables lots of fancy scenarios. For example:</p> <p>You have an in-memory PouchDB that replicates with a local PouchDB, acting as a cache.</p> <p>You have many remote CouchDB databases that the user may access, and they are all replicated to the same local PouchDB.</p> <p>You have many local PouchDB databases, which are mirrored to a single remote CouchDB as a backup store.</p> <p>The only limits are your imagination and your disk space.</p> https://tedneward.github.io/Research/storage/ravendb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/ravendb/index.html <p><a href="https://ravendb.net/">Website</a></p> https://tedneward.github.io/Research/storage/rel/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/rel/index.html <p><a href="https://reldb.org/c/">Website</a> | <a href="https://github.com/DaveVoorhis/Rel">Source</a></p> https://tedneward.github.io/Research/storage/rocksdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/rocksdb/index.html <p><a href="https://rocksdb.org/">Website</a> | <a href="https://github.com/facebook/rocksdb/">Source</a></p> https://tedneward.github.io/Research/storage/opemtsdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/opemtsdb/index.html <p><a href="https://opentsdb.net/">Website</a> | <a href="https://github.com/OpenTSDB/opentsdb">Source</a> | <a href="https://opentsdb.net/docs/build/html/index.html#opentsdb-3-0">Docs</a></p> <p>OpenTSDB was written to address a common need: store, index and serve metrics collected from computer systems (network gear, operating systems, applications) at a large scale, and make this data easily accessible and graphable.</p> <p>Thanks to HBase's scalability, OpenTSDB allows you to collect thousands of metrics from tens of thousands of hosts and applications, at a high rate (every few seconds). OpenTSDB will never delete or downsample data and can easily store hundreds of billions of data points.</p> https://tedneward.github.io/Research/storage/outerbase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/outerbase/index.html <p><a href="https://www.outerbase.com/">Website</a></p> <h3>Articles</h3> <ul> <li><a href="https://www.outerbase.com/blog/outerbase-studio-open-source-database-management/">"Outerbase Studio"</a> (<a href="https://github.com/outerbase/studio">Source</a>)</li> </ul> https://tedneward.github.io/Research/storage/planet-scale/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/planet-scale/index.html <p><a href="https://planetscale.com/">Website</a></p> <p>Built on <a href="./vitess.md">Vitess</a></p> <p>"A branching database allows you to create different branches from your main database that you can use for different purposes. You can imagine this workflow as something very similar to a version control system like Git where you make changes to a project without breaking the entire codebase.</p> <p>"You can create a copy of your main or production database into a separate database branch. Then, you can experiment in that database branch, make changes and when you’re completely satisfied with them, merge them back to your main database. Thus it’s highly efficient and intuitive to play around and experiment with your database while keeping your production database intact."</p> https://tedneward.github.io/Research/storage/prestodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/prestodb/index.html <p><a href="https://prestodb.io">Website</a></p> <p>See also: <a href="couchbase-lite.html">couchbase-lite</a></p> https://tedneward.github.io/Research/storage/realm/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/realm/index.html <p><a href="https://realm.io/products/realm-database">Realm Database</a> | <a href="https://realm.io/products/realm-platform">Realm Platform</a> | <a href="https://realm.io/products/realm-studio">Realm Studio</a></p> <p>Realm Object Server and Realm Database combine to make the Realm Platform.</p> https://tedneward.github.io/Research/storage/restdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/restdb/index.html <p>"restdb.io is probably the easiest online NoSQL database backend for web and serverless "low code" applications. Model your information quickly. The data management application, schema and REST API are instantly available using our powerful API Automation technology.</p> <p>"In the backend, users collaborate and work with data in multiple ways. A centralized search let's users quickly search through the entire database quickly (if given access). It's also a breeze to navigate the database relationships.</p> <p>"restdb.io NoSQL databases can also serve web pages. Use HTML, Javascript, jQuery, Angular, ReactJS (or your favorite framework) on restdb.io Pages with server-side data-binding. Add your own secure domains for the web pages and the REST API."</p> <p><a href="https://restdb.io/">Website</a></p> https://tedneward.github.io/Research/storage/rxdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/rxdb/index.html <p><a href="https://rxdb.info/">Website</a> | <a href="https://github.com/pubkey/rxdb">Source</a></p> <p>RxDB is based on a storage interface that enables you to swap out the underlying storage engine. This increases code reuse because the same database code can be used in different JavaScript environments by just switching out the storage settings. You can use RxDB on top of IndexedDB, OPFS, LokiJS, Dexie.js, in-memory, SQLite, in a WebWorker thread and even on top of FoundationDB.</p> https://tedneward.github.io/Research/storage/orientdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/orientdb/index.html <p><a href="https://orientdb.org/">Community website</a> | <a href="https://github.com/orientechnologies/orientdb">Source</a></p> https://tedneward.github.io/Research/storage/owncloud/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/owncloud/index.html <p><a href="https://owncloud.com/">Website</a> | <a href="https://github.com/owncloud/core">Source</a></p> https://tedneward.github.io/Research/storage/planetscale/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/planetscale/index.html <p><a href="https://planetscale.com/">Website</a> | Commercial</p> <p>What’s interesting about PlanetScale is that it’s a branching database, a novel workflow that has emerged only recently but appears to be quite promising. But what is a branching database? A branching database allows you to create different branches from your main database that you can use for different purposes. You can imagine this workflow as something very similar to a version control system like Git where you make changes to a project without breaking the entire codebase.</p> <p>You can create a copy of your main or production database into a separate database branch. Then, you can experiment in that database branch, make changes and when you’re completely satisfied with them, merge them back to your main database. Thus it’s highly efficient and intuitive to play around and experiment with your database while keeping your production database intact.</p> <p>PlanetScale also adds branches to your database so you can test schema changes on an isolated development branch. This branch is separate from the production branch or your main database. This provides an isolated working environment without explicitly creating a staging or testing environment for experimenting with schema changes. You can read more on what branching means and how PlanetScale uses this concept in their documentation page.</p> <p>PlanetScale automatically creates daily backups and protects direct schema changes on your production branch right out of the box, reducing the possibility of accidentally corrupting the production database.</p> <p>Non-blocking schema changes are yet another interesting feature it provides. You can create and deploy a request to update your tables without disrupting your users when you want to change your database schema. It is also simple to integrate with third-party tools like Prisma.</p> <p>Prisma is an open-source database toolkit that provides a type-safe ORM (Object-Relational Mapping) layer for modern web applications. When you want to change your database schema, you can use Prisma’s migration feature to generate the necessary SQL scripts that can be applied to your database in a non-blocking way, without disrupting your users. This way you can integrate your PlanetScale with Prisma to efficiently update your schema whilst your application remains online.</p> https://tedneward.github.io/Research/storage/probase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/probase/index.html <p><a href="https://www.microsoft.com/en-us/research/project/probase/">Website</a></p> <blockquote> <p>The goal of Probase to enable machines to better understand human communication. For example, in natural language processing and speech analysis, knowledgebases can help reduce the ambiguities in language. As Probase has a knowledgebase as large as the concept space (of wordly facts) in a human mind, it has unique advantages in these applications.Besides, with the probabilistic knowledge provided by Probase, we build several interesting applications, such as topic search, web table search and document understanding.</p> </blockquote> https://tedneward.github.io/Research/storage/redis/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/redis/index.html <p><a href="http://redis.io/">Website</a> | <a href="https://github.com/redis/redis">Source</a></p> <p>Docker: <code>docker run -p 6379:6379 -it --rm redis</code></p> <p><a href="https://redis.io/commands/">Commands</a></p> <h2><a href="https://redis.io/docs/">Docs</a> (notes)</h2> <p>Redis is a data structure server. At its core, Redis provides a collection of native data types that help you solve a wide variety of problems, from caching to queuing to event processing.</p> <h3>Keys</h3> <p>Redis keys are binary safe, this means that you can use any binary sequence as a key, from a string like "foo" to the content of a JPEG file. The empty string is also a valid key.</p> <p>Very long keys are not a good idea. For instance a key of 1024 bytes is a bad idea not only memory-wise, but also because the lookup of the key in the dataset may require several costly key-comparisons. Even when the task at hand is to match the existence of a large value, hashing it (for example with SHA1) is a better idea, especially from the perspective of memory and bandwidth.</p> <p>Very short keys are often not a good idea. There is little point in writing "u1000flw" as a key if you can instead write "user:1000:followers". The latter is more readable and the added space is minor compared to the space used by the key object itself and the value object. While short keys will obviously consume a bit less memory, your job is to find the right balance.</p> <p>Try to stick with a schema. For instance "object-type:id" is a good idea, as in "user:1000". Dots or dashes are often used for multi-word fields, as in "comment:4321:reply.to" or "comment:4321:reply-to".</p> <p>The maximum allowed key size is 512 MB.</p> <h3>Strings</h3> <p>Redis strings are the most basic Redis data type, representing a sequence of bytes.</p> <p>The Redis String type is the simplest type of value you can associate with a Redis key. It is the only data type in Memcached, so it is also very natural for newcomers to use it in Redis.</p> <p>Since Redis keys are strings, when we use the string type as a value too, we are mapping a string to another string. The string data type is useful for a number of use cases, like caching HTML fragments or pages.</p> <h3>Lists</h3> <p>Redis lists are lists of strings sorted by insertion order.</p> <h3>Sets</h3> <p>Redis sets are unordered collections of unique strings that act like the sets from your favorite programming language (for example, Java HashSets, Python sets, and so on). With a Redis set, you can add, remove, and test for existence O(1) time (in other words, regardless of the number of set elements).</p> <h3>Hashes</h3> <p>Redis hashes are record types modeled as collections of field-value pairs. As such, Redis hashes resemble Python dictionaries, Java HashMaps, and Ruby hashes.</p> <h3>Sorted sets</h3> <p>Redis sorted sets are collections of unique strings that maintain order by each string's associated score.</p> <h3>Streams</h3> <p>A Redis stream is a data structure that acts like an append-only log. Streams help record events in the order they occur and then syndicate them for processing.</p> <h3>Geospatial indexes</h3> <p>Redis geospatial indexes are useful for finding locations within a given geographic radius or bounding box.</p> <h3>Bitmaps</h3> <p>Redis bitmaps let you perform bitwise operations on strings.</p> <h3>Bitfields</h3> <p>Redis bitfields efficiently encode multiple counters in a string value. Bitfields provide atomic get, set, and increment operations and support different overflow policies.</p> <h3>HyperLogLog</h3> <p>The Redis HyperLogLog data structures provide probabilistic estimates of the cardinality (i.e., number of elements) of large sets.</p> <h3>Extensions</h3> <p>To extend the features provided by the included data types, use one of these options:</p> <ul> <li>Custom server-side functions in Lua.</li> <li>Custom Redis modules using the modules API or check out the community-supported modules.</li> <li>Use JSON, querying, time series, and other capabilities provided by Redis Stack.</li> </ul> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="https://redis.com/ebook/redis-in-action/">Redis in Action</a> - Josiah L. Carlson</li> <li><a href="http://openmymind.net/2012/1/23/The-Little-Redis-Book/">The Little Redis Book</a> - K. Seguin (PDF, Epub)</li> <li><a href="https://www.digitalocean.com/community/books/how-to-manage-a-redis-database-ebook">How To Manage a Redis Database</a> - Mark Drake (PDF, EPUB)</li> </ul> https://tedneward.github.io/Research/storage/rethinkdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/rethinkdb/index.html <p>"RethinkDB is the first open-source scalable database built for realtime applications. It exposes a new database access model, in which the developer can tell the database to continuously push updated query results to applications without polling for changes. RethinkDB allows developers to build scalable realtime apps in a fraction of the time with less effort."</p> <p><a href="https://rethinkdb.com/">Website</a> | <a href="https://github.com/rethinkdb/rethinkdb">Source</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li>Publish-Subscribe with RethinkDB: <a href="https://rethinkdb.com/docs/publish-subscribe/javascript/">JavaScript</a> <a href="https://rethinkdb.com/docs/publish-subscribe/python/">Python</a> <a href="https://rethinkdb.com/docs/publish-subscribe/ruby/">Ruby</a></li> </ul> https://tedneward.github.io/Research/storage/s3/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/s3/index.html <p><a href="https://docs.aws.amazon.com/s3/index.html">Website</a> | <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/index.html">User Guide</a> (<a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-userguide.pdf">PDF</a>) | <a href="https://docs.aws.amazon.com/AmazonS3/latest/API/Welcome.html">API Reference</a> (<a href="https://docs.aws.amazon.com/AmazonS3/latest/API/s3-api.pdf">PDF</a>) | <a href="https://docs.aws.amazon.com/amazonglacier/latest/dev/index.html">Glacier Developer Guide</a> (<a href="https://docs.aws.amazon.com/amazonglacier/latest/dev/glacier-dg.pdf">PDF</a>)</p> https://tedneward.github.io/Research/storage/models/event-sourcing/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/event-sourcing/index.html <p>A storage approach that stores a system’s state as an append-only sequence of events. Instead of updating the current state directly, each change to the system is recorded as an event. These events are then used to reconstruct the system’s state at any point in time.</p> <p>Important Considerations for Event Sourcing</p> <ul> <li>Event Immutability: Events should be immutable. Once created, they cannot be changed.</li> <li>Event Sourcing vs. Event-Driven Architecture: While closely related, Event Sourcing is a pattern for storing and reconstructing state, while Event-Driven Architecture is a pattern for loosely coupling components.</li> <li>Eventual Consistency: Event Sourcing often leads to eventual consistency, meaning the system’s state may not be immediately updated after an event is recorded.</li> <li>Performance: Event Sourcing can be performance-intensive, especially for systems with high write loads or complex state transitions.</li> <li>Complexity: Implementing Event Sourcing can be more complex than traditional approaches due to the need for event handling, storage, and reconstruction.</li> <li>When to Use Event Sourcing</li> </ul> <p>Event Sourcing is well-suited for:</p> <ul> <li>Systems that require a complete audit trail of changes.</li> <li>Systems with complex state transitions or business rules.</li> <li>Systems that need to be able to replay past events to recover from failures or explore different scenarios.</li> <li>Systems that benefit from eventual consistency.</li> </ul> <p>Event Sourcing vs. Event-Driven Architecture: While Event-Driven Architecture and Event Sourcing are often used together, they are distinct concepts:</p> <ul> <li>Event-Driven Architecture: Focuses on decoupling components using events as a communication mechanism.</li> <li>Event Sourcing: Focuses on storing and reconstructing state using events.</li> </ul> <p><a href="https://github.com/oskardudycz/EventSourcing.NetCore/tree/main/Workshops/IntroductionToEventSourcing">Introduction to Event Sourcing Workshop</a></p> <p><a href="https://github.com/oskardudycz/EventSourcing.NetCore">Examples and Tutorials of Event Sourcing in .NET</a></p> <p><a href="https://leanpub.com/eventmodeling-and-eventsourcing">Understanding Eventsourcing</a>: The book by Martin Dilger</p> https://tedneward.github.io/Research/storage/models/object/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/object/index.html <p>(Placeholder)</p> https://tedneward.github.io/Research/storage/models/text/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/text/index.html <p>(Placeholder)</p> <p>Thinking primarily ElasticSearch and the like.</p> https://tedneward.github.io/Research/storage/mongodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/mongodb/index.html <p><a href="https://www.mongodb.com/">Website</a> | <a href="https://github.com/mongodb/mongo">Source</a> | <a href="https://docs.mongodb.com/manual/">Docs</a> | <a href="https://www.mongodb.com/cloud/atlas">Cloud-hosted</a></p> <p><a href="https://github.com/fakemongo/fongo">"Fake Mongo", aka fongo</a>: faked out in-memory mongo for java</p> <h2>Books</h2> <ul> <li><a href="https://www.tutorialspoint.com/mongodb">Introduction to MongoDB</a> - Tutorials Point</li> <li><a href="https://riptutorial.com/Download/mongodb.pdf">Learning MongoDB</a> - Based on Unaffiliated Stack Overflow Documentation (PDF)</li> <li><a href="https://github.com/chicagoruby/MongoDB_Koans">MongoDB Koans</a></li> <li><a href="https://goalkicker.com/MongoDBBook/">MongoDB Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/mongodb">MongoDB Succinctly, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="http://openmymind.net/2011/3/28/The-Little-MongoDB-Book/">The Little MongoDB Book</a></li> </ul> <hr> <p><a href="https://docs.mongodb.com/manual/tutorial/store-javascript-function-on-server/">Store a JS function on the server</a>: "(NOTE: There are performance limitations to running JavaScript inside of MongoDB.) There is a special system collection named <code>system.js</code> that can store JavaScript functions for reuse. To store a function, you can use the <code>db.collection.save()</code>, as in the following examples:</p> <pre><code>db.system.js.save( { _id: "echoFunction", value : function(x) { return x; } } ) db.system.js.save( { _id : "myAddFunction" , value : function (x, y){ return x + y; } } ); </code></pre> <p>Once you save a function in the <code>system.js</code> collection, you can use the function from any JavaScript context; e.g. <code>$where</code> operator, <code>mapReduce</code> command or <code>db.collection.mapReduce()</code>."</p> https://tedneward.github.io/Research/storage/nedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nedb/index.html <p>"Embedded persistent or in memory database for Node.js, nw.js, Electron and browsers, 100% JavaScript, no binary dependency. API is a subset of MongoDB's. You can use NeDB as an in-memory only datastore or as a persistent datastore. One datastore is the equivalent of a MongoDB collection. Under the hood, NeDB's persistence uses an append-only format, meaning that all updates and deletes actually result in lines added at the end of the datafile, for performance reasons. The database is automatically compacted (i.e. put back in the one-line-per-document format) every time you load each database within your application."</p> <p><a href="https://github.com/louischatriot/nedb">Source</a></p> <p>Author considers it to be feature-complete.</p> https://tedneward.github.io/Research/storage/nmemory/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nmemory/index.html <p><a href="https://nmemory.net/">Website</a></p> https://tedneward.github.io/Research/storage/nosdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nosdb/index.html <p><a href="https://www.alachisoft.com/nosdb/">Website</a> | <a href="https://github.com/Alachisoft/NCache">Github</a></p> https://tedneward.github.io/Research/storage/objectivitydb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/objectivitydb/index.html <p><a href="https://objectivity.com/objectivity-db/">Website</a></p> <p>Commercial</p> https://tedneward.github.io/Research/storage/models/graph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/graph/index.html <p><a href="https://en.wikipedia.org/wiki/Graph_database">Wikipedia</a></p> <p>"A graph is a structure composed of vertices and edges. Both vertices and edges can have an arbitrary number of key/value-pairs called properties. Vertices denote discrete objects such as a person, a place, or an event. Edges denote relationships between vertices. For instance, a person may know another person, have been involved in an event, and/or have recently been at a particular place. Properties express non-relational information about the vertices and edges. Example properties include a vertex having a name and an age, and an edge having a timestamp and/or a weight.</p> <p>"If a user's domain is composed of a heterogeneous set of objects (vertices) that can be related to one another in a multitude of ways (edges), then a graph may be the right representation to use. In a graph, each vertex is seen as an atomic entity (not simply a "row in a table") that can be linked to any other vertex or have properties added or removed at will. This empowers the data modeler to think in terms of actors within a world of complex relations as opposed to, in relational databases, statically-typed tables joined in aggregate. Once a domain is modeled, that model must then be exploited in order to yield novel, differentiating information. Graph computing has a rich history that includes not only query languages devoid of table-join semantics, but also algorithms that support complex reasoning: path analysis, vertex clustering and ranking, subgraph identification, and more. The world of applied graph computing offers a flexible, intuitive data structure along with a host of algorithms able to effectively leverage that structure." -- from <a href="http://tinkerpop.apache.org/">Apache TinkerPop</a></p> <p>List of graph dbs to add (from TinkerPop page):</p> <ul> <li><a href="https://cn.aliyun.com/product/gdb">Alibaba Graph Database</a> - A real-time, reliable, cloud-native graph database service that supports property graph model.</li> <li><a href="https://aws.amazon.com/neptune/">Amazon Neptune</a> - Fully-managed graph database service.</li> <li><a href="https://s2graph.apache.org/">Apache S2Graph</a> - OLTP graph database running on Apache HBase.</li> <li><a href="https://github.com/lambdazen/bitsy/wiki">Bitsy</a> - A small, fast, embeddable, durable in-memory graph database.</li> <li><a href="https://github.com/blazegraph">Blazegraph</a> - RDF graph database with OLTP support.</li> <li><a href="https://github.com/MartinHaeusler/chronos/tree/master/org.chronos.chronograph">ChronoGraph</a> - A versioned graph database.</li> <li><a href="https://www.datastax.com/products/datastax-enterprise-graph">DSEGraph</a> - DataStax graph database with OLTP and OLAP support.</li> <li><a href="https://grakn.ai/">GRAKN.AI</a> - Distributed OLTP/OLAP knowledge graph system.</li> <li>Hadoop (Spark) - OLAP graph processor using Spark.</li> <li><a href="https://github.com/rayokota/hgraphdb">HGraphDB</a> - OLTP graph database running on Apache HBase.</li> <li><a href="https://www.huaweicloud.com/en-us/product/ges.html">Huawei Graph Engine Service</a> - Fully-managed, distributed, at-scale graph query and analysis service that provides a visualized interactive analytics platform.</li> <li><a href="https://console.ng.bluemix.net/catalog/services/ibm-graph/">IBM Graph</a> - OLTP graph database as a service.</li> <li><a href="https://github.com/SteelBridgeLabs/neo4j-gremlin-bolt">neo4j-gremlin-bolt</a> - OLTP graph database (using Bolt Protocol).</li> <li><a href="https://github.com/pietermartin/sqlg">Sqlg</a> - OLTP implementation on SQL databases.</li> <li><a href="https://stardog.com/">Stardog</a> - RDF graph database with OLTP and OLAP support.</li> <li><a href="https://tinkerpop.apache.org/docs/current/reference/#tinkergraph-gremlin">TinkerGraph</a> - In-memory OLTP and OLAP reference implementation.</li> <li><a href="https://github.com/rmagen/unipop">Unipop</a> - OLTP Elasticsearch and JDBC backed graph.</li> </ul> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="http://graphdatabases.com">Graph Databases</a></li> </ul> https://tedneward.github.io/Research/storage/models/raw/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/raw/index.html <p>(Placeholder)</p> <p>"Blob" storage. Amazon S3, for example--no structure at all.</p> https://tedneward.github.io/Research/storage/models/time-series/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/time-series/index.html <p><a href="https://en.wikipedia.org/wiki/Time_series_database">Wikipedia</a></p> <p>"A time series database is a specialized database that efficiently stores and retrieves time-stamped data. Each time series is stored individually as an optimized list of values, enabling fast data retrieval and cost-effective storage." --<a href="https://www.honeycomb.io/blog/time-series-database">Honeycomb</a></p> <p>"Time series databases are specialized databases designed to manage data that is organized and indexed by time. Unlike traditional databases, which are optimized for general-purpose data storage, TSDBs focus on efficiently storing, querying, and analyzing sequences of time-stamped data points." -- <a href="https://www.datacamp.com/blog/time-series-database">Datacamp</a></p> <p><strong><em>Characteristics of Time Series Databases</em></strong>: There are a few things that TSDBs do differently than traditional databases.</p> <p><strong>Optimized for time-stamped data.</strong> At their core, TSDBs are built to handle data with timestamps as a fundamental attribute. Every data point in a TSDB includes a timestamp, which serves as its primary index. This allows these databases to efficiently store and retrieve time-ordered sequences and provide quick access to historical trends or recent events. Most TSDBs use time-based partitioning, meaning the data is stored in partitions based on time intervals (e.g., hourly, daily). This enables efficient pruning, where queries ignore irrelevant partitions altogether. They can also implement time buckets, grouping data into predefined time windows (e.g., 1 minute, 1 hour) for faster aggregations.</p> <p><strong>High ingestion rates.</strong> Time series data is often generated at a rapid pace—think of IoT devices sending thousands of data points per second or a server monitoring tool capturing system metrics in real time. TSDBs are optimized for these high write rates and can ingest vast amounts of data without slowing down or losing information. This is usually achieved using append-only data storage models and in-memory buffers to prevent locks or transactional bottlenecks.</p> <p><strong>Efficient queries for time ranges.</strong> Analyzing time series data often involves querying specific time intervals or windows, such as “last 24 hours” or “this year compared to last year.” TSDBs are built with this in mind, offering specialized query capabilities that allow users to quickly retrieve data over defined time ranges. They also support aggregations like averages, sums, or trends to offer valuable analytics without complex query logic.</p> <p>The query optimization techniques include:</p> <ul> <li>Pre-aggregated data: TSDBs often pre-calculate summaries for common time intervals (e.g., hourly or daily averages). <ul> <li>Sliding window algorithms: These help efficiently compute metrics over moving time windows, such as rolling averages.</li> </ul> </li> </ul> <p><strong>Data compression and retention policies.</strong> To manage the vast amount of time series data generated over time, TSDBs use advanced data compression techniques. These methods reduce storage requirements while preserving query performance. TSDBs usually include retention policies so the users can define how long data should be kept. For example, a system might retain detailed data for the past month while downsampling for older data. Downsampling is the process of reducing the granularity of data over time. For example: Raw temperature readings might be recorded every 10 seconds for the most recent 7 days; or for older data, the system might downsample to hourly averages to save space while still retaining historical trends.</p> <p>Examples of advanced compression techniques include:</p> <ul> <li>Delta encoding: Storing the difference between consecutive values instead of the full value. <ul> <li>Gorilla compression: A method used to efficiently compress floating-point time series data by storing changes in binary format.</li> </ul> </li> </ul> <p><strong>Writes dominate.</strong> Our primary requirement for a TSDB is that it should always be available to take writes. As we have hundreds of systems exposing multiple data items, the write rate might easily exceed tens of millions of data points each second. In constrast, the read rate is usually a couple orders of magnitude lower as it is primarily from automated systems watching ’important’ time series, data visualization systems presenting dashboards for human consumption, or from human operators wishing to diagnose an observed problem.</p> <p><strong>State transitions.</strong> We wish to identify issues that emerge from a new software release, an unexpected side effect of a configuration change, a network cut and other issues that result in a significant state transition. Thus, we wish for our TSDB to support fine-grained aggregations over short-time windows. The ability to display state transitions within tens of seconds is particularly prized as it allows automation to quickly remediate problems before they become wide spread.</p> <p><strong>High availability.</strong> Even if a network partition or other failure leads to disconnection between different datacenters, systems operating within any given datacenter ought to be able to write data to local TSDB machines and be able to retrieve this data on demand.</p> <p><strong>Fault tolerance.</strong> We wish to replicate all writes to multiple regions so we can survive the loss of any given datacenter or geographic region due to a disaster.</p> <p><strong>Lack of ACID.</strong> These systems do not store any user data so traditional ACID guarantees are not a core requirement for TSDBs. However, a high percentage of writes must succeed at all times, even in the face of disasters that might render entire datacenters unreachable. Additionally, recent data points are of higher value than older points given the intuition that knowing if a particular system or service is broken right now is more valuable to an operations engineer than knowing if it was broken an hour ago.</p> <h2>Databases</h2> <ul> <li><a href="https://github.com/Netflix/atlas">atlas</a> In-memory dimensional time series database from Netflix.</li> <li><a href="/storage/cassandra">Cassandra</a> Apache Cassandra is an open source NoSQL distributed database trusted by thousands of companies for scalability and high availability without compromising performance.</li> <li><a href="https://clickhouse.com/">ClickHouse</a> An open-source, high performance columnar OLAP database management system for real-time analytics using SQL.</li> <li><a href="https://crate.io/use-cases/time-series/">cratedb</a> The SQL database for complex, large scale time series workloads in industrial IoT.</li> <li><a href="https://druid.apache.org/">druid</a> A high performance real-time analytics database.</li> <li><a href="https://fauna.com">fauna</a> Fauna is a flexible, developer-friendly, transactional database delivered as a secure and scalable cloud API with native GraphQL.</li> <li><a href="/storage/influxdb/">InfluxDB</a> Is the essential time series toolkit - dashboards, queries, tasks and agents all in one place.</li> <li><a href="https://kairosdb.github.io/">KairosDB</a> Fast Time Series Database on Cassandra.</li> <li><a href="/storage/opentsdb">OpenTSDB</a> The Scalable Time Series Database.</li> <li><a href="https://prometheus.io/">prometheus</a> An open-source systems monitoring and alerting toolkit originally built at <a href="https://soundcloud.com">SoundCloud</a>.</li> <li><a href="https://github.com/questdb/questdb">QuestDB</a> An open source SQL database designed to process time series data, faster.</li> <li><a href="https://siridb.com/">SiriDB</a> An highly-scalable, robust and super fast time series database.</li> <li><a href="/storage/timescaledb">TimescaleDB</a> TimescaleDB is the leading open-source relational database with support for time-series data.</li> <li><a href="https://tdengine.com">TDengine</a> An open-source time-series database with high-performance, scalability and SQL support.</li> <li><a href="https://github.com/graphite-project/whisper">Whisper (Graphite)</a>: a file-based time-series database format for Graphite. (<a href="http://graphite.readthedocs.org/">Docs</a>) | <a href="https://graphiteapp.org/">Graphite website</a></li> </ul> <h3>Managed database services</h3> <ul> <li><a href="https://aws.amazon.com/timestream/">Amazon Timestream</a> A fast, scalable, and serverless time series database service for IoT and operational applications.</li> <li><a href="https://azure.microsoft.com/en-us/services/postgresql/#overview">Azure Database for PostgreSQL</a> Fully managed, intelligent, and scalable PostgreSQL with support for TimeScaleDB.</li> <li><a href="https://azure.microsoft.com/en-us/services/time-series-insights/">Azure time series insights</a> Visualize IoT data in real time.</li> <li><a href="https://www.influxdata.com/products/influxdb-cloud/">InfluxDB Cloud</a> It’s a fast, elastic, serverless real-time monitoring platform, dashboarding engine, analytics service and event and metrics processor.</li> <li><a href="https://www.timescale.com/products/#Timescale-Cloud">TimeScaleDB Cloud</a> Managed TimeScaleDB service.</li> <li><a href="https://cloud.tdengine.com/">TDengine Cloud</a> Serverless, fully managed cloud service for time series data</li> </ul> <h2>Packages</h2> <h3>Python</h3> <ul> <li><a href="https://github.com/arundo/adtk">adtk</a> A Python toolkit for rule-based/unsupervised anomaly detection in time series.</li> <li><a href="https://github.com/aeon-toolkit/aeon">aeon</a> A unified framework for machine learning with time series.</li> <li><a href="https://github.com/SeldonIO/alibi-detect">alibi-detect</a> Algorithms for outlier, adversarial and drift detection.</li> <li><a href="https://github.com/winedarksea/AutoTS">AutoTS</a> A time series package for Python designed for rapidly deploying high-accuracy forecasts at scale.</li> <li><a href="https://github.com/AutoViML/Auto_TS">Auto_TS</a> Automatically build ARIMA, SARIMAX, VAR, FB Prophet and XGBoost Models on Time Series data sets with a Single Line of Code. Now updated with Dask to handle millions of rows.</li> <li><a href="https://github.com/cesium-ml/cesium">cesium</a> Open-Source Platform for Time Series Inference.</li> <li><a href="https://github.com/unit8co/darts">darts</a> Time Series Made Easy in Python. A python library for easy manipulation and forecasting of time series.</li> <li><a href="https://github.com/deeptime-ml/deeptime">deeptime</a> Python library for analysis of time series data including dimensionality reduction, clustering, and Markov model estimation.</li> <li><a href="https://github.com/DynamicTimeWarping/dtw-python">dtw-python</a> Python port of R's Comprehensive Dynamic Time Warp algorithm package.</li> <li><a href="https://github.com/tinkoff-ai/etna">etna</a> ETNA is an easy-to-use time series forecasting framework.</li> <li><a href="https://github.com/microsoft/FOST">fost</a> Forecasting open source tool aims to provide an easy-use tool for spatial-temporal forecasting.</li> <li><a href="https://github.com/descendant-ai/functime">functime</a> Time-series machine learning and embeddings at scale.</li> <li><a href="https://github.com/awslabs/gluon-ts">gluon-ts</a> Probabilistic time series modeling in Python from AWS.</li> <li><a href="https://github.com/equinor/gordo">gordo</a> Building thousands of models with time series data to monitor systems.</li> <li><a href="https://github.com/linkedin/greykite">greykite</a> A flexible, intuitive and fast forecasting library from LinkedIn.</li> <li><a href="https://github.com/hmmlearn/hmmlearn">hmmlearn</a> Hidden Markov Models in Python, with <code>scikit-learn</code> like API.</li> <li><a href="https://github.com/DataCanvasIO/HyperTS">HyperTS</a> A Full-Pipeline Automated Time Series (AutoTS) Analysis Toolkit.</li> <li><a href="https://github.com/facebookresearch/kats">kats</a> A kit to analyze time series data, a lightweight, easy-to-use, generalizable, and extendable framework to perform time series analysis, from understanding the key statistics and characteristics, detecting change points and anomalies, to forecasting future trends.</li> <li><a href="https://github.com/cvjena/libmaxdiv">libmaxdiv</a> Implementation of the Maximally Divergent Intervals algorithm for Anomaly Detection in multivariate spatio-temporal time-series.</li> <li><a href="https://github.com/CamDavidsonPilon/lifelines">lifelines</a> Survival analysis in Python.</li> <li><a href="https://github.com/zillow/luminaire">luminaire</a> A python package that provides ML driven solutions for monitoring time series data. Luminaire provides several anomaly detection and forecasting capabilities that incorporate correlational and seasonal patterns in the data over time as well as uncontrollable variations.</li> <li><a href="https://github.com/matrix-profile-foundation/mass-ts">mass-ts</a> Mueen's Algorithm for Similarity Search, a library used for searching time series sub- sequences under z-normalized Euclidean distance for similarity.</li> <li><a href="https://github.com/matrix-profile-foundation/matrixprofile">matrixprofile</a> A Python library making time series data mining tasks, utilizing matrix profile algorithms, accessible to everyone.</li> <li><a href="https://github.com/salesforce/Merlion">Merlion</a> A Python library for time series intelligence. It provides an end-to-end machine learning framework that includes loading and transforming data, building and training models, post-processing model outputs, and evaluating model performance.</li> <li><a href="https://github.com/Nixtla/neuralforecast">neuralforecast</a> Scalable and user friendly neural brain forecasting algorithms.</li> <li><a href="https://github.com/Nixtla/nixtla">nixtla</a> Automated time series processing and forecasting.</li> <li><a href="https://github.com/uber/orbit">orbit</a> A package for Bayesian forecasting with object-oriented design and probabilistic models under the hood from Uber.</li> <li><a href="https://github.com/pastas/pastas">pastas</a> An open-source Python framework for the analysis of hydrological time series.</li> <li><a href="https://github.com/alkaline-ml/pmdarima">pmdarima</a> A statistical library designed to fill the void in Python's time series analysis capabilities, including the equivalent of R's <code>auto.arima</code> function.</li> <li><a href="https://github.com/facebook/prophet">prophet</a> Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.</li> <li><a href="https://github.com/antoinecarme/pyaf">pyaf</a> PyAF is an Open Source Python library for Automatic Time Series Forecasting built on top of popular pydata modules.</li> <li><a href="https://github.com/wwrechard/pydlm">PyDLM</a> Bayesian time series modeling package. Based on the Bayesian dynamic linear model (Harrison and West, 1999) and optimized for fast model fitting and inference.</li> <li><a href="https://github.com/RJT1990/pyflux">PyFlux</a> Open source time series library for Python.</li> <li><a href="https://github.com/PYFTS/pyFTS">pyFTS</a> An open source library for Fuzzy Time Series in Python.</li> <li><a href="https://github.com/yzhao062/Pyod">Pyod</a> A Python toolbox for scalable outlier detection (Anomaly Detection).</li> <li><a href="https://github.com/WenjieDu/PyPOTS">PyPOTS</a> A python toolbox/library for data mining on partially-observed time series (A.K.A. irregularly-sampled time series), supporting tasks of forecasting/imputation/classification/clustering on incomplete multivariate time series with missing values.</li> <li><a href="https://github.com/olivercliff/pyspi">pyspi</a> Comparative analysis of pairwise interactions in multivariate time series.</li> <li><a href="https://github.com/business-science/pytimetk">pytimetk</a> The time series toolkit for python.</li> <li><a href="https://github.com/kLabUM/rrcf">rrcf</a> Implementation of the Robust Random Cut Forest algorithm for anomaly detection on streams.</li> <li><a href="https://github.com/mikekeith52/scalecast">scalecast</a> A scalable forecasting approach for Timeseries in Python</li> <li><a href="https://github.com/carlomazzaferro/scikit-hts">scikit-hts</a> Hierarchical Time Series Forecasting with a familiar API.</li> <li><a href="https://github.com/dmbee/seglearn">seglearn</a> A python package for machine learning time series or sequences.</li> <li><a href="https://gitlab.com/shyft-os/shyft">shyft</a> Time-series for python and c++, including distributed storage and calculations Hydrologic Forecasting Toolbox, high-performance flexible stacks, including calibration Energy-market models and micro services.</li> <li><a href="https://github.com/cjekel/similarity_measures">similarity_measures</a> Quantify the difference between two arbitrary curves.</li> <li><a href="https://github.com/JoaquinAmatRodrigo/skforecast">skforecast</a> Time series forecasting with scikit-learn models.</li> <li><a href="https://github.com/alan-turing-institute/sktime">sktime</a> A <code>scikit-learn</code> compatible Python toolbox for learning with time series.</li> <li><a href="https://github.com/Nixtla/statsforecast">statsforecast</a> Lightning :zap: fast forecasting with statistical and econometric models.</li> <li><a href="https://www.statsmodels.org/stable/tsa.html">statsmodels.tsa</a> Time Series Analysis (tsa) <code>statsmodels.tsa</code> contains model classes and functions that are useful for time series analysis.</li> <li><a href="https://github.com/TDAmeritrade/stumpy">stumpy</a> A powerful and scalable Python library that can be used for a variety of time series data mining tasks.</li> <li><a href="https://github.com/davidhallac/TICC">TICC</a> A python solver for efficiently segmenting and clustering a multivariate time series.</li> <li><a href="https://github.com/X-DataInitiative/tick">tick</a> Module for statistical learning, with a particular emphasis on time-dependent modelling.</li> <li><a href="https://github.com/AzulGarza/timecopilot">TimeCopilot</a> An open-source forecasting agent that combines the power of large language models with state-of-the-art time series foundation models.</li> <li><a href="https://github.com/microprediction/timemachines">timemachines</a> Continuously evaluated, functional, incremental, time-series forecasting.</li> <li><a href="https://github.com/MBrouns/timeseers">TimeSeers</a> A hierarchical Bayesian Time Series model based on Prophet, written in PyMC3.</li> <li><a href="https://github.com/google-research/timesfm">TimesFM</a> TimesFM (Time Series Foundation Model) is a pretrained time-series foundation model developed by Google Research for time-series forecasting.</li> <li><a href="https://pypi.org/project/time-series-generator/">Time Series Generator</a> Provides a solution for the direct multi-step outputs limitation in Keras.</li> <li><a href="https://github.com/datamllab/tods">tods</a> An Automated Time-series Outlier Detection System.</li> <li><a href="https://github.com/philipdarke/torchtime">torchtime</a> Time series data sets for PyTorch.</li> <li><a href="https://github.com/WenjieDu/TSDB">TSDB</a> Time-Series DataBase: A Python toolbox helping load time-series datasets easily.</li> <li><a href="https://github.com/timeseriesAI/tsai">tsai</a> State-of-the-art Deep Learning library for Time Series and Sequences.</li> <li><a href="https://github.com/WenjieZ/TSCV">tscv</a> Time Series Cross-Validation - an extension for scikit-learn.</li> <li><a href="https://github.com/predict-idlab/tsflex">tsflex</a> Flexible time series feature extraction &amp; processing.</li> <li><a href="https://github.com/tslearn-team/tslearn">tslearn</a> The machine learning toolkit for time series analysis in Python.</li> <li><a href="https://github.com/hsbc/tslumen">tslumen</a> A library for Time Series Exploratory Data Analysis (EDA).</li> <li><a href="https://github.com/cerlymarco/tsmoothie">tsmoothie</a> A python library for time-series smoothing and outlier detection in a vectorized way.</li> </ul> <h4>Date and Time</h4> <p><em>Libraries for working with dates and times.</em></p> <ul> <li><a href="https://github.com/sffjunkie/astral">astral</a> Python calculations for the position of the sun and moon.</li> <li><a href="https://github.com/arrow-py/arrow">Arrow</a> - A Python library that offers a sensible and human-friendly approach to creating, manipulating, formatting and converting dates, times and timestamps.</li> <li><a href="https://github.com/KoffeinFlummi/Chronyk">Chronyk</a> - A Python 3 library for parsing human-written times and dates.</li> <li><a href="https://github.com/dateutil/dateutil">dateutil</a> - Extensions to the standard Python <a href="https://docs.python.org/3/library/datetime.html">datetime</a> module.</li> <li><a href="https://github.com/myusuf3/delorean/">delorean</a> - A library for clearing up the inconvenient truths that arise dealing with datetimes.</li> <li><a href="https://github.com/timofurrer/maya">maya</a> - Datetimes for Humans.</li> <li><a href="https://github.com/zachwill/moment">moment</a> - A Python library for dealing with dates/times. Inspired by <a href="http://momentjs.com/">Moment.js</a>.</li> <li><a href="https://github.com/sdispater/pendulum">Pendulum</a> - Python datetimes made easy.</li> <li><a href="https://github.com/shinux/PyTime">PyTime</a> - An easy-to-use Python module which aims to operate date/time/datetime by string.</li> <li><a href="https://launchpad.net/pytz">pytz</a> - World timezone definitions, modern and historical. Brings the <a href="https://en.wikipedia.org/wiki/Tz_database">tz database</a> into Python.</li> <li><a href="https://github.com/dirn/When.py">when.py</a> - Providing user-friendly functions to help perform common date and time actions.</li> </ul> <h4>Feature Engineering</h4> <ul> <li><a href="https://github.com/raphaelvallat/antropy">AntroPy</a> Time-efficient algorithms for computing the entropy and complexity of time-series.</li> <li><a href="https://github.com/chlubba/catch22">catch22</a> CAnonical Time-series CHaracteristics, 22 high-performing time-series features in C, Python and Julia.</li> <li><a href="https://github.com/alteryx/featuretools">featuretools</a> An open source python library for automated feature engineering.</li> <li><a href="https://github.com/google/temporian">temporian</a> Temporian is an open-source Python library for preprocessing ⚡ and feature engineering 🛠 temporal data 📈 for machine learning applications 🤖</li> <li><a href="https://github.com/Nixtla/tsfeatures">tsfeatures</a> Calculates various features from time series data. Python implementation of the R package tsfeatures.</li> <li><a href="https://github.com/fraunhoferportugal/tsfel">tsfel</a> An intuitive library to extract features from time series.</li> <li><a href="https://github.com/predict-idlab/tsflex">tsflex</a> Flexible &amp; efficient time series feature extraction &amp; processing package.</li> <li><a href="https://github.com/blue-yonder/tsfresh">tsfresh</a> The package contains many feature extraction methods and a robust feature selection algorithm.</li> </ul> <h4>Time Series Segmentation &amp; Change Point Detection</h4> <ul> <li><a href="https://github.com/hildensia/bayesian_changepoint_detection">bayesian_changepoint_detection</a> Methods to get the probability of a change point in a time series. Both online and offline methods are available.</li> <li><a href="https://github.com/ruipgil/changepy">changepy</a> Change point detection in time series in pure python.</li> <li><a href="https://github.com/zhaokg/Rbeast">RBEAST</a> Bayesian Change-Point Detection and Time Series Decomposition.</li> <li><a href="https://github.com/deepcharles/ruptures">ruptures</a> A Python library for off-line change point detection. This package provides methods for the analysis and segmentation of non-stationary signals.</li> <li><a href="https://github.com/alan-turing-institute/TCPDBench">TCPDBench</a> Turing Change Point Detection Benchmark, a benchmark evaluation of change point detection algorithms.</li> </ul> <h4>Time Series Generation and Augmentation</h4> <ul> <li><a href="https://github.com/sdv-dev/DeepEcho">DeepEcho</a> Synthetic Data Generation for mixed-type, multivariate time series.</li> <li><a href="https://github.com/firmai/deltapy">deltapy</a> Tabular Data Augmentation &amp; Feature Engineering.</li> <li><a href="https://github.com/uchidalab/time_series_augmentation">time_series_augmentation</a> An example of time series augmentation methods with Keras.</li> <li><a href="https://github.com/TimeSynth/TimeSynth">TimeSynth</a> A multipurpose library for synthetic time series in Python.</li> <li><a href="https://github.com/arundo/tsaug">tsaug</a> A Python package for time series augmentation.</li> <li><a href="https://github.com/alexandervnikitin/tsgm">tsgm</a> Synthetic time series generation and time series augmentations.</li> </ul> <h4>Visualization</h4> <ul> <li><a href="https://github.com/altair-viz/altair">atlair</a> Declarative statistical visualization library for Python.</li> <li><a href="https://github.com/matplotlib/matplotlib">matplotlib</a> A comprehensive library for creating static, animated, and interactive visualizations in Python.</li> <li><a href="https://github.com/plotly/plotly.py">plotly</a> A graphing library makes interactive, publication-quality graphs.</li> <li><a href="https://github.com/predict-idlab/plotly-resampler">plotly-resampler</a> Wrapper for Plotly figures, making large sequential plots scalable.</li> <li><a href="https://github.com/mwaskom/seaborn">seaborn</a> A data visualization library based on <a href="https://matplotlib.org">matplotlib</a> that provides a high-level interface for drawing attractive and informative statistical graphics.</li> <li><a href="https://github.com/predict-idlab/tsdownsample">tsdownsample</a> Extremely fast time series downsampling for visualisation.</li> </ul> <h4>Benchmarking &amp; Contests</h4> <ul> <li><a href="https://github.com/microprediction/timeseries-elo-ratings">timeseries-elo-ratings</a> computes <a href="https://microprediction.github.io/timeseries-elo-ratings/html_leaderboards/overall.html">Elo ratings</a>.</li> <li><a href="https://github.com/microprediction/m6">m6</a> Provides utilities for the <a href="https://mofc.unic.ac.cy/the-m6-competition/">M6-Competition</a></li> <li>Microprediction time-series <a href="https://www.microprediction.com/competitions">competitions</a></li> <li><a href="https://github.com/waico/SKAB">SKAB</a> Skoltech Anomaly Benchmark. Time-series data for evaluating Anomaly Detection algorithms.</li> </ul> <h3>R</h3> <ul> <li><a href="https://cran.r-project.org/package=bcp">bcp</a> Bayesian Analysis of Change Point Problems.</li> <li><a href="https://google.github.io/CausalImpact/CausalImpact.html">CausalImpact</a> An R package for causal inference using Bayesian structural time-series models.</li> <li><a href="https://github.com/rkillick/changepoint">changepoint</a> Implements various mainstream and specialised changepoint methods for finding single and multiple changepoints within data.</li> <li><a href="https://cran.r-project.org/package=cpm">cpm</a> Sequential and Batch Change Detection Using Parametric and Nonparametric Methods.</li> <li><a href="https://github.com/rkillick/EnvCpt">EnvCpt</a> Detection of Structural Changes in Climate and Environment Time Series.</li> <li><a href="https://github.com/tidyverts/fable">fable</a> A <a href="https://github.com/tidyverts">tidyverts</a> package for tidy time series forecasting.</li> <li><a href="https://github.com/tidyverts/fasster">fasster</a> A <a href="https://github.com/tidyverts">tidyverts</a> package for forecasting with additive switching of seasonality, trend and exogenous regressors.</li> <li><a href="https://github.com/tidyverts/feasts">feasts</a> A <a href="https://github.com/tidyverts">tidyverts</a> package for feature extraction and statistics for time series.</li> <li><a href="https://cran.r-project.org/package=fpop">fpop</a> Segmentation using Optimal Partitioning and Function Pruning.</li> <li><a href="https://github.com/config-i1/greybox">greybox</a> Regression model building and forecasting in R.</li> <li><a href="https://github.com/business-science/modeltime">modeltime</a> Modeltime unlocks time series forecast models and machine learning in one framework.</li> <li><a href="https://github.com/tdhock/penaltyLearning">penaltyLearning</a> Algorithms for supervised learning of penalty functions for change detection.</li> <li><a href="https://github.com/hendersontrent/Rcatch22">Rcatch22</a> R package for calculation of 22 CAnonical Time-series CHaracteristics.</li> <li><a href="https://github.com/config-i1/smooth">smooth</a> The set of smoothing functions used for time series analysis and in forecasting.</li> <li><a href="https://github.com/hendersontrent/theft">theft</a> R package for Tools for Handling Extraction of Features from Time series.</li> <li><a href="https://github.com/business-science/timetk">timetk</a> A <code>tidyverse</code> toolkit to visualize, wrangle, and transform time series data.</li> <li><a href="https://github.com/tidyverts/tsibble">tsibble</a> A <a href="https://github.com/tidyverts">tidyverts</a> package with tidy temporal data frames and tools.</li> <li><a href="https://github.com/PetoLau/TSrepr">tsrepr</a> TSrepr: R package for time series representations.</li> </ul> <h3>Java</h3> <ul> <li><a href="https://github.com/patrickzib/SFA">SFA</a> Scalable Time Series Data Analytics.</li> <li><a href="https://github.com/uea-machine-learning/tsml">tsml</a> Java time series machine learning tools in a Weka compatible toolkit.</li> </ul> <h3>JavaScript</h3> <h4>Visualization</h4> <ul> <li><a href="https://github.com/square/cubism">cubism</a> A <a href="http://d3js.org">D3</a> plugin for visualizing time series. Use Cubism to construct better realtime dashboards, pulling data from <a href="https://github.com/square/cubism/wiki/Graphite">Graphite</a>, <a href="https://github.com/square/cubism/wiki/Cube">Cube</a> and other sources.</li> <li><a href="https://github.com/apache/echarts">echarts</a> A free, powerful charting and visualization library offering an easy way of adding intuitive, interactive, and highly customizable charts to your commercial products.</li> <li><a href="https://www.fusioncharts.com/fusiontime">fusiontime</a> Helps you visualize time-series and stock data in JavaScript, with just a few lines of code.</li> <li><a href="https://github.com/highcharts/highcharts">highcharts</a> A JavaScript charting library based on SVG, with fallbacks to VML and canvas for old browsers.</li> <li><a href="https://github.com/awslabs/synchro-charts">synchro-charts</a> A front-end component library that provides a collection of components to visualize time-series data.</li> </ul> <h3>Spark</h3> <ul> <li><a href="https://github.com/twosigma/flint">flint</a> A Time Series Library for Apache Spark.</li> </ul> <h3>MATLAB</h3> <ul> <li><a href="https://github.com/benfulcher/hctsa">hctsa</a> Highly comparative time-series analysis.</li> </ul> <h2>Annotation and Labeling</h2> <ul> <li><a href="https://github.com/alan-turing-institute/AnnotateChange">AnnotateChange</a> - A simple flask application to collect annotations for the Turing Change Point Dataset, a benchmark dataset for change point detection algorithms.</li> <li><a href="https://github.com/baidu/Curve">Curve</a> - An open-source tool to help label anomalies on time-series data</li> <li><a href="https://github.com/Microsoft/TagAnomaly">TagAnomaly</a> - Anomaly detection analysis and labeling tool, specifically for multiple time series (one time series per category)</li> <li><a href="https://github.com/CrowdCurio/time-series-annotator">time-series-annotator</a> - Time Series Annotation Library implements classification tasks for time series.</li> <li><a href="https://github.com/avenix/WDK">WDK</a> - The Wearables Development Toolkit (WDK) is a set of tools to facilitate the development of activity recognition applications with wearable devices.</li> </ul> <h2>Reading</h2> <h3>Blogs</h3> <ul> <li><a href="https://medium.com/pinterest-engineering/goku-building-a-scalable-and-high-performant-time-series-database-system-a8ff5758a181">Goku: Building a Scalable and High-Performant Time Series Database System (Pinterest)</a></li> </ul> <h3>Papers</h3> <ul> <li><a href="https://arxiv.org/abs/2412.20512"><strong>Dive into Time-Series Anomaly Detection: A Decade Review</strong></a>, <em>Paul Boniol, Qinghua Liu, Mingyi Huang, Themis Palpanas, John Paparrizos</em>, 2024</li> <li><a href="(https://arxiv.org/abs/2106.10466)"><strong>TS2Vec: Towards Universal Representation of Time Series</strong></a>, <em>Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, Bixiong Xu</em>, 2022 - <a href="https://github.com/yuezhihan/ts2vec">code</a></li> <li><a href="https://proceedings.mlr.press/v139/xu21h"><strong>Conformal prediction interval for dynamic time-series</strong></a>, <em>Chen Xu, Yao Xie</em>, International Conference on Machine Learning 2021 (long presentation) - <a href="https://github.com/hamrel-cxu/EnbPI">code</a></li> <li><a href="https://arxiv.org/abs/1809.04356v3"><strong>Deep learning for time series classification: a review</strong></a>, <em>H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, P-A. Muller</em>, Data Mining and Knowledge Discovery 2019 - <a href="https://github.com/hfawaz/dl-4-tsc">code</a></li> <li><a href="http://stanford.edu/~boyd/papers/ggs.html"><strong>Greedy Gaussian Segmentation of Multivariate Time Series</strong></a>, <em>D. Hallac, P. Nystrup, and S. Boyd</em>, Advances in Data Analysis and Classification, 13(3), 727–751, 2019. - <a href="https://github.com/cvxgrp/GGS">code</a></li> <li><a href="https://arxiv.org/abs/1910.11162"><strong>U-Time: A Fully Convolutional Network for Time Series Segmentation Applied to Sleep Staging</strong></a>, <em>Mathias Perslev, Michael Jensen, Sune Darkner, Poul Jørgen Jennum, Christian Igel</em>, <em>NeurIPS</em>, 2019. - <a href="https://github.com/perslev/U-Time">code</a></li> <li><a href="https://arxiv.org/abs/cs/0605103"><strong>A Better Alternative to Piecewise Linear Time Series Segmentation</strong></a>, <em>Daniel Lemire</em>, SIAM Data Mining, 2007. - <a href="https://github.com/lemire/PiecewiseLinearTimeSeriesApproximation">code</a></li> <li><a href="https://papers.nips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf"><strong>Time-series Generative Adversarial Networks</strong></a>, <em>Jinsung Yoon, Daniel Jarrett, Mihaela van der Schaar</em>, NeurIPS, 2019. - <a href="https://github.com/jsyoon0823/TimeGAN">code</a></li> <li><a href="https://arxiv.org/abs/1511.03677"><strong>Learning to Diagnose with LSTM Recurrent Neural Networks</strong></a>, <em>Zachary C. Lipton, David C. Kale, Charles Elkan, Randall Wetzel</em>, arXiv:1511.03677, 2015. - <a href="https://github.com/aqibsaeed/Multilabel-timeseries-classification-with-LSTM">code</a></li> <li><a href="https://openreview.net/forum?id=gjNcH0hj0LM"><strong>Coherence-based Label Propagation over Time Series for Accelerated Active Learning</strong></a>, <em>Yooju Shin, Susik Yoon, Sundong Kim, Hwanjun Song, Jae-Gil Lee, Byung Suk Lee</em>, ICLR, 2022. - <a href="https://github.com/kaist-dmlab/TCLP">code</a></li> <li><a href="https://www.vldb.org/pvldb/vol8/p1816-teller.pdf">Facebook Gorilla</a>: "Large-scale internet services aim to remain highly-available and responsive for their users even in the presence of unexpected failures. As these services have grown to support a global audience, they have scaled beyond a few systems running on hundreds of machines to thousands of individual systems running on many thousands of machines, often across multiple geo-replicated datacenters. An important requirement to operating these large scale services is to accurately monitor the health and performance of the underlying system and quickly identify and diagnose problems as they arise. Facebook uses a time series database (TSDB) to store system measuring data points and provides quick query functionalities on top. We next specify some of the constraints that we need to satisy for monitoring and operating Facebook and then describe Gorilla, our new inmemory TSDB that can store tens of millions of datapoints (e.g., CPU load, error rate, latency etc.) every second and respond queries over this data within milliseconds."</li> <li><a href="https://dl.acm.org/doi/abs/10.14778/3705829.3705862">Goku: A Schemaless Time Series Database for Large Scale Monitoring at Pinterest</a></li> </ul> <h3>Books</h3> <ul> <li><a href="https://www.cambridge.org/pl/academic/subjects/computer-science/pattern-recognition-and-machine-learning/bayesian-time-series-models?format=HB">Bayesian Time Series Models</a> 💲 <em>David Barber, A. Taylan Cemgil, Silvia Chiappa</em>, Cambridge Academic Press 2011</li> <li><a href="https://www.knime.com/codeless-time-series-analysis-with-knime">Codeless Time Series Analysis with KNIME</a> 💲 <em>Corey Weisinger, Maarit Widmann, and Daniele Tonini</em>, Packt Publishing 2022</li> <li><a href="https://otexts.com/fpp3/">Forecasting principles and practice (3rd ed)</a> 🆓 <em>Rob J Hyndman and George Athanasopoulos</em> 2021</li> <li><a href="https://www.packtpub.com/product/practical-time-series-analysis/9781788290227">Practical Time Series Analysis</a> 💲 <em>Avishek Pal, PKS Prakash</em>, Packt 2017</li> <li><a href="https://github.com/PacktPublishing/Practical-Time-Series-Analysis">repo with code</a></li> <li><a href="https://www.oreilly.com/library/view/practical-time-series/9781492041641/">Practical Time Series Analysis: Prediction with Statistics and Machine Learning</a> 💲 <em>Aileen Nielsen</em>, O’Reilly 2019 <ul> <li><a href="https://github.com/AileenNielsen/TimeSeriesAnalysisWithPython">repo with code</a></li> </ul> </li> <li><a href="https://www.packtpub.com/product/machine-learning-for-time-series-with-python/9781801819626">Machine Learning for Time-Series with Python</a> 💲 <em>Ben Auffarth</em>, Packt Publishing 2021</li> <li><a href="https://github.com/PacktPublishing/Machine-Learning-for-Time-Series-with-Python">repo with code</a></li> <li><a href="https://phdinds-aim.github.io/time_series_handbook/Preface/Preface.html">Time Series Analysis Handbook</a> 🆓 <em>Students of PhD in Data Science Batch 2023 at the Asian Institute of Management</em>.</li> <li><a href="https://amzn.to/3GITSpD">Visualization of Time-Oriented Data</a> 💲 <em>Wolfgang Aigner, Silvia Miksch, Heidrun Schumann, Christian Tominski</em>, Springer-Verlag 2011</li> </ul> <h3>Courses</h3> <ul> <li><a href="https://courses.analyticsvidhya.com/courses/creating-time-series-forecast-using-python">Analytics Vidhya - Time Series Forecasting using Python</a></li> <li><a href="https://www.coursera.org/projects/intro-time-series-analysis-in-r">Coursera - Intro to Time Series Analysis in R</a></li> <li><a href="https://www.coursera.org/learn/tensorflow-sequences-time-series-and-prediction">Coursera - Sequences, Time Series and Prediction</a></li> <li><a href="https://www.coursera.org/learn/practical-time-series-analysis">Coursera - Practical Time Series Analysis</a></li> <li><a href="https://data4sci.com/timeseries">Data For Science - Time Series for Everyone</a></li> <li><a href="https://www.kaggle.com/learn/time-series">Kaggle - Time Series</a></li> <li><a href="https://www.udacity.com/course/time-series-forecasting--ud980">Udacity - Time Series Forecasting</a></li> </ul> <h3>Tutorials</h3> <ul> <li><a href="https://www.pymc.io/projects/examples/en/latest/time_series/Forecasting_with_structural_timeseries.html">Forecasting with Structural AR Timeseries | PyMC</a></li> <li><a href="https://scikit-learn.org/stable/auto_examples/applications/plot_cyclical_feature_engineering.html">Time-related feature engineering - scikit learn</a></li> </ul> <h2>Repos with Models</h2> <ul> <li><a href="https://github.com/damitkwr/ESRNN-GPU">ESRNN-GPU</a> PyTorch GPU implementation of the ES-RNN model for time series forecasting.</li> <li><a href="https://github.com/jaungiers/LSTM-Neural-Network-for-Time-Series-Prediction">LSTM-Neural-Network-for-Time-Series-Prediction</a> LSTM built using Keras Python package to predict time series steps and sequences.</li> <li><a href="https://github.com/angus924/rocket">ROCKET</a> Exceptionally fast and accurate time series classification using random convolutional kernels.</li> <li><a href="https://github.com/hzy46/TensorFlow-Time-Series-Examples">TensorFlow-Time-Series-Examples</a> Time Series Prediction with <code>tf.contrib.timeseries</code></li> <li><a href="https://github.com/zhykoties/TimeSeries">TimeSeries</a> Implementation of deep learning models for time series in PyTorch.</li> <li><a href="https://github.com/cauchyturing/UCR_Time_Series_Classification_Deep_Learning_Baseline">UCR_Time_Series_Classification_Deep_Learning_Baseline</a> Fully Convlutional Neural Networks for state-of-the-art time series classification.</li> <li><a href="https://github.com/ragulpr/wtte-rnn">wtte-rnn</a> WTTE-RNN a framework for churn and time to event prediction.</li> </ul> <h2>Applications</h2> <ul> <li><a href="https://github.com/binjr/binjr">binjr</a> A Time Series Data Browser.</li> <li><a href="https://www.comp-engine.org/">CompEngine</a> A self-organizing database of time-series data, that allows you to upload time-series data and interactively visualize similar data that have been measured by others.</li> </ul> <h2>Awesome lists</h2> <ul> <li><a href="https://github.com/lmmentel/awesome-time-series">https://github.com/lmmentel/awesome-time-series</a></li> <li><a href="https://github.com/cuge1995/awesome-time-series">https://github.com/cuge1995/awesome-time-series</a></li> <li><a href="https://github.com/MaxBenChrist/awesome_time_series_in_python">https://github.com/MaxBenChrist/awesome_time_series_in_python</a></li> <li><a href="https://github.com/vinta/awesome-python">https://github.com/vinta/awesome-python</a></li> <li><a href="https://github.com/xephonhq/awesome-time-series-database">https://github.com/xephonhq/awesome-time-series-database</a> - A curated list of awesome time series databases, benchmarks and papers.</li> <li><a href="https://github.com/microprediction/awesome-python-benchmarks">https://github.com/microprediction/awesome-python-benchmarks</a> - Timeseries benchmarking mostly.</li> <li><a href="https://github.com/Alro10/deep-learning-time-series">https://github.com/Alro10/deep-learning-time-series</a> - List of state of the art papers focus on deep learning and resources, code and experiments using deep learning for time series forecasting.</li> <li><a href="https://github.com/benfulcher/hctsa/wiki/Related-time-series-resources">https://github.com/benfulcher/hctsa/wiki/Related-time-series-resources</a> - A collection of good resources for time-series analysis (including in other programming languages like python and R).</li> </ul> https://tedneward.github.io/Research/storage/mysql/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/mysql/index.html <h2>Books</h2> <ul> <li><a href="https://dev.mysql.com/doc/mysql-tutorial-excerpt/8.0/en/tutorial.html">MySQL 8.0 Tutorial Excerpt</a> (HTML) <a href="https://downloads.mysql.com/docs/mysql-tutorial-excerpt-8.0-en.pdf">(PDF)</a></li> <li><a href="https://goalkicker.com/MySQLBook/">MySQL Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> </ul> https://tedneward.github.io/Research/storage/neo4j/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/neo4j/index.html <p><a href="https://neo4j.com/">Website</a> | Closed-source</p> <p>Comes in local/desktop version (free) and cloud-hosted versions.</p> <p>Uses <a href="/languages/cypher">Cypher</a> query language for ad-hoc query of the nodes and arcs.</p> <h2>Books</h2> <ul> <li><a href="https://neo4j.com/fullstack-graphql-applications-with-grandstack/">Fullstack GraphQL Applications with GRANDStack – Essential Excerpts</a> - William Lyon (PDF) <em>(email requested)</em></li> <li><a href="https://neo4j.com/graph-algorithms-book/">Graph Algorithms: Practical Examples in Apache Spark and Neo4j</a> - Mark Needham and Amy E. Hodler (PDF, EPUB, MOBI) <em>(email requested)</em></li> <li><a href="http://neo4j.com/books/graph-databases/">Graph Databases 2nd edition</a> - Ian Robinson, Jim Webber and Emil Eifrém (PDF, EPUB, MOBI) <em>(email requested)</em></li> <li><a href="https://neo4j.com/graph-databases-for-dummies/">Graph Databases For Dummies</a> - Dr. Jim Webber and Rik Van Bruggen (PDF) <em>(email requested)</em></li> </ul> https://tedneward.github.io/Research/storage/nocobase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nocobase/index.html <p><a href="https://www.nocobase.com/">Website</a> | <a href="https://github.com/nocobase/nocobase">Source</a></p> https://tedneward.github.io/Research/storage/nuodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nuodb/index.html <p><a href="https://nuodb.com/">Website</a> | Looks commercial?</p> https://tedneward.github.io/Research/storage/models/document/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/document/index.html <p>Structure is conceptually "star"-like, with minimal (or no) relationships outside of the document recognized by the storage system. (Developers can, and usually will, store unique data elements across documents as a way of putting structure in at the application level, but this is typically unrecognized by the storage system itself.)</p> https://tedneward.github.io/Research/storage/models/hierarchical/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/hierarchical/index.html <p>(Placeholder)</p> https://tedneward.github.io/Research/storage/models/relational/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/relational/index.html <p>Built, more or less, on the Codd model of relationships between tuples of data.</p> <p><a href="./codd.pdf">Codd's original paper</a></p> <p>Codd's Twelve (Thirteen) Rules:</p> <ul> <li> <p><strong><em>Rule 0:</em></strong> <strong>The foundation rule:</strong> For any system that is advertised as, or claimed to be, a relational data base management system, that system must be able to manage data bases entirely through its relational capabilities.</p> </li> <li> <p><strong><em>Rule 1:</em></strong> <strong>The information rule:</strong> All information in a relational data base is represented explicitly at the logical level and in exactly one way – by values in tables.</p> </li> <li> <p><strong><em>Rule 2:</em></strong> <strong>The guaranteed access rule:</strong> Each and every datum (atomic value) in a relational data base is guaranteed to be logically accessible by resorting to a combination of table name, primary key value and column name.</p> </li> <li> <p><strong><em>Rule 3:</em></strong> <strong>Systematic treatment of null values:</strong> Null values (distinct from the empty character string or a string of blank characters and distinct from zero or any other number) are supported in fully relational DBMS for representing missing information and inapplicable information in a systematic way, independent of data type.</p> </li> <li> <p><strong><em>Rule 4:</em></strong> <strong>Dynamic online catalog based on the relational model:</strong> The data base description is represented at the logical level in the same way as ordinary data, so that authorized users can apply the same relational language to its interrogation as they apply to the regular data.</p> </li> <li> <p><strong><em>Rule 5:</em></strong> <strong>The comprehensive data sublanguage rule:</strong> A relational system may support several languages and various modes of terminal use (for example, the fill-in-the-blanks mode). However, there must be at least one language whose statements are expressible, per some well-defined syntax, as character strings and that is comprehensive in supporting all of the following items:</p> <ul> <li>Data definition.</li> <li>View definition.</li> <li>Data manipulation (interactive and by program).</li> <li>Integrity constraints.</li> <li>Authorization.</li> <li>Transaction boundaries (begin, commit and rollback).</li> </ul> </li> <li> <p><strong><em>Rule 6:</em></strong> <strong>The view updating rule:</strong> All views that are theoretically updatable are also updatable by the system.</p> </li> <li> <p><strong><em>Rule 7:</em></strong> <strong>Relational Operations Rule / Possible for high-level insert, update, and delete:</strong> The capability of handling a base relation or a derived relation as a single operand applies not only to the retrieval of data but also to the insertion, update and deletion of data.</p> </li> <li> <p><strong><em>Rule 8:</em></strong> <strong>Physical data independence:</strong> Application programs and terminal activities remain logically unimpaired whenever any changes are made in either storage representations or access methods.</p> </li> <li> <p><strong><em>Rule 9:</em></strong> <strong>Logical data independence:</strong> Application programs and terminal activities remain logically unimpaired when information-preserving changes of any kind that theoretically permit unimpairment are made to the base tables.</p> </li> <li> <p><strong><em>Rule 10:</em></strong> <strong>Integrity independence:</strong> Integrity constraints specific to a particular relational data base must be definable in the relational data sublanguage and storable in the catalog, not in the application programs.</p> </li> <li> <p><strong><em>Rule 11:</em></strong> <strong>Distribution independence:</strong> The end-user must not be able to see that the data is distributed over various locations. Users should always get the impression that the data is located at one site only.</p> </li> <li> <p><strong><em>Rule 12:</em></strong> <strong>The nonsubversion rule:</strong> If a relational system has a low-level (single-record-at-a-time) language, that low level cannot be used to subvert or bypass the integrity rules and constraints expressed in the higher level relational language (multiple-records-at-a-time).</p> </li> </ul> <h2>Reading</h2> <ul> <li><a href="https://opentextbc.ca/dbdesign01/">Database Design – 2nd Edition</a> - Adrienne Watt, Nelson Eng @ BCcampus Open Pressbooks (HTML, PDF, EPUB, Kindle)</li> <li><a href="http://www.dcs.warwick.ac.uk/~hugh/TTM/Database-Explorations-revision-2.pdf">Database Explorations</a> (PDF)</li> <li><a href="http://public.dhe.ibm.com/software/dw/db2/express-c/wiki/Database_fundamentals.pdf">Database Fundamentals</a> (PDF)</li> <li><a href="http://www.dcs.warwick.ac.uk/~hugh/TTM/DTATRM.pdf">Databases, Types, and The Relational Model: The Third Manifesto</a> (PDF)</li> <li><a href="http://webdam.inria.fr/Alice/">Foundations of Databases</a></li> <li><a href="http://www.redbook.io">Readings in Database Systems, 5th Ed.</a></li> <li><a href="http://people.cs.aau.dk/~csj/Thesis/">Temporal Database Management</a> - Christian S. Jensen</li> <li><a href="http://web.cecs.pdx.edu/~maier/TheoryBook/TRD.html">The Theory of Relational Databases</a></li> <li><a href="http://www.thethirdmanifesto.com/">The Third Manifesto</a></li> <li><a href="https://www.dcs.warwick.ac.uk/~hugh/TTM/Why-Are-There-No-Relational-DBMSs.pdf">Why Are There No Relational DBMSs?</a> - Hugh Darwen</li> <li><a href="/storage/models/ERModel-Chen.pdf">The Entity-Relationship Model-Toward a Unified View of Data</a> - Peter Pin-Shan Chen</li> </ul> <p>Some interesting relational-oriented sites</p> <ul> <li><a href="https://github.com/decentralion/PokemonSQLTutorial">PokemonSQLTutorial</a></li> </ul> https://tedneward.github.io/Research/storage/models/vector/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/vector/index.html <p>"A vector database is a data storage system used to manage, index, and query high-dimensional vector data. Vectors, in this context, represent data points in multi-dimensional space, often used in machine learning, data mining, and other advanced analytical applications. Vector databases are useful in tasks involving the computation of distances or similarities between vectors, such as recommendation systems, image and video recognition, and natural language processing. Unlike traditional databases that manage structured rows and columns, vector databases handle more complex, high-dimensional data representations, which are essential for applications requiring efficient similarity searches and pattern recognition. Some general purpose databases, such as PostgreSQL and Cassandra, now support both traditional data formats and vector data." --<a href="https://www.instaclustr.com/education/vector-database/top-10-open-source-vector-databases/">NetApp Instaclustr</a></p> <p><a href="https://www.pinecone.io/learn/vector-database/">What is a vector database?</a>: "A vector database indexes and stores vector embeddings for fast retrieval and similarity search, with capabilities like CRUD operations, metadata filtering, horizontal scaling, and serverless."</p> <p><a href="https://www.oracle.com/database/vector-database/">What is a vector database?</a>: "A lesser-known data type, vectors, has seized the spotlight recently as an enabler of generative AI. But vectors—and databases capable of storing and analyzing them—have been toiling backstage for many years. They’re used in geospatial mapping and analysis for city planning, transportation logistics, and environmental analysis. More recently, vectors have been used in recommendation engines for retail products as well as music and video streaming sites."</p> <h2>Implementations</h2> <ul> <li><a href="https://www.pinecone.io/">Pinecone</a> (Commercial?)</li> <li><a href="https://www.trychroma.com/">Chroma</a> <a href="https://github.com/chroma-core/chroma">Source</a>: the open-source AI application database. Batteries included.</li> <li><a href="https://turbopuffer.com/">turbopuffer</a> - Commercial?</li> <li><a href="https://github.com/weaviate/weaviate">Weaviate</a> (<a href="https://weaviate.io/developers/weaviate">Docs</a>): an open source, AI-native vector database.</li> <li><a href="https://milvus.io/">Milvus</a> <a href="https://github.com/milvus-io/milvus">Source</a>: an open-source vector database built for GenAI applications.</li> <li><a href="https://qdrant.tech/">Qdrant</a> <a href="https://github.com/qdrant/qdrant">Source</a>: a vector similarity search engine and vector database.</li> <li><a href="../../postgresql">PostgresQL</a> via pgvector</li> <li><a href="../../cassandra">Cassandra</a></li> <li><a href="../../redis">Redis</a></li> <li>Valkey</li> <li><a href="../../cockroachdb">CockroachDB</a></li> </ul> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://thenewstack.io/how-to-master-vector-databases/">https://thenewstack.io/how-to-master-vector-databases/</a></li> <li><a href="https://venturebeat.com/ai/vector-databases-shiny-object-syndrome-and-the-case-of-a-missing-unicorn">Vector databases: Shiny object syndrome and the case of a missing unicorn</a></li> <li><a href="https://venturebeat.com/ai/from-shiny-object-to-sober-reality-the-vector-database-story-two-years-later">From shiny object to sober reality: The vector database story, two years later</a></li> </ul> https://tedneward.github.io/Research/storage/ndatabase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/ndatabase/index.html <p><a href="https://ndatabase.wixsite.com/home">Website</a> | <a href="https://archive.codeplex.com/">Source</a></p> <p>Appears to be archived in CodePlex; not sure where new development is happening? Last copyright on the website is 2014.</p> https://tedneward.github.io/Research/storage/nextcloud/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nextcloud/index.html <p><a href="https://nextcloud.com/">Website</a> | <a href="https://github.com/nextcloud/server">Source</a></p> https://tedneward.github.io/Research/storage/nocodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nocodb/index.html <p><a href="https://www.nocodb.com/">Website</a> | <a href="https://github.com/nocodb">Source</a></p> <p>Offers a "signup"--maybe cloud-hosted service?</p> <p>Feature list:</p> <ul> <li>Connects to any existing SQL database (MySQL, Postgres, MS SQL)</li> <li>Spreadsheet UI</li> <li>Views: Grid, Gallery, Kanban, Calendar, Custom</li> <li>Send Notifications (Slack)</li> <li>APIs: REST, GraphQL</li> </ul> <p>Docker setup/run:</p> <ul> <li>SQLite: <code>docker run -d --name nocodb -v "$(pwd)"/nocodb:/usr/app/data/ -p 8080:8080 nocodb/nocodb/latest</code></li> <li>MySQL: <code>docker run -d --name nocodb-mysql -v "$(pwd)"/nocodb:/usr/app/data/ -p 8080:8080 \ -e NC_DB="mysql2://host.docker.internal:3306?u=root&amp;p=password&amp;d=d1" \ -e NC_AUTH_JWT_SECRET="569a1821-0a93-45e8-87ab-eb857f20a010" \ nocodb/nocodb:latest</code></li> <li>PostgreSQL: <code>docker run -d --name nocodb-postgres -v "$(pwd)"/nocodb:/usr/app/data/ -p 8080:8080 \ -e NC_DB="pg://host.docker.internal:5432?u=root&amp;p=password&amp;d=d1" \ -e NC_AUTH_JWT_SECRET="569a1821-0a93-45e8-87ab-eb857f20a010" \ nocodb/nocodb:latest</code></li> <li>MSSQL: <code>docker run -d --name nocodb-mssql -v "$(pwd)"/nocodb:/usr/app/data/ -p 8080:8080 \ -e NC_DB="mssql://host.docker.internal:1433?u=root&amp;p=password&amp;d=d1" \ -e NC_AUTH_JWT_SECRET="569a1821-0a93-45e8-87ab-eb857f20a010" \ nocodb/nocodb:latest</code></li> </ul> <p>... or use docker-compose:</p> <pre><code>git clone https://github.com/nocodb/nocodb # for MySQL cd nocodb/docker-compose/mysql # for PostgreSQL cd nocodb/docker-compose/pg # for MSSQL cd nocodb/docker-compose/mssql docker-compose up -d </code></pre> <h2>Articles</h2> <ul> <li><a href="https://www.xda-developers.com/nocodb-guide/">"I replaced Airtable with NocoDb; here's how"</a></li> </ul> https://tedneward.github.io/Research/storage/objectbox/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/objectbox/index.html <p><a href="https://objectbox.io/">Website</a> | <a href="https://github.com/objectbox/">Source</a></p> <p><a href="https://objectbox.io/mongodb/">MongoDB integration</a></p> https://tedneward.github.io/Research/storage/models/double-entry/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/double-entry/index.html <p><a href="https://pgrs.net/2025/06/17/double-entry-ledgers-missing-primitive-in-modern-software/">Double-entry ledgers are a missing primitive in software</a>: "A double-entry ledger at its core is a few simple concepts put together:</p> <ul> <li>The current amount or balance of a thing</li> <li>A historical record of how the amount got to that amount (immutable, append only, etc)</li> <li>Where that amount came from at each step</li> </ul> <p>"The fact that every transfer only moves amounts, never creates them from scratch, is a built-in error check.2 And the historical record serves as an audit log."</p> <h2>Implementations</h2> <ul> <li><a href="https://github.com/pgr0ss/pgledger">pgledger</a>: pure PostgreSQL</li> <li><a href="/tools/ledger">ledger</a></li> </ul> https://tedneward.github.io/Research/storage/models/key-value/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/key-value/index.html <p>(Placeholder)</p> <p>Triple-stores, a la RDF, go in here?</p> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://www.nan.fyi/database">"Build Your Own Database"</a>: A step-by-step guide to building a key-value database from scratch.</li> </ul> https://tedneward.github.io/Research/storage/models/spatial/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/spatial/index.html <p>(Placeholder)</p> https://tedneward.github.io/Research/storage/models/wide-column/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/models/wide-column/index.html <p>(Placeholder)</p> https://tedneward.github.io/Research/storage/nebula-graph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nebula-graph/index.html <p><a href="https://www.nebula-graph.io/">Website</a> | <a href="https://github.com/vesoft-inc/nebula">Source</a></p> https://tedneward.github.io/Research/storage/nitrite/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/nitrite/index.html <p><a href="https://www.dizitart.org/nitrite-database.html">Website</a> | <a href="https://github.com/nitrite/nitrite-java">Source</a></p> <h2>Example</h2> <pre><code class="language-java">Nitrite db = Nitrite.builder() .compressed() .filePath("/tmp/test.db") .openOrCreate("user", "password"); // Create a Nitrite Collection NitriteCollection collection = db.getCollection("test"); // create a document to populate data Document doc = createDocument("firstName", "John") .put("lastName", "Doe") .put("birthDay", new Date()) .put("data", new byte[] {1, 2, 3}) .put("fruits", new ArrayList&lt;String&gt;() {{ add("apple"); add("orange"); }}) .put("note", "a quick brown fox jump over the lazy dog"); // insert the document collection.insert(doc); // update the document collection.update(eq("firstName", "John"), createDocument("lastName", "Wick")); </code></pre> <h2>Articles</h2> <ul> <li><a href="https://tkainrad.dev/posts/nitrite-embedded-nosql-db/">"Nitrite as an embedded NoSQL database for Java"</a></li> </ul> https://tedneward.github.io/Research/storage/noms/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/noms/index.html <p><a href="https://github.com/attic-labs/noms">Source</a></p> <p>Noms is a decentralized database philosophically descendant from the Git version control system.</p> <p>Like Git, Noms is:</p> <ul> <li>Versioned: By default, all previous versions of the database are retained. You can trivially track how the database evolved to its current state, easily and efficiently compare any two versions, or even rewind and branch from any previous version.</li> <li>Synchronizable: Instances of a single Noms database can be disconnected from each other for any amount of time, then later reconcile their changes efficiently and correctly.</li> </ul> <p>Unlike Git, Noms is a database, so it also:</p> <ul> <li>Primarily stores structured data, not files and directories (see: the Noms type system)</li> <li>Scales well to large amounts of data and concurrent clients</li> <li>Supports atomic transactions (a single instance of Noms is CP, but Noms is typically run in production backed by S3, in which case it is "effectively CA")</li> <li>Supports efficient indexes (see: Noms prolly-trees)</li> <li>Features a flexible query model (see: GraphQL)</li> </ul> <p>A Noms database can reside within a file system or in the cloud:</p> <ul> <li>The (built-in) NBS ChunkStore implementation provides two back-ends which provide persistence for Noms databases: one for storage in a file system and one for storage in an S3 bucket.</li> </ul> <p>Finally, because Noms is content-addressed, it yields a very pleasant programming model.</p> <p>Working with Noms is declarative. You don't INSERT new data, UPDATE existing data, or DELETE old data. You simply declare what the data ought to be right now. If you commit the same data twice, it will be deduplicated because of content-addressing. If you commit almost the same data, only the part that is different will be written.</p> https://tedneward.github.io/Research/storage/objectdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/objectdb/index.html <p><a href="https://www.objectdb.com/">Website</a> | <a href="https://www.objectdb.com/download">Download</a></p> https://tedneward.github.io/Research/storage/akavache/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/akavache/index.html <p>"Akavache is an asynchronous, persistent (i.e. writes to disk) key-value store created for writing desktop and mobile applications in C#, based on SQLite3. Akavache is great for both storing important data (i.e. user settings) as well as cached local data that expires."</p> <p>"Akavache is a library that makes common app patterns easy, and unifies caching of different object types (i.e. HTTP responses vs. JSON objects vs. images). It's built on a core key-value byte array store (conceptually similar to a Dictionary&lt;string, byte[]&gt;), and on top of that store, extensions are added to support:</p> <ul> <li>Arbitrary objects via JSON.NET</li> <li>Fetching and loading Images and URLs from the Internet</li> <li>Storing and automatically encrypting User Credentials"</li> </ul> <p><a href="https://github.com/reactiveui/Akavache">Source</a></p> https://tedneward.github.io/Research/storage/bbolt/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/bbolt/index.html <p><a href="https://github.com/etcd-io/bbolt">Source</a></p> <p>bbolt is a fork of Ben Johnson's <a href="../bolt">Bolt</a> key/value store. The purpose of this fork is to provide the Go community with an active maintenance and development target for Bolt; the goal is improved reliability and stability. bbolt includes bug fixes, performance enhancements, and features not found in Bolt while preserving backwards compatibility with the Bolt API.</p> https://tedneward.github.io/Research/storage/byoredis/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/byoredis/index.html <p><a href="https://build-your-own.org/redis/">Website</a> | <a href="https://build-your-own.org/redis/src.tgz">Source</a></p> <p>Table of Contents</p> <p>Part 1. Getting Started</p> <ol> <li><a href="https://build-your-own.org/redis/01_intro_redis.html">Introduction</a></li> <li><a href="https://build-your-own.org/redis/02_intro_sockets.html">Introduction to Sockets</a></li> <li><a href="https://build-your-own.org/redis/03_hello_cs.html">Hello Server/Client</a></li> <li><a href="https://build-your-own.org/redis/04_proto.html">Protocol Parsing</a></li> <li><a href="https://build-your-own.org/redis/05_event_loop_intro.html">The Event Loop and Nonblocking IO</a></li> <li><a href="https://build-your-own.org/redis/06_event_loop_impl.html">The Event Loop Implementation</a></li> <li><a href="https://build-your-own.org/redis/07_basic_server.html">Basic Server: get, set, del</a></li> </ol> <p>Part 2. Essential Topics</p> <ol> <li><a href="https://build-your-own.org/redis/08_hashtables.html">Data Structure: Hashtables</a></li> <li><a href="https://build-your-own.org/redis/09_serialization.html">Data Serialization</a></li> <li><a href="https://build-your-own.org/redis/10_avltree.html">The AVL Tree: Implementation and Testing</a></li> <li><a href="https://build-your-own.org/redis/11_sortedset.html">The AVL Tree and the Sorted Set</a></li> <li><a href="https://build-your-own.org/redis/12_timer.html">The Event Loop and Timers</a></li> <li><a href="https://build-your-own.org/redis/13_heap.html">The Heap Data Structure and the TTL</a></li> <li><a href="https://build-your-own.org/redis/14_thread.html">The Thread Pool and Asynchronous Tasks</a></li> </ol> <p>Appendixes</p> <ol> <li><a href="https://build-your-own.org/redis/a1_hints.html">Hints to exercises</a></li> <li><a href="https://build-your-own.org/redis/a2_reading.html">Further Reading</a></li> </ol> https://tedneward.github.io/Research/storage/cloudflared1/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cloudflared1/index.html <p><a href="https://www.cloudflare.com/lp/d1/">Website</a> |</p> <p>It uses a different data model than the traditional SQL data model, which looks and works a lot like a document-oriented database like MongoDB. We know that in a traditional SQL database, data is stored in tables consisting of rows and columns. Consider a table called “employees” with the following columns as “id”, “name”, “age”, and “department”. In contrast, CloudFare D1 stores data in the form of objects like a document database. So to translate the above example, we have an object called “Employee” that has the following properties: “id”, “name”, “age”, and “department”.</p> https://tedneward.github.io/Research/storage/couchbase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/couchbase/index.html <p><a href="https://www.couchbase.com/">Website</a></p> <p>Community Editions are open-source.</p> <hr> <h1>Couchbase Mobile</h1> <p>Formerly Couchbase Lite.</p> <p>Mobile syncing client to Couchbase.</p> <p>Source: <a href="https://github.com/couchbase/couchbase-lite-ios">iOS/macOS</a> | <a href="https://github.com/couchbase/couchbase-lite-core">C++ "core"</a> | <a href="https://github.com/couchbase/couchbase-lite-net">.NET</a> | <a href="https://github.com/couchbase/couchbase-lite-java">Java</a> | <a href="https://github.com/couchbase/couchbase-lite-android">Android</a> | <a href="https://github.com/couchbase/couchbase-lite-java-common">Java and Android common</a> | <a href="https://github.com/couchbase/couchbase-lite-android-ce">Android Community Edition</a></p> https://tedneward.github.io/Research/storage/dbm/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dbm/index.html <p>DBM allows an application program to store key-value pairs in a file and reuse them later. Each of keys and values is a string or a sequence of bytes. The key of each record must be unique within the database and a value is associated to it. You can retrieve a stored record with its key very quickly. Thanks to simple structure of DBM, its performance can be extremely high.</p> <h2>Implementations</h2> <ul> <li> <p>C++: <a href="https://dbmx.net/tkrzw/">Tkrzw: a set of implementations of DBM</a>: "Tkrzw is a C++ library implementing DBM with various algorithms. It features high degrees of performance, concurrency, scalability and durability. The following classes are provided.</p> <ul> <li>HashDBM : File database manager implementation based on hash table.</li> <li>TreeDBM : File database manager implementation based on B+ tree.</li> <li>SkipDBM : File database manager implementation based on skip list.</li> <li>TinyDBM : On-memory database manager implementation based on hash table.</li> <li>BabyDBM : On-memory database manager implementation based on B+ tree.</li> <li>CacheDBM : On-memory database manager implementation with LRU deletion.</li> <li>Std(Hash|Tree)DBM : On-memory DBM implementations using std::unordered_map and std::map.</li> <li>(Poly|Shard)DBM : Polymorphic and sharding database manager adapters.</li> <li>AsyncDBM : Asynchronous database manager adapter.</li> <li>(File|Mem)Index : Secondary index implementations.</li> </ul> <p>"You can use some bridging interfaces to use Tkrzw in other programming languages than C++. Currently, C, Java, Python, Ruby, and Go interfaces are provided. The C interface is bundled in the main package. The other interfaces are packaged separately."</p> </li> <li> <p>C++: <a href="https://dbmx.net/kyotocabinet/">Kyoto Cabinet</a></p> </li> <li>C: <a href="https://dbmx.net/tokyocabinet/">Tokyo Cabinet</a></li> </ul> https://tedneward.github.io/Research/storage/dicedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dicedb/index.html <p><a href="https://dicedb.io/">Website</a> | <a href="https://github.com/dicedb/dice">Source</a></p> <p>Docker: <code>docker run -p 7379:7379 dicedb/dicedb</code></p> <ul> <li>open source under GNU AGPL 3.0</li> <li>Redis-compliant - a drop-in (36.4%) replacement of Redis (here's why)</li> <li>Reactive - native support for query subscriptions and push</li> <li>Scalable - scales out and scales in with load *</li> <li>Highly available - failovers and durability *</li> <li>Unified cache - multi-tenant and shared everything architecture *</li> <li>Optimized for modern hardware - multi-threaded with shared-nothing architecture *</li> </ul> <p>and we aim to be the high-performance caching backbone for modern architecture.</p> https://tedneward.github.io/Research/storage/dragonflydb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dragonflydb/index.html <p><a href="https://www.dragonflydb.io/">Website</a> | <a href="https://github.com/dragonflydb/dragonfly">Source</a></p> <p>Claims to be API-compatible with Redis or memcached.</p> <p>Single binary.</p> https://tedneward.github.io/Research/storage/dynomite/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dynomite/index.html <p>"Dynomite, inspired by <a href="http://www.allthingsdistributed.com/files/amazon-dynamo-sosp2007.pdf">Dynamo whitepaper</a>, is a thin, distributed dynamo layer for different storage engines and protocols. Currently these include Redis and Memcached. Dynomite supports multi-datacenter replication and is designed for high availability.</p> <p>"The ultimate goal with Dynomite is to be able to implement high availability and cross-datacenter replication on storage engines that do not inherently provide that functionality. The implementation is efficient, not complex (few moving parts), and highly performant."</p> <p><a href="https://github.com/Netflix/dynomite">Source</a></p> https://tedneward.github.io/Research/storage/eventstoredb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/eventstoredb/index.html <p><a href="https://www.eventstore.com/eventstoredb">Website</a> | <a href="https://github.com/EventStore/EventStore">Source</a></p> https://tedneward.github.io/Research/storage/faunadb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/faunadb/index.html <p>"The database built for serverless, featuring native GraphQL. Add a full-featured global datastore to your apps in minutes. Access effortlessly from the browser and from mobile. Never again worry about data correctness, capacity, redundancy, latency, and availability. FaunaDB is a global serverless database that gives you ubiquitous, low latency access to app data, without sacrificing data correctness and scale. It eliminates layers of app code for manually handling data anomalies, security, and scale, creating a friendlier dev experience for you and better app experience for your users. Relational, document and graph access to the same set of data using modern APIs such as GraphQL that are easy to work with. No cold starts, and no additional middleware necessary. 100% ACID, with globally consistent reads and writes without compromising latency and scale. Trust your app data. Eliminate unnecessary code. Data automatically replicated across global regions instantly, correctly. Requests are routed automatically to the region closest to the user for low-latency reads and writes ensuring snappiest possible app experience. Start risk-free with one of the most generous free tiers available. Throttling-free transparent scaling, so your app never encounters unplanned capacity hiccups or outages. Eliminate complexity with built-in authorization that is easy to use, yet powerful. Lockdown data with a rich, attribute-based security model. Never leave your data unprotected. Sign up and go. Never again deal with database provisioning, clustering, backups, and maintenance. Spans multiple clouds so you never see downtime. It just works."</p> <p><a href="https://fauna.com/">Website</a> | No Github?</p> https://tedneward.github.io/Research/storage/flyway/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/flyway/index.html <p>Evolve your database schema easily and reliably across all your instances | <a href="https://flywaydb.org">Website</a> | <a href="https://github.com/flyway/flyway">Github</a></p> https://tedneward.github.io/Research/storage/graphengine/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/graphengine/index.html <p><a href="https://www.graphengine.io/">Website</a> | <a href="https://github.com/Microsoft/GraphEngine">Source</a></p> <p>The distributed RAM store provides a globally addressable high-performance key-value store over a cluster of machines. Through the RAM store, GE enables the fast random data access power over a large distributed data set.</p> <p>The capability of fast data exploration and distributed parallel computing makes GE a natural large graph processing platform. GE supports both low-latency online query processing and high-throughput offline analytics on billion-node large graphs.</p> <h4>Strongly-typed RAM Store</h4> <ul> <li><strong>Schema Matters.</strong> Schema does matter when we need to process data efficiently. Strongly-typed data modeling is crucial for compact data storage, fast data access, and clear data semantics.</li> <li><strong>One Byte Counts.</strong> GE is good at managing billions of run-time objects of varied sizes. One byte counts as the number of objects goes large. GE provides fast memory allocation and efficient memory reallocation with high memory utilization ratios.</li> <li><strong>GE fits in where cache is not good enough.</strong> GE makes use of RAM to speed up the data access as well as the computation. RAM is directly used as run-time storage instead of just being used for serving static resources.</li> </ul> <h4>General Computation Engine</h4> <p>GE is also a flexible computation engine powered by declarative message passing. GE is for you, if you are building a system that needs to perform fine-grained user-specified server-side computation.</p> <p>From the perspective of graph computation, GE is not a graph system specifically optimized for a certain graph operation. Instead, with its built-in data and computation modeling capability, we can develop graph computation modules with ease. In other words, GE can easily morph into a system supporting a specific graph computation.</p> <h4>General-purpose Large Scale Graph Processing Engine</h4> <p>GE is a graph data management system. It is designed to manage real-life graphs with rich associated data instead of just graph topology.</p> <p>On the one hand, it addresses the grand random data access challenge of graph computation at the bottom layer. On the other hand, it addresses the diversity issue of both graph data and graph computation by allowing users to freely specify their own graph models as well as the computation paradigms.</p> https://tedneward.github.io/Research/storage/hypergraphdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/hypergraphdb/index.html <p><a href="http://www.hypergraphdb.org/">Website</a> | <a href="https://github.com/hypergraphdb/hypergraphdb">Source</a></p> <h3>Example: Hello World</h3> <pre><code>HyperGraph graph = new HyperGraph("/path/to/workdir/bje"); String hello = graph.get(graph.add("Hello World")); System.out.println(hello.toLowerCase()); graph.close(); </code></pre> <ul> <li>graph.add returns something that graph.get consumes - HGHandle, that's how atoms are referred to.</li> <li>graph.get returns a String object with working toLowerCase method - this shows that hypergraphDB is also an object-oriented database.</li> <li>if you manage several hypergraph instances, use HGEnvironment.get(location).</li> <li>do not forget to close after usage, because that might lead to corruption. It's recommended to wrap your code in a try block, and ensure graph.close() in a finally block.</li> </ul> <h3>Example: Store/retrieve data</h3> <pre><code>String someObject = "Lorem ipsum"; HGHandle handle1 = graph.add(someObject); System.out.println(graph.get(handleToSomeObject)); System.out.println(((String) graph.get(handle1)).toUpperCase()); </code></pre> <ul> <li>graph.add(...) returnes a "handle" with which you can access the datum directly (as seen in graph.get a line below). A handle generally is an auto-generated database ID.</li> <li>using add/get to store and retrieve data is not necessarily the typical way hypergraphdb is used because: <ul> <li>graph.get() requires you to know the handle. This is often not the case, especially when you want data that you did not store just moments ago (hence most of the time! :-) ). Therefore most accesses typically happen by querying.</li> <li>If graph.add(someObject) is called more than once (for example by accident, each time you run a given program), you would end up with duplicates that can be disturbing when querying.</li> </ul> </li> <li>the second println demonstrate that we get a fully functional java object, in that case, it has a working toUpperCase method. This shows that hypergraphDB is not only a graph-database, but also a full object-oriented database.</li> </ul> <h3>Example: Store and retrieve data without knowing the handle - querying basics</h3> <p>Querying is done conveniently by using the static helper class "hg":</p> <pre><code>import org.hypergraphdb.HGQuery.hg.*; </code></pre> <p>We stored only one element in our database, so we would immediately find "Lorem ipsum" simply by querying for type String:</p> <pre><code>System.out.println(hg.getOne(graph, hg.type(String.class))); -&gt; "Lorem ipsum"; </code></pre> <p>hg.getOne returns you any one (of possibly many) matching result as a ready-to-use object, just as did graph.get. But often we have several items that match a certain criteria. We get can get those packed in a List, with hg.getAll: In order to have two atoms, we just add the same object as above:</p> <pre><code>HGHandle handle2 = graph.add(someObject); </code></pre> <p>and then query for all Strings:</p> <pre><code>for (Object s : hg.getAll(graph, hg.type(String.class))) System.out.println(s); -&gt; Lorem ipsum Lorem ipsum </code></pre> <p>We get two results here, that happen to be distinct duplicate copies of the same data (we prove that later).</p> <p>As you see, querying is generally used in one of this ways:</p> <pre><code>hg.getOne (graphInstance, QueryCondition); // -&gt; any one matching object. hg.getAll (graphInstance, QueryCondition); // -&gt; all matching objects as a List. </code></pre> <p>where "QueryCondition" in our example is hg.type(String.class), but of course there is more.</p> <h3>Query</h3> <p>Sometimes you need handles and also you do not want to dereference and deserialize all results of a query into memory. To query by returning handles is easy. It's the same as with getOne/getAll, but instead of h.getOne you use hg.findOne. Instead of hg.getAll, use hg.findAll.</p> <p>We use this to confirm that we created actual duplicate atoms in the "Lorem ipsum" example above. We printout the handles, and check for equality with the handles obtained before:</p> <pre><code>for (Object s : hg.findAll(graph, hg.type(String.class))) { System.out.println(s); System.out.println((s.equals(handle1) || s.equals(handle2))); } -&gt; 259b3dbd-4e4f-4566-b850-1029f99e6d1b true dceadb0c-318b-4249-917a-559d2f077fcc true </code></pre> <h3>Store data uniquely</h3> <p>How to make sure that a given data is stored only once, even when -by accident or not- the data is stored twice?</p> <pre><code>String object2 = "dolor sit amet"; HGHandle noDup1 = hg.assertAtom(graph, object2); HGHandle noDup2 = hg.assertAtom(graph, object2); //trying hard to duplicate System.out.println("Are those two handles duplicates, i.e. two distinct handles? : " + (!noDup1.equals(noDup2))); -&gt; "Are those two handles duplicates, i.e. two distinct handles? : false" </code></pre> <p>Note that logically there is a cost associated with checking if a given datum already exists. If you don't need unique atoms, graph.add is faster.</p> <h3>Create Links and query for Links</h3> <p>Till now there was nothing graph, only object-oriented database functionality. We also did not do particularly interesting queries.</p> <p>Let's make a link and query for it:</p> <pre><code>HGHandle duplicateLink = graph.add(new HGPlainLink(handle1, handle2)); List&lt;HGHandle&gt; dupsList = hg.findAll(graph, hg.link(handle1, handle2)); System.out.println("querying for link returned that duplicate Link? :" + dupsList.contains(duplicateLink)); =&gt; querying for link returned that duplicate Link? :true </code></pre> https://tedneward.github.io/Research/storage/infinitegraph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/infinitegraph/index.html <p><a href="https://objectivity.com/infinitegraph/">Website</a> |</p> https://tedneward.github.io/Research/storage/jena/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/jena/index.html <p><a href="https://jena.apache.org/">Website</a> | <a href="https://gitbox.apache.org/repos/asf?p=jena.git">Source</a> or <a href="https://github.com/apache/jena">Github Mirror</a></p> https://tedneward.github.io/Research/storage/kdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/kdb/index.html <p>kdb+ is</p> <ul> <li>a high-performance cross-platform historical timeseries columnar database</li> <li>an in-memory compute engine</li> <li>a realtime streaming processor</li> <li>an expressive query and programming language called q</li> </ul> <p>kdb+ is a database. You can use it through interfaces such as ODBC, or from Python, but its power and performance are best accessed through its own language, q.</p> <p>Q is a general-purpose programming language. You can write programs for anything in q.</p> <p><a href="http://www.kx.com/kdb-plus.php">Website</a></p> https://tedneward.github.io/Research/storage/lensvm/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/lensvm/index.html <p><a href="https://docs.source.network/lensvm">Website</a> | <a href="https://github.com/sourcenetwork/lens">Source</a></p> https://tedneward.github.io/Research/storage/litedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/litedb/index.html <p><a href="https://www.litedb.org/">Website</a> | <a href="https://github.com/mbdavid/litedb">Source</a></p> <p>Features:</p> <ul> <li>Portable UWP and Xamarin iOS/Android</li> <li>ACID transactions</li> <li>Single datafile (like SQLite)</li> <li>Recovery data in writing failure (WAL mode)</li> <li>Map your POCO class to BsonDocument</li> <li>Fluent API for custom mapping</li> <li>Cross collections references (DbRef)</li> <li>Store files and stream data (like GridFS in MongoDB)</li> <li>LINQ query support</li> </ul> <p>Nearly 1-for-1 feature parity with any Mongo-esque document database.</p> <p>Quick example:</p> <pre><code class="language-cs">// Create your POCO class entity public class Customer { public int Id { get; set; } public string Name { get; set; } public string[] Phones { get; set; } public bool IsActive { get; set; } } // Open database (or create if doesn't exist) using(var db = new LiteDatabase(@"C:\Temp\MyData.db")) { // Get a collection (or create, if doesn't exist) var col = db.GetCollection&lt;Customer&gt;("customers"); // Create your new customer instance var customer = new Customer { Name = "John Doe", Phones = new string[] { "8000-0000", "9000-0000" }, IsActive = true }; // Insert new customer document (Id will be auto-incremented) col.Insert(customer); // Update a document inside a collection customer.Name = "Jane Doe"; col.Update(customer); // Index document using document Name property col.EnsureIndex(x =&gt; x.Name); // Use LINQ to query documents (filter, sort, transform) var results = col.Query() .Where(x =&gt; x.Name.StartsWith("J")) .OrderBy(x =&gt; x.Name) .Select(x =&gt; new { x.Name, NameUpper = x.Name.ToUpper() }) .Limit(10) .ToList(); // Let's create an index in phone numbers (using expression). It's a multikey index col.EnsureIndex(x =&gt; x.Phones); // and now we can query phones var r = col.FindOne(x =&gt; x.Phones.Contains("8888-5555")); } </code></pre> <p>Custom class mapping</p> <pre><code class="language-cs">BsonMapper.Global.RegisterType&lt;DateTimeOffset&gt; ( serialize: obj =&gt; { var doc = new BsonDocument(); doc["DateTime"] = obj.DateTime.Ticks; doc["Offset"] = obj.Offset.Ticks; return doc; }, deserialize: doc =&gt; new DateTimeOffset(doc["DateTime"].AsInt64, new TimeSpan(doc["Offset"].AsInt64)) ); </code></pre> <p>File (binary) storage</p> <pre><code class="language-cs">// Get file storage with Int Id var storage = db.GetStorage&lt;int&gt;(); // Upload a file from file system to database storage.Upload(123, @"C:\Temp\picture-01.jpg"); // And download later storage.Download(123, @"C:\Temp\copy-of-picture-01.jpg"); </code></pre> https://tedneward.github.io/Research/storage/mariadb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/mariadb/index.html <p><a href="https://mariadb.com/">Website</a> | <a href="https://github.com/mariadb-developers">Source</a></p> https://tedneward.github.io/Research/storage/metadata-connect/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/metadata-connect/index.html <p><a href="https://objectivity.com/metadata-connect/">Website</a></p> https://tedneward.github.io/Research/storage/arangodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/arangodb/index.html <p><a href="https://www.arangodb.com/">Website</a> | <a href="https://github.com/arangodb/arangodb">Source</a></p> <p>Provides its own query language (AQL). Compares itself against <a href="../mongodb">MongoDB</a>, <a href="../neo4j">Neo4j</a>, and <a href="../cassandra">Cassandra</a>.</p> https://tedneward.github.io/Research/storage/berkeleydb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/berkeleydb/index.html <p><a href="https://docs.oracle.com/cd/E17076_04/html/index.html">Website</a> <a href="https://libdb.org/">Older website</a> | <a href="https://github.com/berkeleydb/libdb">Source</a></p> https://tedneward.github.io/Research/storage/cassandra/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cassandra/index.html <p><a href="https://cassandra.apache.org/">Website</a> | <a href="https://gitbox.apache.org/repos/asf/cassandra.git">Source</a></p> <p><a href="https://cassandra.apache.org/doc/latest/">Docs</a>:</p> <ul> <li><a href="https://cassandra.apache.org/doc/latest/cassandra/architecture/index.html">Architecture</a></li> <li><a href="https://cassandra.apache.org/doc/latest/cassandra/data_modeling/index.html">Data Modeling</a> <ul> <li><a href="https://www.datastax.com/blog/basic-rules-cassandra-data-modeling">"Basic Rules of Cassandra Data Modeling"</a></li> </ul> </li> <li><a href="https://cassandra.apache.org/doc/latest/cassandra/cql/index.html">Cassandra Query Language</a></li> <li><a href="https://cassandra.apache.org/doc/latest/cassandra/tools/index.html">Tools</a></li> <li><a href="https://cassandra.apache.org/_/native_protocol.html">Native Protocols</a></li> </ul> https://tedneward.github.io/Research/storage/cockroachdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cockroachdb/index.html <p>"A relational database for next generation, cloud-native applications. CockroachDB offers the familiarity and power of SQL with the comfort of your existing ORMs. Automated sharding ensures great performance as you scale. CockroachDB offers a multi-region database that operates with single-region latency and ensures data is always available and an RPO of zero.</p> <p><a href="https://www.cockroachlabs.com/">Website</a> | <a href="https://github.com/cockroachdb/cockroach">Github</a></p> https://tedneward.github.io/Research/storage/couchdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/couchdb/index.html <p><a href="https://couchdb.apache.org/">Website</a> | <a href="https://github.com/apache/couchdb">Source</a> | <a href="http://docs.couchdb.com/en/latest/intro/index.html">Docs</a></p> <p><a href="http://guide.couchdb.org/">CouchDB: The Definitive Guide</a> | <a href="https://github.com/oreilly/couchdb-guide">Source</a></p> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="http://guide.couchdb.org">CouchDB: The Definitive Guide</a></li> </ul> <hr> <p><a href="https://github.com/couchapp/couchapp">CouchApps</a></p> https://tedneward.github.io/Research/storage/defradb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/defradb/index.html <p><a href="https://docs.source.network/defradb">Website</a> | <a href="https://github.com/sourcenetwork/defradb">Source</a></p> <p>DefraDB is an application-centric database that prioritizes data ownership, personal privacy, and information security. Its data model, powered by the convergence of MerkleCRDTs and the content-addressability of IPLD, enables a multi-write-master architecture. It features DQL, a query language compatible with GraphQL but providing extra convenience. By leveraging peer-to-peer infrastructure, it can be deployed nimbly in novel topologies. Access control is determined by a relationship-based DSL, supporting document or field-level policies, secured by the SourceHub infrastructure. DefraDB is a core part of the Source technologies that enable new paradigms of local-first software, edge compute, access-control management, application-centric features, data trustworthiness, and much more.<br> '</p> https://tedneward.github.io/Research/storage/ditto/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/ditto/index.html <p><a href="https://www.ditto.com/">Website</a> | <a href="https://docs.ditto.live/home/introduction">Docs</a></p> https://tedneward.github.io/Research/storage/duckdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/duckdb/index.html <p><a href="https://duckdb.org/">Website</a> | <a href="https://github.com/duckdb/duckdb">Source</a></p> <p>When to use DuckDB</p> <ul> <li>Processing and storing tabular datasets, e.g. from CSV or Parquet files</li> <li>Interactive data analysis, e.g. Joining &amp; aggregate multiple large tables</li> <li>Concurrent large changes, to multiple large tables, e.g. appending rows, adding/removing/updating columns</li> <li>Large result set transfer to client</li> </ul> <p>When to not use DuckDB</p> <ul> <li>High-volume transactional use cases (e.g. tracking orders in a webshop)</li> <li>Large client/server installations for centralized enterprise data warehousing</li> <li>Writing to a single database from multiple concurrent processes</li> <li>Multiple concurrent processes reading from a single writable database</li> </ul> <h2>Articles</h2> <p><a href="https://thenewstack.io/duckdb-in-process-python-analytics-for-not-quite-big-data/">"DuckDB: In-Process Python Analytics for Not-Quite-Big Data"</a></p> https://tedneward.github.io/Research/storage/eclipsestore/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/eclipsestore/index.html <p><a href="https://eclipsestore.io/">Website</a> | <a href="https://github.com/eclipse-store/store">Source</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li>InfoQ: <a href="https://www.infoq.com/presentations/eclipse-store/">Ultra-Fast In-Memory Database Applications with Java</a></li> </ul> https://tedneward.github.io/Research/storage/existdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/existdb/index.html <p><a href="https://exist-db.org/exist/apps/homepage/index.html">Website</a> | <a href="https://github.com/eXist-db/exist">Source</a></p> <p><a href="https://learning.oreilly.com/library/view/exist/9781449337094/">Book</a> (OReilly Safari)</p> https://tedneward.github.io/Research/storage/ferretdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/ferretdb/index.html <p><a href="https://www.ferretdb.com/">Website</a> | <a href="https://github.com/ferretdb">Source</a></p> <h2>Articles</h2> <ul> <li><a href="https://blog.ferretdb.io/building-dynamic-forms-with-formio-ferretdb/">https://blog.ferretdb.io/building-dynamic-forms-with-formio-ferretdb/</a></li> </ul> https://tedneward.github.io/Research/storage/foundationdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/foundationdb/index.html <p><a href="https://www.foundationdb.org/">Website</a> | <a href="https://github.com/apple/foundationdb/">Source</a> | <a href="https://apple.github.io/foundationdb/">Docs</a></p> <p>FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as an ordered key-value store and employs ACID transactions for all operations. It is especially well-suited for read/write workloads but also has excellent performance for write-intensive workloads. Users interact with the database using a API language binding. You can begin local development today.</p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://thenewstack.io/foundationdb-a-distributed-database-that-cant-be-killed/#">"FoundationDB: A Distributed Database that Can't Be Killed"</a></li> </ul> <h1>Data Modeling</h1> <p>FoundationDB's core provides a simple data model coupled with powerful transactions. This combination allows building richer data models and libraries that inherit the scalability, performance, and integrity of the database. The goal of data modeling is to design a mapping of data to keys and values that enables effective storage and retrieval. Good decisions will yield an extensible, efficient abstraction. This document covers the fundamentals of data modeling with FoundationDB.</p> <ul> <li>For general guidance on application development using FoundationDB, see :doc:<code>developer-guide</code>.</li> <li>For detailed API documentation specific to each supported language, see :doc:<code>api-reference</code>.</li> </ul> <h1>The core data model</h1> <p>FoundationDB's core data model is an ordered key-value store. Also known as an ordered associative array, map, or dictionary, this is a common data structure composed of a collection of key-value pairs in which all keys are unique. Starting with this simple model, an application can create higher-level data models by mapping their elements to individual keys and values.</p> <p>In FoundationDB, both keys and values are simple byte strings. Apart from storage and retrieval, the database does not interpret or depend on the content of values. In contrast, keys are treated as members of a total order, the lexicographic order over the underlying bytes, in which keys are sorted by each byte in order. For example:</p> <ul> <li><code>'0'</code> is sorted before <code>'1'</code></li> <li><code>'apple'</code> is sorted before <code>'banana'</code></li> <li><code>'apple'</code> is sorted before <code>'apple123'</code></li> <li>keys starting with <code>'mytable\'</code> are sorted together (e.g. <code>'mytable\row1'</code>, <code>'mytable\row2'</code>, ...)</li> </ul> <p>The ordering of keys is especially relevant for range operations. An application should structure keys to produce an ordering that allows efficient data retrieval with range reads.</p> <p>.. _encoding-data-types:</p> <h1>Encoding data types</h1> <p>Because keys and values in FoundationDB are always byte strings, an application developer must serialize other data types (e.g., integers, floats, arrays) before storing them in the database. For values, the main concerns for serialization are simply CPU and space efficiency. For keys, there's an additional consideration: it's often important for keys to preserve the order of the data types (whether primitive or composite) they encode. For example:</p> <h2>Integers</h2> <ul> <li>The standard tuple layer provides an order-preserving, signed, variable length encoding.</li> <li>For positive integers, a big-endian fixed length encoding is order-preserving.</li> <li>For signed integers, a big-endian fixed length two's-complement encoding with the most significant (sign) bit inverted is order-preserving.</li> </ul> <h2>Unicode strings</h2> <ul> <li>For unicode strings ordered lexicographically by unicode code point, use UTF-8 encoding. (This approach is used by the tuple layer.)</li> <li>For unicode strings ordered by a particular collation (for example, a case insensitive ordering for a particular language), use an appropriate string collation transformation and then apply UTF-8 encoding. Internationalization or "locale" libraries in most environments and programming languages provide a string collation transformation, for example <code>C &lt;http://pubs.opengroup.org/onlinepubs/7908799/xsh/wcsxfrm.html&gt;</code><em>, <code>C++ &lt;http://www.cplusplus.com/reference/std/locale/collate/transform/&gt;</code></em>, <code>Python &lt;http://docs.python.org/py3k/library/locale.html#locale.strxfrm&gt;</code><em>, <code>Ruby &lt;https://github.com/ninjudd/icunicode#readme&gt;</code></em>, <code>Java &lt;http://docs.oracle.com/javase/1.5.0/docs/api/java/text/Collator.html#getCollationKey(java.lang.String)&gt;</code><em>, the <code>ICU &lt;http://icu-project.org/apiref/icu4c/classCollator.html#ae524fd43a06d4429e2c76bef35874d4c&gt;</code></em> library, etc. Usually the output of this function is a unicode string, which needs to be further encoded in a code-point ordered encoding such as UTF-8 to get a byte string.</li> </ul> <h2>Floating point numbers</h2> <p>The tuple layer provides an order-preserving, signed, fixed length encoding for both single- and double-precision floating point<br> numbers based off of the IEEE big-endian encoding with some modifications to make it correctly ordered. Within this representation,<br> -0 and +0 are not equal and negative NaN values will sort before all non-NaN values and positive NaN values will sort after<br> all non-NaN values. Otherwise, the representation is consistent with the mathematical ordering.</p> <h2>Composite types</h2> <p>An application's data is often represented using composite types, such as structures or records with multiple fields. It's very useful for the application to use <em>composite</em> keys to store such data. In FoundationDB, composite keys can be conveniently represented as <em>tuples</em> that are mapped to individual keys for storage.</p> <p>.. note:: For the purpose of illustration, we'll use the FoundationDB's Python language binding, including the :py:func:<code>@fdb.transactional &lt;fdb.transactional&gt;</code> decorator described in :doc:<code>api-python</code>. The design patterns illustrated are applicable to all of the :doc:<code>languages &lt;api-reference&gt;</code> supported by FoundationDB.</p> <p>.. _data-modeling-tuples:</p> <h1>Tuples</h1> <p>FoundationDB's keys are ordered, making tuples a particularly useful tool for data modeling. FoundationDB provides a :ref:<code>tuple layer &lt;api-python-tuple-layer&gt;</code> (available in each language binding) that encodes tuples into keys. This layer lets you store data using a tuple like <code>(state, county)</code> as a key. Later, you can perform reads using a prefix like <code>(state,)</code>. The layer works by preserving the natural ordering of the tuples.</p> <p>You could implement a naive encoding of tuples of strings into keys by using a tab character as a simple delimiter. You could do this with the following Python code::</p> <pre><code>def tuple_to_key_with_tab(tup): return '\t'.join(str(i) for i in tup) # Example: Order first by state, then by county @fdb.transactional def set_county_population(tr, state, county, pop): tr[tuple_to_key_with_tab((state, county))] = str(pop) </code></pre> <p>In this example, population figures for the United States are stored using keys formed from the tuple of state and county.</p> <p>Of course, this encoding would only work if all the bytes in the individual keys in the tuple were greater than the delimiter byte. Therefore, FoundationDB's built-in tuple layer implements a more robust encoding supporting elements of various data types: byte strings, unicode strings, signed integers, floating-point numbers, booleans, UUIDs, null values, and nested tuples.</p> <p>.. note:: The tuple layer's encoding is compatible between languages, although some languages are limited in what data types they support. For language-specific documentation of the tuple layer, see the corresponding :doc:<code>api-reference</code> documentation.</p> <p>Because of its ordering of keys, FoundationDB supports efficient range reads on any set of keys that share a prefix. The tuple layer preserves the ordering of tuples sorted by element from left to right; as a result, the leftmost elements of a tuple will always represent a prefix in keyspace and can be used for range reads. A basic principle of data modeling with the tuple layer is to order tuple elements to facilitate such range reads. The examples below illustrate this principle.</p> <p>Sometimes data attributes will have a natural order of containment imposed by your domain. A common example is geographic attributes, such as state and county in the Unites States. By constructing keys from tuples of the form <code>(state, county)</code>, where state is the first tuple element, all data for states will be stored in an adjacent range of keys. This ordering allows you to retrieve the populations for all counties in a given state with a single range read. You could use the tuple layer with the following functions::</p> <pre><code>@fdb.transactional def set_county_population(tr, state, county, pop): tr[fdb.tuple.pack((state, county))] = str(pop) @fdb.transactional def get_county_populations_in_state(tr, state): return [int(pop) for k, pop in tr[fdb.tuple.range((state,))]] </code></pre> <p>Date/timestamp attributes form another example with a natural containment order. If you have attributes of year, month, day, hour, minute, and/or second, you can order them from larger to smaller units in your keys. As a result, you'll be able to retrieve temporally contiguous data with range reads, as above.</p> <h1>A few simple models</h1> <p>Let's begin with a few examples of simple data models built on tuples with :ref:<code>subspaces &lt;developer-guide-sub-keyspaces&gt;</code>.</p> <h2>Arrays</h2> <p>You can easily map arrays to the key-value store using tuples. To model a named, one-dimensional array, you can construct a key for each array element using the array name as the subspace and the array index as the second tuple element.</p> <p>For example, suppose you have a <code>'temps2012'</code> array containing a year's worth of daily temperature averages. The temperatures are indexed by an integer ranging from 1 to 365 representing the day. Your keys would then be constructed from tuples of the form <code>('temps2012', day)</code>.</p> <p>To set and get array elements with this technique, you can use Python functions such as::</p> <pre><code>@fdb.transactional def array_set(tr, array_space, index, value): tr[array_space[index]] = str(value) @fdb.transactional def array_get(tr, array_space, index): return tr[array_space[index]] temp_array = Subspace(('temps2012',)) @fdb.transactional def add_temp(tr, day, temp): array_set(tr, temp_array, day, temp) @fdb.transactional def get_temp(tr, day): val = array_get(tr, temp_array, day) if val.present(): return int(val) else: return None </code></pre> <p>This approach has a few nice properties:</p> <ul> <li>It can be extended to multidimensional arrays simply by adding additional array indexes to the tuples.</li> <li>Unassigned elements consume no storage, so sparse arrays are stored efficiently.</li> </ul> <p>The tuple layer makes these properties easy to achieve, and most well-designed data models using tuples will share them.</p> <p>An array can only have a single value for each index. Likewise, the key-value store can only have a single value for each key. The simple mapping above takes advantage of this correspondence to store the array value as a physical value. In contrast, some data structures are designed to store multiple values. In these cases, data models can store the logical values within the key itself, as illustrated next.</p> <h2>Multimaps</h2> <p>A multimap is a generalization of an associative array in which each key may be associated with multiple values. Multimaps are often implemented as associative arrays in which the values are sets rather than primitive data types.</p> <p>Suppose you have a multimap that records student enrollment in classes, with students as keys and classes as values. Each student can be enrolled in more than one class, so you need to map the key-value pairs of the multimap (with their multiple values) to the database.</p> <p>A simple approach is to use the multimap name (say, <code>'enroll'</code>) as the subspace and construct a key from a tuple of the form <code>('enroll', student, class_name)</code> for each class in which a student is enrolled. Each class will generate a unique key, allowing as many classes as needed. Moreover, all the data in the multimap will be captured in the key, so you can just use an empty string for its value. Using this approach, you can add a class for a student or get all the student's classes with the Python functions::</p> <pre><code>@fdb.transactional def multi_set(tr, multi_space, index, value): tr[multi_space.pack((index, value))] = '' @fdb.transactional def multi_get(tr, multi_space, index): pairs = tr[multi_space.range((index,))] return [multi_space.unpack(k)[-1] for k, v in pairs] @fdb.transactional def multi_is_element(tr, multi_space, index, value): val = tr[multi_space.pack((index, value))] return val.present() enroll_space = Subspace(('enroll',)) @fdb.transactional def add_class(tr, student, class_name): multi_set(tr, enroll_space, student, class_name) @fdb.transactional def get_classes(tr, student): return multi_get(tr, enroll_space, student) </code></pre> <p>The <code>range()</code> method in :py:func:<code>multi_get</code> returns all keys in the subspace that encode tuples with the specified tuple as a prefix. The <code>[-1]</code> extracts the last element of the tuple unpacked from the key, which in this case will encode a class.</p> <p>As this model for multimaps illustrates, data that is treated as a value at one level may be mapped to a key in the database. (The reverse may also occur, as shown in the discussion of indirection below.) Data modeling in FoundationDB is not dictated by how your data is represented in your programming language.</p> <p>.. _data-modeling-tables:</p> <h2>Tables</h2> <p>You can easily use tuples to store data in tabular form with rows and columns. The simplest data model for a table is to make each cell in the table a key-value pair. To do this, you construct a key from a tuple containing the row and column identifiers. As with the array model, unassigned cells in tables constructed using this technique will consume no storage, so sparse tables can be stored efficiently. As a result, a table can safely have a very large number of columns.</p> <p>You can make your model row-oriented or column-oriented by placing either the row or column first in the tuple, respectively. Because the lexicographic order sorts tuple elements from left to right, access is optimized for the element placed first. Placing the row first makes it efficient to read all the cells in a particular row; reversing the order makes reading a column more efficient.</p> <p>Using the table name as the subspace, we could implement the common row-oriented version in Python as follows::</p> <pre><code>@fdb.transactional def table_set_cell(tr, table_space, row, column, value): tr[table_space.pack((row, column))] = str(value) @fdb.transactional def table_get_cell(tr, table_space, row, column): return tr[table_space.pack((row, column))] @fdb.transactional def table_set_row(tr, table_space, row, cols): del tr[table_space.range((row,))] for c, v in cols.iteritems(): table_set_cell(tr, table_space, row, c, v) @fdb.transactional def table_get_row(tr, table_space, row): cols = {} for k, v in tr[table_space.range((row,))]: _, c = table_space.unpack(k) cols[c] = v return cols </code></pre> <h2>Versionstamps</h2> <p>A common data model is to index your data with a sequencing prefix to allow log scans or tails of recent data. This index requires a unique, monotonically increasing value, like an AUTO_INCREMENT PRIMARY KEY in SQL. This could be implemented at the client level by reading the value for conflict checks before every increment. A better solution is the versionstamp, which can be generated at commit-time with no read conflict ranges, providing a unique sequence ID in a single conflict-free write.</p> <p>Versioning commits provides FoundationDB with MVCC guarantees and transactional integrity. Versionstamps write the transaction's commit version as a value to an arbitrary key as part of the same transaction, allowing the client to leverage the version's unique and serial properties. Because the versionstamp is generated at commit-time, the versionstamped key cannot be read in the same transaction that it is written, and the versionstamp's value will be unknown until the transaction is committed. After the transaction is committed, the versionstamp can be obtained.</p> <p>The versionstamp guarantees uniqueness and monotonically increasing values for the entire lifetime of a single FDB cluster. This is even true if the cluster is restored from a backup, as a restored cluster will begin at a higher version than when the backup was taken. Special care must be taken when moving data between two FoundationDB clusters containing versionstamps, as the differing cluster versions might break the monotonicity.</p> <p>There are two concepts of versionstamp depending on your context. At the fdb_c client level, or any binding outside of the Tuple layer, the 'versionstamp' is 10 bytes: the transaction's commit version (8 bytes) and transaction batch order (2 bytes). The user can manually add 2 additional bytes to provide application level ordering. The tuple layer provides a useful api for getting and setting both the 10 byte system version and the 2 byte user version. In the context of the Tuple layer, the 'versionstamp' is all 12 bytes. For examples on how to use the versionstamp in the python binding, see the :doc:<code>api-python</code> documentation.</p> <p>.. _data-modeling-entity-relationship:</p> <h1>Entity-relationship models</h1> <p>Entity-relationship models are often used to describe a database at various levels of abstraction. In this methodology, a <em>logical</em> data model consisting of entities, attributes, and relationships is defined before mapping it to a <em>physical</em> data models specifying keys and other implementation features. Entity-relationship models can be easily modeled in FoundationDB using tuples.</p> <h2>Attributes</h2> <p>Suppose you're storing entity-relationship data for users in an <code>'ER'</code> subspace. You might identify each entity with a unique identifier and define a key for each attribute with the tuple <code>('ER', entity_ID, attribute)</code>. You could then store the user's region using the Python functions::</p> <pre><code>ER_space = Subspace(('ER',)) @fdb.transactional def add_attribute_value(tr, entity_ID, attribute, value): tr[ER_space.pack((entity_ID, attribute))] = str(value) @fdb.transactional def get_attribute_value(tr, entity_ID, attribute): return tr[ER_space.pack((entity_ID, attribute))] @fdb.transactional def add_user_region(tr, user_ID, region): add_attribute_value(tr, user_ID, 'region', region) </code></pre> <h2>Relationships</h2> <p>Using the pattern we saw above with multimaps, you can store relationships and related entities as an element of the key and use an empty string as the physical value. Suppose your users can belong to one or more groups. To add a user to a group or retrieve all groups to which a user belongs, you can use the Python functions::</p> <pre><code>@fdb.transactional def add_relationship(tr, relationship, primary_key, foreign_key): tr[ER_space.pack((relationship, primary_key, foreign_key))] = '' @fdb.transactional def get_relationships(tr, relationship): return [ER_space.unpack(k)[1:] for k, v in tr.get_range_startswith(ER_space.pack((relationship,)), streaming_mode=fdb.StreamingMode.want_all)] @fdb.transactional def get_related_entities(tr, relationship, primary_key): items = tr[ER_space.range((relationship, primary_key))] return [ER_space.unpack(k)[-1] for k, v in items] @fdb.transactional def is_related_entity(tr, relationship, primary_key, foreign_key): return tr[ER_space.pack((relationship, primary_key, foreign_key))].present() @fdb.transactional def add_user_to_group(tr, user_ID, group_name): add_relationship(tr, 'belongs_to', user_ID, group_name) @fdb.transactional def get_users_groups(tr, user_ID): return get_related_entities(tr, 'belongs_to', user_ID) </code></pre> <p>You can extend this code by adding indexes for the related entities (see below) and enforcement of relationship cardinalities (one-to-many, etc.).</p> <p>.. _data-modeling-indexes:</p> <h1>Indexes</h1> <p>A common technique is to store the same data in different ways to allow efficient retrieval for multiple use cases, creating indexes. This technique is especially useful when there are many more reads than writes. For example, you may find it most convenient to store user data based on <code>user_ID</code> but sometimes need to retrieve users based on their region. An index allows this retrieval to be performed efficiently.</p> <p>An index can have a very simple tuple structure consisting of an unique subspace, the relationship being indexed, and a value: <code>(subspace_for_index, relationship, value)</code>. Placing the relationship before the value is what allows efficient retrieval of all the associated values with a single range read.</p> <p>With FoundationDB's transactions, you can easily build an index and guarantee that it stays in sync with the data: just update the index in the same transaction that updates the data.</p> <p>For example, suppose you'd like to add an index to efficiently look up users by region. You can augment the Python function :py:func:<code>add_user</code> with the index and add a new function for retrieval::</p> <pre><code>user_space = Subspace(('user',)) region_index = Subspace(('region_idx',)) @fdb.transactional def add_user(tr, user_ID, name, region): tr[user_space.pack((user_ID, region))] = str(name) tr[region_index.pack((region, user_ID))] = '' @fdb.transactional def get_users_in_region(tr, region): items = tr[region_index.range((region,))] return [region_index.unpack(k)[-1] for k, v in items] </code></pre> <p>To apply this technique to a real use case, you would add code to your update transaction to delete outdated index entries. Note that this approach lets you add as many indexes as desired by updating all the indexes in the same transaction.</p> <h1>Composite models</h1> <p>Most of the techniques we've discussed can be freely combined. Let's look at adding indexes to our basic data model for tables.</p> <p>We've already seen a way to store tabular data in a row-oriented order using table names as subspaces. You can extend this model by simultaneously storing the table in both row-oriented and column-oriented layouts, allowing efficient retrieval of either an entire row or an entire column. We'll create nested subspaces for the indexes using the subscripting syntax we saw above::</p> <pre><code>table_space = Subspace(('table',)) row_index = table_space['row_idx'] col_index = table_space['col_idx'] @fdb.transactional def table_set_cell(tr, row_index, col_index, row, column, value): tr[row_index.pack((row, column))] = str(value) tr[col_index.pack((column, row))] = str(value) @fdb.transactional def table_get_cell(tr, row_index, row, column): return tr[row_index.pack((row, column))] @fdb.transactional def table_get_row(tr, row_index, row): cols = {} for k, v in tr[row_index.range((row,))]: r, c = row_index.unpack(k) cols[c] = v return cols @fdb.transactional def table_get_col(tr, col_index, col): rows = {} for k, v in tr[col_index.range((col,))]: c, r = col_index.unpack(k) rows[r] = v return rows </code></pre> <p>.. _data-modeling-hierarchies:</p> <h1>Hierarchies</h1> <p>Many applications work with hierarchical data represented by nested dictionaries or similar composite data types. Such data is often serialized to or deserialized from a format such as JSON or XML. Looking at a hierarchical object as a tree, you can use a tuple to represent the full path to each leaf (sometimes called a "materialized path"). By storing each full path as a key, you get an index for each leaf. FoundationDB can then efficiently retrieve any individual piece of data or entire sub-tree.</p> <p>For example, suppose you have hierarchical data such as the following nested dictionaries and lists::</p> <pre><code>{'user': { 'jones': { 'friendOf': 'smith', 'group': ['sales', 'service']}, 'smith': { 'friendOf': 'jones', 'group': ['dev', 'research']}}} </code></pre> <p>To distinguish the list elements from dictionary elements and preserve the order of the lists, you can just include the index of each list element before it in the tuple. Using this technique, the data above would be converted to the following tuples::</p> <pre><code>[('user', 'jones', 'friendOf', 'smith'), ('user', 'jones', 'group', 0, 'sales'), ('user', 'jones', 'group', 1, 'service'), ('user', 'smith', 'friendOf', 'jones'), ('user', 'smith', 'group', 0, 'dev'), ('user', 'smith', 'group', 1, 'research')] </code></pre> <p>Suppose you'd like to use this representation to implement a nested keyspace, i.e., a key-value store in which values can themselves be nested dictionaries or lists. Your application receives a stream of serialized JSON objects in which different objects may contain data about the same entities, so you'd like to store the data in a common nested keyspace.</p> <p>You can deserialize the data using Python's standard <code>json</code> module, generate the corresponding set of paths as tuples, and store each tuple in a <code>'hier'</code> subspace::</p> <pre><code>import json, itertools hier_space = Subspace(('hier',)) EMPTY_OBJECT = -2 EMPTY_ARRAY = -1 def to_tuples(item): if item == {}: return [(EMPTY_OBJECT, None)] elif item == []: return [(EMPTY_ARRAY, None)] elif type(item) == dict: return [(k,) + sub for k, v in item.iteritems() for sub in to_tuples(v)] elif type(item) == list: return [(k,) + sub for k, v in enumerate(item) for sub in to_tuples(v)] else: return [(item,)] @fdb.transactional def insert_hier(tr, hier): if type(hier) == str: hier = json.loads(hier) for tup in to_tuples(hier): tr[hier_space.pack(tup)] = '' </code></pre> <p>You can then retrieve any sub-tree from the nested keyspace by giving the partial path to its root. The partial path will just be a tuple that your query function uses as a key prefix for a range read. For example, to retrieve the data for <code>'smith'</code> from the hierarchy above, you would use <code>('user', 'smith')</code>.</p> <p>The retrieved data will be a list of tuples. The final step before returning the data is to convert it back to a nested data structure::</p> <pre><code>def from_tuples(tuples): first = tuples[0] # The first tuple will tell us what kind of object we have if len(first) == 1: return first[0] # Primitive value if first == (EMPTY_OBJECT,None): return {} if first == (EMPTY_ARRAY, None): return [] # For an object or array, we need to group the tuples by their first element groups = [list(g) for k, g in itertools.groupby(tuples, lambda t:t[0])] if first[0] == 0: # array return [from_tuples([t[1:] for t in g]) for g in groups] else: # object return dict((g[0][0], from_tuples([t[1:] for t in g])) for g in groups) @fdb.transactional def get_sub_hier(tr, prefix): return from_tuples([hier_space.unpack(k) for k, v in tr[hier_space.range(prefix)]]) </code></pre> <p>.. _data-modeling-documents:</p> <h1>Documents</h1> <p>Suppose you'd like to use the above representation to implement a simple document-oriented data model. As before, your application receives serialized data in JSON, only now you'd like to store each JSON object as an independent document. To do so, you just need to ensure that each tuple created for that object is stored with a unique identifier for the document. If a <code>doc_id</code> has not already been supplied, you can randomly generate one.</p> <p>To store a path, you can construct a composite key in a <code>'doc'</code> subspace, with the <code>doc_id</code> as the next element, followed by the remainder of the path. You can store the leaf (the last element of the tuple) as the value, which enables storage of larger data sizes (see :ref:<code>data-modeling-performance-guidelines</code>)::</p> <pre><code>import random doc_space = Subspace(('doc',)) @fdb.transactional def insert_doc(tr, doc): if type(doc) == str: doc = json.loads(doc) if not 'doc_id' in doc: doc['doc_id'] = random.randint(0, 100000000) for tup in to_tuples( doc ): tr[doc_space.pack((doc['doc_id'],) + tup[:-1])] = fdb.tuple.pack((tup[-1],)) return doc['doc_id'] @fdb.transactional def get_doc(tr, doc_id): return from_tuples([doc_space.unpack(k)[1:] + fdb.tuple.unpack(v) for k, v in tr[doc_space.range((doc_id,))]]) </code></pre> <p>.. _data-modeling-indirection:</p> <h1>Indirection</h1> <p>It is sometimes beneficial to add a level of indirection to a data model. Instead of using key-value pairs to directly store application data, you can instead store a reference to that data. This approach can be used to model any data structure that would normally use references. You just need to perform any modifications to the data structure in a transaction that leaves it in a consistent state.</p> <p>Suppose you want to maintain data in a singly linked list. The application data can use a tuple structure like those of single-valued relationships. Links will be similar but will use node identifiers as their values. Here is an example of removing the next node from the list::</p> <pre><code>node_space = Subspace(('node',)) @fdb.transactional def remove_next_node(tr, node_ID): next_ID = tr[node_space.pack((node_ID, 'next'))] if next_ID != '': next_next_ID = tr[node_space.pack((next_ID, 'next'))] tr[node_space.pack((node_ID, 'next'))] = next_next_ID del tr[node_space.range((next_ID,))] </code></pre> <p>FoundationDB's transactional guarantees ensure that, even when multiple clients are concurrently modifying the same linked list, the structure will be maintained in a consistent way.</p> <p>.. _data-modeling-performance-guidelines:</p> <h1>Key and value sizes</h1> <p>How you map your application data to keys and values can have a dramatic impact on performance. Below are some guidelines to consider as you design a data model. (For more general discussion of performance considerations, see :ref:<code>developer-guide-peformance-considerations</code>.)</p> <ul> <li> <p>Structure keys so that range reads can efficiently retrieve the most frequently accessed data.</p> </li> <li> <p>If you perform a range read that is, in total, much more than 1 kB, try to restrict your range as much as you can while still retrieving the needed data.</p> </li> <li> <p>Structure keys so that no single key needs to be updated too frequently, which can cause transaction conflicts.</p> </li> <li> <p>If a key is updated more than 10-100 times per second, try to split it into multiple keys.</p> </li> <li>For example, if a key is storing a counter, split the counter into N separate counters that are randomly incremented by clients. The total value of the counter can then read by adding up the N individual ones.</li> <li> <p>Keep key sizes small.</p> </li> <li> <p>Try to keep key sizes below 1 kB. (Performance will be best with key sizes below 32 bytes and <em>cannot</em> be more than 10 kB.)</p> </li> <li>When using the tuple layer to encode keys (as is recommended), select short strings or small integers for tuple elements. Small integers will encode to just two bytes.</li> <li>If your key sizes are above 1 kB, try either to move data from the key to the value, split the key into multiple keys, or encode the parts of the key more efficiently (remembering to preserve any important ordering).</li> <li> <p>Keep value sizes moderate.</p> </li> <li> <p>Try to keep value sizes below 10 kB. (Value sizes <em>cannot</em> be more than 100 kB.)</p> </li> <li>If your value sizes are above 10 kB, consider splitting the value across multiple keys.</li> <li>If you read values with sizes above 1 kB but use only a part of each value, consider splitting the values using multiple keys.</li> <li>If you frequently perform individual reads on a set of values that total to fewer than 200 bytes, try either to combine the values into a single value or to store the values in adjacent keys and use a range read.</li> </ul> <h2>Large Values and Blobs</h2> <p>If your keys or values are much larger than the above guidelines, it may be difficult to find a data model that resizes them appropriately. Unstructured data, such as binary large objects, can be especially challenging to segment manually. In this case, a good option is to use our blob layer. See our tutorial on :doc:<code>Managing Large Values and Blobs &lt;largeval&gt;</code> for further discussion.</p> <h1>Example: Class Scheduling</h1> <p>This tutorial provides a walkthrough of designing and building a simple application in Python using FoundationDB. In this tutorial, we use a few simple data modeling techniques. For a more in-depth discussion of data modeling in FoundationDB, see <a href="https://apple.github.io/foundationdb/data-modeling.html">here</a>.</p> <h1>First steps</h1> <p>Let's begin with "Hello world."</p> <p>Open a Python interactive interpreter and import the FoundationDB API module::</p> <pre><code>$ python &gt;&gt;&gt; import fdb </code></pre> <p>Before using the API, we need to specify the API version. This allows programs to maintain compatibility even if the API is modified in future versions::</p> <pre><code>&gt;&gt;&gt; fdb.api_version(740) </code></pre> <p>Next, we open a FoundationDB database. The API will connect to the FoundationDB cluster indicated by the :ref:<code>default cluster file &lt;default-cluster-file&gt;</code>. ::</p> <pre><code>&gt;&gt;&gt; db = fdb.open() </code></pre> <p>We are ready to use the database. In Python, using the <code>[]</code> operator on the db object is a convenient syntax for performing a read or write on the database. First, let's simply write a key-value pair:</p> <pre><code>&gt;&gt;&gt; db[b'hello'] = b'world' </code></pre> <p>When this command returns without exception, the modification is durably stored in FoundationDB! Under the covers, this function creates a transaction with a single modification. We'll see later how to do multiple operations in a single transaction. For now, let's read back the data::</p> <pre><code>&gt;&gt;&gt; print 'hello', db[b'hello'] hello world </code></pre> <p>If this is all working, it looks like we are ready to start building a real application. For reference, here's the full code for "hello world"::</p> <pre><code>import fdb fdb.api_version(740) db = fdb.open() db[b'hello'] = b'world' print 'hello', db[b'hello'] </code></pre> <h1>Class scheduling application</h1> <p>Let's say we've been asked to build a class scheduling system for students and administrators. We'll walk through the design and implementation of this application. Instead of typing everything in as you follow along, look at the :ref:<code>tutorial-appendix</code> for a finished version of the program. You may want to refer to this code as we walk through the tutorial.</p> <h2>Requirements</h2> <p>We'll need to let users list available classes and track which students have signed up for which classes. Here's a first cut at the functions we'll need to implement::</p> <pre><code>available_classes() # returns list of classes signup(studentID, class) # signs up a student for a class drop(studentID, class) # drops a student from a class </code></pre> <p>.. _tutorial-data-model:</p> <h2>Data model</h2> <p>First, we need to design a :doc:<code>data model &lt;data-modeling&gt;</code>. A data model is just a method for storing our application data using keys and values in FoundationDB. We seem to have two main types of data: (1) a list of classes and (2) a record of which students will attend which classes. Let's keep attending data like this::</p> <pre><code># ('attends', student, class) = '' </code></pre> <p>We'll just store the key with a blank value to indicate that a student is signed up for a particular class. For this application, we're going to think about a key-value pair's key as a :ref:<code>tuple &lt;data-modeling-tuples&gt;</code>. Encoding a tuple of data elements into a key is a very common pattern for an ordered key-value store.</p> <p>We'll keep data about classes like this::</p> <pre><code># ('class', class_name) = seats_available </code></pre> <p>Similarly, each such key will represent an available class. We'll use <code>seats_available</code> to record the number of seats available.</p> <h2>Directories and Subspaces</h2> <p>FoundationDB includes a few tools that make it easy to model data using this approach. Let's begin by<br> opening a :ref:<code>directory &lt;developer-guide-directories&gt;</code> in the database::</p> <pre><code>import fdb fdb.api_version(740) db = fdb.open() scheduling = fdb.directory.create_or_open(db, ('scheduling',)) </code></pre> <p>The :meth:<code>create_or_open</code> method returns a :ref:<code>subspace &lt;developer-guide-sub-keyspaces&gt;</code> where we'll store our application data. Each subspace has a fixed prefix it uses when defining keys. The prefix corresponds to the first element of a tuple. We decided that we wanted <code>'attends'</code> and <code>'class'</code> as our prefixes, so we'll create new subspaces for them within the <code>scheduling</code> subspace.::</p> <pre><code>course = scheduling['class'] attends = scheduling['attends'] </code></pre> <p>Subspaces have a :meth:<code>pack</code> method for defining keys. To store the records for our data model, we can use <code>attends.pack((s, c))</code> and <code>course.pack((c,))</code>.</p> <h2>Transactions</h2> <p>We're going to rely on the powerful guarantees of transactions to help keep all of our modifications straight, so let's look at a nice way that the FoundationDB Python API lets you write a transactional function. By using a decorator, an entire function is wrapped in a transaction. Let's write the very simple <code>add_class</code> function we will use to populate the database's class list::</p> <pre><code>@fdb.transactional def add_class(tr, c): tr[course.pack((c,))] = fdb.tuple.pack((100,)) </code></pre> <p>:py:func:<code>@fdb.transactional &lt;fdb.transactional&gt;</code> is a Python decorator that makes a normal function a transactional function. All functions decorated this way <em>need to have a parameter named</em> <code>tr</code>. This parameter is passed the transaction that the function should use to do reads and writes.</p> <p>When <em>calling</em> a transactionally decorated function, however, you can pass a database instead of a transaction for the <code>tr</code> parameter. The decorator <em>automatically creates a transaction and implements a retry loop</em> to ensure that the transaction eventually commits.</p> <p>For a FoundationDB database <code>db</code>::</p> <pre><code>add_class(db, 'class1') </code></pre> <p>is equivalent to something like::</p> <pre><code>tr = db.create_transaction() while True: try: add_class(tr, 'class1') tr.commit().wait() break except fdb.FDBError as e: tr.on_error(e).wait() </code></pre> <p>If instead you pass a :class:<code>Transaction</code> for the <code>tr</code> parameter, the transaction will be used directly, and it is assumed that the caller implements appropriate retry logic for errors. This permits transactionally decorated functions to be composed into larger transactions.</p> <p>Note that by default, the operation will be retried an infinite number of times and the transaction will never time out. It is therefore recommended that the client choose a default transaction retry limit or timeout value that is suitable for their application. This can be set either at the transaction level using the <code>set_retry_limit</code> or <code>set_timeout</code> transaction options or at the database level with the <code>set_transaction_retry_limit</code> or <code>set_transaction_timeout</code> database options. For example, one can set a one minute timeout on each transaction and a default retry limit of 100 by calling::</p> <pre><code>db.options.set_transaction_timeout(60000) # 60,000 ms = 1 minute db.options.set_transaction_retry_limit(100) </code></pre> <h2>Making some sample classes</h2> <p>Let's make some sample classes and put them in the <code>class_names</code> variable. The Python <code>itertools</code> module is used to make individual classes from combinations of class types, levels, and times::</p> <pre><code>import itertools # Generate 1,620 classes like '9:00 chem for dummies' levels = ['intro', 'for dummies', 'remedial', '101', '201', '301', 'mastery', 'lab', 'seminar'] types = ['chem', 'bio', 'cs', 'geometry', 'calc', 'alg', 'film', 'music', 'art', 'dance'] times = [str(h) + ':00' for h in range(2, 20)] class_combos = itertools.product(times, types, levels) class_names = [' '.join(tup) for tup in class_combos] </code></pre> <h2>Initializing the database</h2> <p>We initialize the database with our class list::</p> <pre><code>@fdb.transactional def init(tr): del tr[scheduling.range(())] # Clear the directory for class_name in class_names: add_class(tr, class_name) </code></pre> <p>After :func:<code>init</code> is run, the database will contain all of the sample classes we created above.</p> <h2>Listing available classes</h2> <p>Before students can do anything else, they need to be able to retrieve a list of available classes from the database. Because FoundationDB sorts its data by key and therefore has efficient range-read capability, we can retrieve all of the classes in a single database call. We find this range of keys with :meth:<code>course.range</code>.::</p> <pre><code>@fdb.transactional def available_classes(tr): return [course.unpack(k)[0] for k, v in tr[course.range(())]] </code></pre> <p>In general, the :meth:<code>Subspace.range</code> method returns a Python <code>slice</code> representing all the key-value pairs starting with the specified tuple. In this case, we want all classes, so we call :meth:<code>course.range</code> with the empty tuple <code>()</code>. FoundationDB's <code>tr[slice]</code> function returns an iterable list of key-values in the range specified by the slice. We unpack the key <code>k</code> and value <code>v</code> in a comprehension. To extract the class name itself, we unpack the key into a tuple using the :meth:<code>Subspace.unpack</code> method and take the first field. (The first and second parts of the tuple, the <code>scheduling</code> and <code>course</code> subspace prefixes, are removed by the <code>unpack</code> hence the reason we take the first field of the tuple.)</p> <h2>Signing up for a class</h2> <p>We finally get to the crucial function. A student has decided on a class (by name) and wants to sign up. The <code>signup</code> function will take a student (<code>s</code>) and a class (<code>c</code>)::</p> <pre><code>@fdb.transactional def signup(tr, s, c): rec = attends.pack((s, c)) tr[rec] = b'' </code></pre> <p>We simply insert the appropriate record (with a blank value).</p> <h2>Dropping a class</h2> <p>Dropping a class is similar to signing up::</p> <pre><code>@fdb.transactional def drop(tr, s, c): rec = attends.pack((s, c)) del tr[rec] </code></pre> <p>Of course, to actually drop the student from the class, we need to be able to delete a record from the database. We do this with the <code>del tr[key]</code> syntax.</p> <h2>Done?</h2> <p>We report back to the project leader that our application is done---students can sign up for, drop, and list classes. Unfortunately, we learn that a new problem has been discovered: popular classes are being over-subscribed. Our application now needs to enforce the class size constraint as students add and drop classes.</p> <h2>Seats are limited!</h2> <p>Let's go back to the data model. Remember that we stored the number of seats in the class in the value of the key-value entry in the class list. Let's refine that a bit to track the <em>remaining</em> number of seats in the class. The initialization can work the same way. (In our example, all classes initially have 100 seats), but the <code>available_classes</code>, <code>signup</code>, and <code>drop</code> functions are going to have to change. Let's start with <code>available_classes</code>:</p> <p>.. code-block:: python<br> :emphasize-lines: 4</p> <pre><code>@fdb.transactional def available_classes(tr): return [course.unpack(k)[0] for k, v in tr[course.range(())] if fdb.tuple.unpack(v)[0]] </code></pre> <p>This is easy -- we simply add a condition to check that the value is non-zero. Let's check out <code>signup</code> next:</p> <p>.. code-block:: python<br> :emphasize-lines: 4,5,6,7,8,9</p> <pre><code>@fdb.transactional def signup(tr, s, c): rec = attends.pack((s, c)) if tr[rec].present(): return # already signed up seats_left = fdb.tuple.unpack(tr[course.pack((c,))])[0] if not seats_left: raise Exception('No remaining seats') tr[course.pack((c,))] = fdb.tuple.pack((seats_left - 1,)) tr[rec] = b'' </code></pre> <p>We now have to check that we aren't already signed up, since we don't want a double sign up to decrease the number of seats twice. Then we look up how many seats are left to make sure there is a seat remaining so we don't push the counter into the negative. If there is a seat remaining, we decrement the counter.</p> <p>Similarly, the <code>drop</code> function is modified as follows:</p> <p>.. code-block:: python<br> :emphasize-lines: 4,5</p> <pre><code>@fdb.transactional def drop(tr, s, c): rec = attends.pack((s, c)) if not tr[rec].present(): return # not taking this class tr[course.pack((c,))] = fdb.tuple.pack((fdb.tuple.unpack(tr[course.pack((c,))])[0] + 1,)) del tr[rec] </code></pre> <p>Once again we check to see if the student is signed up and if not, we can just return as we don't want to incorrectly increase the number of seats. We then adjust the number of seats by one by taking the current value, incrementing it by one, and then storing back.</p> <h2>Concurrency and consistency</h2> <p>The <code>signup</code> function is starting to get a bit complex; it now reads and writes a few different key-value pairs in the database. One of the tricky issues in this situation is what happens as multiple clients/students read and modify the database at the same time. Couldn't two students both see one remaining seat and sign up at the same time?</p> <p>These are tricky issues without simple answers---unless you have transactions! Because these functions are defined as FoundationDB transactions, we can have a simple answer: Each transactional function behaves as if it is the only one modifying the database. There is no way for a transaction to 'see' another transaction change the database, and each transaction ensures that either all of its modifications occur or none of them do.</p> <p>Looking deeper, it is, of course, possible for two transactions to conflict. For example, if two people both see a class with one seat and sign up at the same time, FoundationDB must allow only one to succeed. This causes one of the transactions to fail to commit (which can also be caused by network outages, crashes, etc.). To ensure correct operation, applications need to handle this situation, usually via retrying the transaction. In this case, the conflicting transaction will be retried automatically by the <code>@fdb.transactional</code> decorator and will eventually lead to the correct result, a 'No remaining seats' exception.</p> <h2>Idempotence</h2> <p>Occasionally, a transaction might be retried even after it succeeds (for example, if the client loses contact with the cluster at just the wrong moment). This can cause problems if transactions are not written to be idempotent, i.e. to have the same effect if committed twice as if committed once. There are generic design patterns for :ref:<code>making any transaction idempotent &lt;developer-guide-unknown-results&gt;</code>, but many transactions are naturally idempotent. For example, all of the transactions in this tutorial are idempotent.</p> <h2>More features?!</h2> <p>Of course, as soon as our new version of the system goes live, we hear of a trick that certain students are using. They are signing up for all classes immediately, and only later dropping those that they don't want to take. This has led to an unusable system, and we have been asked to fix it. We decide to limit students to five classes:</p> <p>.. code-block:: python<br> :emphasize-lines: 9,10</p> <pre><code>@fdb.transactional def signup(tr, s, c): rec = attends.pack((s, c)) if tr[rec].present(): return # already signed up seats_left = fdb.tuple.unpack(tr[course.pack((c,))])[0] if not seats_left: raise Exception('No remaining seats') classes = tr[attends.range((s,))] if len(list(classes)) == 5: raise Exception('Too many classes') tr[course.pack((c,))] = fdb.tuple.pack((seats_left - 1,)) tr[rec] = b'' </code></pre> <p>Fortunately, we decided on a data model that keeps all of the attending records for a single student together. With this approach, we can use a single range read in the <code>attends</code> subspace to retrieve all the classes that a student is signed up for. We simply throw an exception if the number of classes has reached the limit of five.</p> <h2>Composing transactions</h2> <p>Oh, just one last feature, we're told. We have students that are trying to switch from one popular class to another. By the time they drop one class to free up a slot for themselves, the open slot in the other class is gone. By the time they see this and try to re-add their old class, that slot is gone too! So, can we make it so that a student can switch from one class to another without this worry?</p> <p>Fortunately, we have FoundationDB, and this sounds an awful lot like the transactional property of atomicity---the all-or-nothing behavior that we already rely on. All we need to do is to <em>compose</em> the <code>drop</code> and <code>signup</code> functions into a new <code>switch</code> function. This makes the <code>switch</code> function exceptionally easy::</p> <pre><code>@fdb.transactional def switch(tr, s, old_c, new_c): drop(tr, s, old_c) signup(tr, s, new_c) </code></pre> <p>The simplicity of this implementation belies the sophistication of what FoundationDB is taking care of for us.</p> <p>By dropping the old class and signing up for the new one inside a single transaction, we ensure that either both steps happen, or that neither happens. The first notable thing about the <code>switch</code> function is that it is transactionally decorated, but it also calls the transactionally decorated functions <code>signup</code> and <code>drop</code>. Because these decorated functions can accept either a database or an existing transaction as the <code>tr</code> argument, the switch function can be called with a database by a simple client, and a new transaction will be automatically created. However, once this transaction is created and passed in as <code>tr</code>, the calls to <code>drop</code> and <code>signup</code> both share the same <code>tr</code>. This ensures that they see each other's modifications to the database, and all of the changes that both of them make in sequence are made transactionally when the switch function returns. This compositional capability is very powerful.</p> <p>Also note that, if an exception is raised, for example, in <code>signup</code>, the exception is not caught by <code>switch</code> and so will be thrown to the calling function. In this case, the transaction object (owned by the decorator) is destroyed, automatically rolling back all database modifications, leaving the database completely unchanged by the half-executed function.</p> <h2>Are we done?</h2> <p>Yep, we’re done and ready to deploy. If you want to see this entire application in one place plus some multithreaded testing code to simulate concurrency, look at the :ref:<code>tutorial-appendix</code>, below.</p> <h2>Deploying and scaling</h2> <p>Since we store all state for this application in FoundationDB, deploying and scaling this solution up is impressively painless. Just run a web server, the UI, this back end, and point the whole thing at FoundationDB. We can run as many computers with this setup as we want, and they can all hit the database at the same time because of the transactional integrity of FoundationDB. Also, since all of the state in the system is stored in the database, any of these computers can fail without any lasting consequences.</p> <h1>Next steps</h1> <ul> <li>See :doc:<code>data-modeling</code> for guidance on using tuple and subspaces to enable effective storage and retrieval of data.</li> <li>See :doc:<code>developer-guide</code> for general guidance on development using FoundationDB.</li> <li>See the :doc:<code>API References &lt;api-reference&gt;</code> for detailed API documentation.</li> </ul> <p>.. _tutorial-appendix:</p> <h1>Appendix: SchedulingTutorial.py</h1> <p>Here's the code for the scheduling tutorial::</p> <pre><code>import itertools import traceback import fdb import fdb.tuple fdb.api_version(740) #################################### ## Initialization ## #################################### # Data model: # ('attends', student, class) = '' # ('class', class_name) = seats_left db = fdb.open() db.options.set_transaction_timeout(60000) # 60,000 ms = 1 minute db.options.set_transaction_retry_limit(100) scheduling = fdb.directory.create_or_open(db, ('scheduling',)) course = scheduling['class'] attends = scheduling['attends'] @fdb.transactional def add_class(tr, c): tr[course.pack((c,))] = fdb.tuple.pack((100,)) # Generate 1,620 classes like '9:00 chem for dummies' levels = ['intro', 'for dummies', 'remedial', '101', '201', '301', 'mastery', 'lab', 'seminar'] types = ['chem', 'bio', 'cs', 'geometry', 'calc', 'alg', 'film', 'music', 'art', 'dance'] times = [str(h) + ':00' for h in range(2, 20)] class_combos = itertools.product(times, types, levels) class_names = [' '.join(tup) for tup in class_combos] @fdb.transactional def init(tr): del tr[scheduling.range(())] # Clear the directory for class_name in class_names: add_class(tr, class_name) #################################### ## Class Scheduling Functions ## #################################### @fdb.transactional def available_classes(tr): return [course.unpack(k)[0] for k, v in tr[course.range(())] if fdb.tuple.unpack(v)[0]] @fdb.transactional def signup(tr, s, c): rec = attends.pack((s, c)) if tr[rec].present(): return # already signed up seats_left = fdb.tuple.unpack(tr[course.pack((c,))])[0] if not seats_left: raise Exception('No remaining seats') classes = tr[attends.range((s,))] if len(list(classes)) == 5: raise Exception('Too many classes') tr[course.pack((c,))] = fdb.tuple.pack((seats_left - 1,)) tr[rec] = b'' @fdb.transactional def drop(tr, s, c): rec = attends.pack((s, c)) if not tr[rec].present(): return # not taking this class tr[course.pack((c,))] = fdb.tuple.pack((fdb.tuple.unpack(tr[course.pack((c,))])[0] + 1,)) del tr[rec] @fdb.transactional def switch(tr, s, old_c, new_c): drop(tr, s, old_c) signup(tr, s, new_c) #################################### ## Testing ## #################################### import random import threading def indecisive_student(i, ops): student_ID = 's{:d}'.format(i) all_classes = class_names my_classes = [] for i in range(ops): class_count = len(my_classes) moods = [] if class_count: moods.extend(['drop', 'switch']) if class_count &lt; 5: moods.append('add') mood = random.choice(moods) try: if not all_classes: all_classes = available_classes(db) if mood == 'add': c = random.choice(all_classes) signup(db, student_ID, c) my_classes.append(c) elif mood == 'drop': c = random.choice(my_classes) drop(db, student_ID, c) my_classes.remove(c) elif mood == 'switch': old_c = random.choice(my_classes) new_c = random.choice(all_classes) switch(db, student_ID, old_c, new_c) my_classes.remove(old_c) my_classes.append(new_c) except Exception as e: traceback.print_exc() print("Need to recheck available classes.") all_classes = [] def run(students, ops_per_student): threads = [ threading.Thread(target=indecisive_student, args=(i, ops_per_student)) for i in range(students)] for thr in threads: thr.start() for thr in threads: thr.join() print("Ran {} transactions".format(students * ops_per_student)) if __name__ == "__main__": init(db) print("initialized") run(10, 10) </code></pre> https://tedneward.github.io/Research/storage/harbour/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/harbour/index.html <p>Harbour consists of a compiler and runtime libraries with multiple UI and database backends, its own make system and a large collection of libraries and interfaces to many popular APIs.</p> <p><a href="https://harbour.github.io/">Website</a> | <a href="https://github.com/harbour/core">Source</a> | <a href="https://harbour.wiki/">Wiki</a></p> <p>Last stable release (3.0.0), in 2011.</p> <p>Transpiles to C, but also has a runtime that can execute embedded code/instructions ("PCODE"), which can be either in a binary format or a source format.</p> <h2>FAQ</h2> <h4>What is Harbour?</h4> <p>Harbour is an open-source implementation of a xBase computer language, that is compatible on any platforms supporting C based applications, meaning MS Windows, Mac OS, Linux, Unix, iOS, and can generate 32-bit and 64-bit applications and libraries.<br> xBase is the generic term for all programming languages that derive from the original dBASE programming language and database formats (<a href="https://en.wikipedia.org/wiki/XBase">https://en.wikipedia.org/wiki/XBase</a>) that was first released in 1979. Harbour is classified as a 4GL language since its compiler generates pure C source code. Harbour-generated applications are self-contained, meaning they do not require external runtime libraries. Harbour programs can be compiled to use an embedded VM (virtual machine including in final EXE), or without any VM (could run faster but bigger executable). Harbour has an embedded “stand-by” compiler that is similar to VFP’s macros.<br> Harbour can be classified as an object-oriented, functional and data-centric language that also supports procedures and functions. Harbour can also generate multi-threaded applications, such as web and socket servers. Virtually any C libraries can be embedded with Harbour.</p> <h4>Where does it come from?</h4> <p>The original Harbour implementation was started in 1999 as a 100% clone of Clipper, a defunct language that was a compiler version of dBase. During the last 20 years, Harbour went through majors changes, in part due to a forked version called xHarbour. Most features of xHarbour, a commercial fork, are now re-implemented in Harbour, for the exception of a form builder (closer to FoxPro for Windows than VFP). Most of the development of Harbour was done by a handful of extremely skilled C developers. All the source code and contributor information can be found on GitHub at <a href="https://github.com/harbour/core">https://github.com/harbour/core</a></p> <h4>What could Harbour be used for?</h4> <p>Since Harbour generates pure C code and has a powerful preprocessor, any other C code and libraries can be merged in. You can even write C code inside your PRG files.</p> <p>Harbour can be used to create Desktop apps, Web apps, COM objects (32-bit and 64-bit), and device-drivers. I even saw someone compile Harbour with Objective-C and create a native iOS app, and another person created a WebAssembly.</p> <h4>How does Harbour work?</h4> <p>Harbour is a language and a series of tools. Its syntax is originally from Clipper but has evolved to include all other modern language features like OOP, Hash Arrays (Python Dictionaries), UTF8 support, multi-threading, multi-OS support. Harbour has its root in C, which means that most of the C tooling is also in Harbour. It has a preprocessor, a compiler, a make tool (optional) and a debugger (replaceable). It does not have a CLI, but Visual Code can be used to facilitate rapid prototyping and debugging.</p> <h2>Examples</h2> <p>Hello world</p> <pre><code>// Typical welcome message PROCEDURE Main() ? "Hello, world!" RETURN </code></pre> <p>Interesting example using two (!) console windows simultaneously</p> <pre><code>/* * demonstration/test code for using more then one console window. * It needs GT driver which supports such functionality, i.e. * GTWVT in MS-Windows or GTXWC in XWindow. * * Copyright 2008 Przemyslaw Czerpak &lt;druzus / at / priv.onet.pl&gt; * */ #include "box.ch" #ifdef __PLATFORM__WINDOWS #if ! defined( __HBSCRIPT__HBSHELL ) REQUEST HB_GT_WVT_DEFAULT #endif #define THREAD_GT "WVT" #else REQUEST HB_GT_STD_DEFAULT #define THREAD_GT "XWC" #endif proc main() local pGT, pGT1, pGT2 #if defined( __HBSCRIPT__HBSHELL ) #if defined( __PLATFORM__WINDOWS ) hbshell_gtSelect( "GTWVT" ) #elif defined( __PLATFORM__UNIX ) hbshell_gtSelect( "GTXWC" ) #endif #endif ? "This is small test for using more then one console window." ? "It needs GT which supports such functionality i.e. GTWVT in" ? "MS-Windows or GTXWC in XWindow" wait ? "Create two new GTs:" pGT1 := hb_gtCreate( THREAD_GT ) ? "1 =&gt;", pGT1 pGT2 := hb_gtCreate( THREAD_GT ) ? "2 =&gt;", pGT1 pGT := hb_gtSelect( pGT1 ) SetColor( "W+/R" ) DispBox( 10, 10, 20, 50, HB_B_DOUBLE_UNI + " " ) ?? "This test is shown in 1-st GT window" hb_gtSelect( pGT2 ) SetColor( "W+/B" ) DispBox( 15, 30, 20, 70, HB_B_DOUBLE_UNI + " " ) ?? "This test is shown in 2-nd GT window" hb_gtSelect( pGT ) ? "New console window should be visible now" wait ? "Destroy 1-st window..." pGT1 := NIL ?? "done" wait ? "Destroy 2-nd window..." pGT2 := NIL ?? "done" wait "Press any key to exit" return </code></pre> <h2>Other sources</h2> <ul> <li><a href="https://harbour.wiki/">Wiki</a>: A collection of articles on Harbour and surrounds</li> </ul> https://tedneward.github.io/Research/storage/iboxdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/iboxdb/index.html <p><a href="https://iboxdb.github.io/index.html">Website</a></p> <p>No source?</p> https://tedneward.github.io/Research/storage/influxdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/influxdb/index.html <p><a href="https://www.influxdata.com/">Website</a></p> https://tedneward.github.io/Research/storage/jsdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/jsdb/index.html <p><a href="https://github.com/small-tech/jsdb">Source</a></p> <p>From the README:</p> <h4>Use case</h4> <p>A small and simple data layer for basic persistence and querying. Built for us in Small Web places and used in Site.js and Place.</p> <h4>Features</h4> <ul> <li> <p>Transparent: if you know how to work with arrays and objects and call methods in JavaScript, you already know how to use JSDB? It’s not called JavaScript Database for nothing.</p> </li> <li> <p>Automatic: it just works. No configuration.</p> </li> <li> <p>100% code coverage: meticulously tested. Note that this does not mean it is bug free ;)</p> </li> </ul> <h4>Limitations</h4> <ul> <li> <p>Small Data: this is for small data, not Big Data™.</p> </li> <li> <p>For Node.js: will not work in the browser. (Although data tables are plain ECMAScript Modules (ESM; es6 modules) and can be loaded in the browser.)</p> </li> <li> <p>Runs on untrusted nodes: this is for data kept on untrusted nodes (servers). Use it judiciously if you must for public data, configuration data, etc. If you want to store personal data or model human communication, consider end-to-end encrypted and peer-to-peer replicating data structures instead to protect privacy and freedom of speech. Keep an eye on the work taking place around the Hypercore Protocol.</p> </li> <li> <p>In-memory: all data is kept in memory and, without tweaks, cannot exceed 1.4GB in size. While JSDB will work with large datasets, that’s not its primary purpose and it’s definitely not here to help you farm people for their data, so please don’t use it for that. (If that’s what you want, quite literally every other database out there is for your use case so please use one of those instead.)</p> </li> <li> <p>Streaming writes on update: writes are streamed to disk to an append-only transaction log as JavaScript statements and are both quick (in the single-digit miliseconds region on a development laptop with an SSD drive) and as safe as we can make them (synchronous at the kernel level).</p> </li> <li> <p>No schema, no migrations: again, this is meant to be a very simple persistence, query, and observation layer for local server-side data. If you want schemas and migrations, take a look at nearly every other database out there.</p> </li> </ul> https://tedneward.github.io/Research/storage/keyv/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/keyv/index.html <p><a href="https://github.com/jaredwray/keyv">Source</a></p> <p>Install Keyv: <code>npm install --save keyv</code></p> <p>By default everything is stored in memory, you can optionally also install a storage adapter.</p> <pre><code>npm install --save @keyv/redis npm install --save @keyv/mongo npm install --save @keyv/sqlite npm install --save @keyv/postgres npm install --save @keyv/mysql </code></pre> <p>Create a new Keyv instance, passing your connection string if applicable. Keyv will automatically load the correct storage adapter.</p> <pre><code>const Keyv = require('keyv'); // One of the following const keyv = new Keyv(); const keyv = new Keyv('redis://user:pass@localhost:6379'); const keyv = new Keyv('mongodb://user:pass@localhost:27017/dbname'); const keyv = new Keyv('sqlite://path/to/database.sqlite'); const keyv = new Keyv('postgresql://user:pass@localhost:5432/dbname'); const keyv = new Keyv('mysql://user:pass@localhost:3306/dbname'); // Handle DB connection errors keyv.on('error', err =&gt; console.log('Connection Error', err)); await keyv.set('foo', 'expires in 1 second', 1000); // true await keyv.set('foo', 'never expires'); // true await keyv.get('foo'); // 'never expires' await keyv.delete('foo'); // true await keyv.clear(); // undefined </code></pre> <p>You can namespace your Keyv instance to avoid key collisions and allow you to clear only a certain namespace while using the same database.</p> <pre><code>const users = new Keyv('redis://user:pass@localhost:6379', { namespace: 'users' }); const cache = new Keyv('redis://user:pass@localhost:6379', { namespace: 'cache' }); await users.set('foo', 'users'); // true await cache.set('foo', 'cache'); // true await users.get('foo'); // 'users' await cache.get('foo'); // 'cache' await users.clear(); // undefined await users.get('foo'); // undefined await cache.get('foo'); // 'cache' </code></pre> <p>You can also use third-party storage adapters or build your own. Keyv will wrap these storage adapters in TTL functionality and handle complex types internally.</p> <pre><code>const Keyv = require('keyv'); const myAdapter = require('./my-storage-adapter'); const keyv = new Keyv({ store: myAdapter }); </code></pre> <p>Any store that follows the Map api will work.</p> <pre><code>new Keyv({ store: new Map() }); </code></pre> <p>For example, quick-lru is a completely unrelated module that implements the Map API.</p> <pre><code>const Keyv = require('keyv'); const QuickLRU = require('quick-lru'); const lru = new QuickLRU({ maxSize: 1000 }); const keyv = new Keyv({ store: lru }); </code></pre> https://tedneward.github.io/Research/storage/leveldb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/leveldb/index.html <p>You are probably using LevelDB without knowing it since it's part of the Chrome browser (exposed as <a href="indexeddb.html">IndexedDB</a>). If we compare it to Redis, it's even lighter, has more reliable writes and, unlike Redis, you don't store your entire data in memory but in the file system. Another difference is Redis has more data structures (sets, lists, etc.) that LevelDB is lacking. The similarity is that both store key-value pairs and both can save JSON. C/C++ library.</p> <p><a href="https://github.com/google/leveldb">Github</a> | <a href="https://github.com/google/leveldb/blob/master/doc/index.md">Documentation</a> is bundled with the source</p> <h2>Features and limitations</h2> <p>Keys and values are arbitrary byte arrays.</p> <p>Data is stored sorted by key.</p> <p>Callers can provide a custom comparison function to override the sort order.</p> <p>The basic operations are Put(key,value), Get(key), Delete(key).</p> <p>Multiple changes can be made in one atomic batch.</p> <p>Users can create a transient snapshot to get a consistent view of data.</p> <p>Forward and backward iteration is supported over the data.</p> <p>Data is automatically compressed using the Snappy compression library.</p> <p>External activity (file system operations etc.) is relayed through a virtual interface so users can customize the operating system interactions.</p> <p>This is not a SQL database. It does not have a relational data model, it does not support SQL queries, and it has no support for indexes.</p> <p>Only a single process (possibly multi-threaded) can access a particular database at a time.</p> <p>There is no client-server support builtin to the library. An application that needs such support will have to wrap their own server around the library.</p> <p>What situations is it good for?<br> * Whenever you need an easy way to persist data without worrying about RAM but with speed very close to memory access<br> * Smartphones, Raspberry Pi or similar small device<br> * Offline web apps - if you use LevelDB on the server you might be able to use the same code for the browser storage (<a href="indexeddb.html">IndexedDB</a>)<br> * Building block for a database. LevelDB (or a variant) is used in BigTable, Hadoop’s HBase, Cassandra and Riak.</p> <h3>Example (in NodeJS)</h3> <p><code>npm install [levelup](https://www.npmjs.com/package/levelup)</code>: most popular node package for LevelDB</p> <pre><code>var levelup = require('levelup') var db = levelup('./mydb') db.put('post!' + Date.now(), {author: 'fred', title: 'Angular', content: '&lt;p&gt; bla bla bla &lt;/p&gt;'}, {encoding:'json'}, function (err) { if (err) return console.log('Ooops!', err) // some kind of I/O error db.put('post!' + Date.now(), {author: 'fred', title: 'Responsive Design', content: '&lt;p&gt; blu blu blu &lt;/p&gt;'}, {encoding:'json'}, function (err) { if (err) return console.log('Ooops!', err) // some kind of I/O error // to get a stream of all posts in reverse chronological order db.createReadStream({ reverse: true, encoding: 'json' }) .on('data', function (data) { console.log(data.key, '=', data.value) }) .on('error', function (err) { console.log('Oh my!', err) }) .on('close', function () { console.log('Stream closed') }) .on('end', function () { console.log('Stream ended') }) }) }); </code></pre> <p>API is very simple (basic CRUD ops). Keys must be unique, hence the use of Date.now() (which returns a ridiculous number of seconds representing now); keys are strings.</p> <h2>Resources</h2> <ul> <li>Podcast episode about LevelDB - <a href="http://nodeup.com/fortyone">http://nodeup.com/fortyone</a></li> <li>Blog post about LevelDB Internals - <a href="http://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/">http://www.igvita.com/2012/02/06/sstable-and-log-structured-storage-leveldb/</a></li> <li>Slides by the creator of LevelUp - Rod Vagg - <a href="http://rvagg.github.com/presentations/nodejsdub/#/">http://rvagg.github.com/presentations/nodejsdub/#/</a></li> <li>GUI for LevelDB, built by Paolo Fragomeni - <a href="https://github.com/hij1nx/levelweb">https://github.com/hij1nx/levelweb</a></li> </ul> https://tedneward.github.io/Research/storage/lokijs/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/lokijs/index.html <p><a href="http://techfort.github.io/LokiJS/">Website</a> | <a href="https://github.com/techfort/LokiJS">Source</a></p> <p>LokiJS is a document oriented database written in javascript, published under MIT License. Its purpose is to store javascript objects as documents in a nosql fashion and retrieve them with a similar mechanism. Runs in node (including cordova/phonegap and node-webkit), nativescript and the browser. LokiJS is ideal for the following scenarios:</p> <ul> <li>client-side in-memory db is ideal (e.g., a session store)</li> <li>performance critical applications</li> <li>cordova/phonegap mobile apps where you can leverage the power of javascript and avoid interacting with native databases</li> <li>data sets loaded into a browser page and synchronised at the end of the work session</li> <li>node-webkit desktop apps</li> <li>nativescript mobile apps that mix the power and ubiquity of javascript with native performance and ui</li> </ul> <p>LokiJS supports indexing and views and achieves high-performance through maintaining unique and binary indexes (indices) for data.</p> <p>If you're working in a node.js environment, run <code>npm install lokijs</code> and make sure to call <code>var loki = require('lokijs')</code></p> <p>Creating the db:</p> <pre><code>var db = new loki('Example'); </code></pre> <p>Create a collection, specifying name, Type, index fields and whether the collection is transactional:</p> <pre><code>var users = db.addCollection('users', { indices: ['email'] }); // note that indices can be a single string or an array of strings </code></pre> <p>Note that indexes and transactional flag are optional parameters. Transactions in LokiJS simply allow you to run operations and automatically restore the database to the stage it was before the beginning of the transaction if an error occurred.</p> <p>Add a bunch of users in the database:</p> <pre><code>var odin = users.insert( { name : 'odin', email: 'odin.soap@lokijs.org', age: 38 } ); var thor = users.insert( { name : 'thor', email : 'thor.soap@lokijs.org', age: 25 } ); var stan = users.insert( { name : 'stan', email : 'stan.soap@lokijs.org', age: 29 } ); var oliver = users.insert( { name : 'oliver', email : 'oliver.soap@lokijs.org', age: 31 } ); var hector = users.insert( { name : 'hector', email : 'hector.soap@lokijs.org', age: 15} ); var achilles = users.insert( { name : 'achilles', email : 'achilles.soap@lokijs.org', age: 31 } ); </code></pre> <p>Operate an update:</p> <pre><code>stan.name = 'Stan Laurel'; // update object (this really only syncs the index) users.update(stan); </code></pre> <p>DynamicViews (recommended approach):</p> <pre><code>var dv = users.addDynamicView('a_complex_view'); dv.applyWhere(function aCustomFilter(obj){ return obj.name.length &lt; 5 &amp;&amp; obj.age &gt; 30; }); //view the data console.log(dv.data()); // apply some changes users.insert({ name: 'ratatosk', email: 'rata@tosk.r', age: 10320 }); // behold the dynamicview updating itself by inspecting the data console.log(dv.data()); </code></pre> <p>'Where' filter functions:</p> <pre><code>function ageView(obj){ return obj.age &gt; 30; } // a little more complicated, users with names longer than 3 characters and age over 30 function aCustomFilter(obj){ return obj.name.length &lt; 5 &amp;&amp; obj.age &gt; 30; } // test the filters var result = users.where(ageView); var anotherResult = users.where(aCustomFilter); </code></pre> <p>Map Reduce (live example on lokijs.org ):</p> <pre><code>function getDuration( obj ){ return obj.complete ? null : obj.duration; } function getAverage( array ){ var cumulator = 0; var i = array.length &gt;&gt;&gt; 0; var actual = 0; while(i--){ if(array[i] != null){ cumulator += array[i]; actual++; } } return ( cumulator / actual).toFixed(2); } var avgDuration = todos.mapReduce( getDuration, getAverage ); </code></pre> <p>Querying via method chaining :</p> <pre><code>users.chain() .find({'age':{'$gt': 25}}) .where(function(obj){ return obj.name.indexOf("in") != -1 }) .simplesort("age") .offset(50) .limit(10) .data() </code></pre> <p>Simple named transform :</p> <pre><code>users.addTransform('progeny', [ { type: 'find', value: { 'age': {'$lte': 40} } }, { type: 'simplesort', property: 'age', desc: true } ]); var results = users.chain('progeny').data(); </code></pre> <p><a href="https://github.com/techfort/LokiJS/tree/master/examples">Examples</a></p> https://tedneward.github.io/Research/storage/markdowndb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/markdowndb/index.html <p><a href="https://markdowndb.com/">Website</a> | <a href="https://github.com/datopian/markdowndb">Source</a> | <a href="https://markdowndb.com/blog/basic-tutorial">Tutorial</a></p> <p>Stores to a SQLite database.</p> https://tedneward.github.io/Research/storage/minio/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/minio/index.html <p><a href="https://min.io/">Website</a> | <a href="https://github.com/minio/minio">Source</a></p> <p><em>Entered maintenance mode</em></p> https://tedneward.github.io/Research/storage/arcadedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/arcadedb/index.html <p><a href="https://arcadedb.com/">Website</a> | <a href="https://github.com/ArcadeData/arcadedb">Source</a></p> <p>ArcadeDB is fully transactional DBMS with support for ACID transactions, structured and unstructured data, native graph engine (no joins but links between records), full-text indexing, geospatial querying, and advanced security.</p> <p>ArcadeDB supports the following models:</p> <ul> <li>Graph Database (compatible with Neo4j Cypher, Apache Tinkerpop Gremlin and OrientDB SQL)</li> <li>Document Database (compatible with the MongoDB driver + MongoDB queries and OrientDB SQL)</li> <li>Key/Value (compatible with the Redis driver)</li> <li>Search Engine</li> <li>Time Series</li> <li>Vector Embedding</li> </ul> <p>ArcadeDB understands multiple languages:</p> <ul> <li>SQL (from OrientDB SQL)</li> <li>Neo4j Cypher (Open Cypher)</li> <li>Apache Gremlin (Apache Tinkerpop v3.7.x)</li> <li>GraphQL Language</li> <li>MongoDB Query Language</li> </ul> <p>ArcadeDB can be used as:</p> <ul> <li>Embedded from any language on top of the Java Virtual Machine</li> <li>Remotely by using HTTP/JSON</li> <li>Remotely by using a Postgres driver (ArcadeDB implements Postgres Wire protocol)</li> <li>Remotely by using a Redis driver (only a subset of the operations are implemented)</li> <li>Remotely by using a MongoDB driver (only a subset of the operations are implemented)</li> </ul> https://tedneward.github.io/Research/storage/bolt/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/bolt/index.html <p><a href="https://github.com/boltdb/bolt">Source</a> (Archived/"done")</p> <p>The goal of the project is to provide a simple, fast, and reliable database for projects that don't require a full database server such as Postgres or MySQL.</p> <p>Since Bolt is meant to be used as such a low-level piece of functionality, simplicity is key. The API will be small and only focus on getting values and setting values.</p> <p>Project Status: "Bolt is stable, the API is fixed, and the file format is fixed. Full unit test coverage and randomized black box testing are used to ensure database consistency and thread safety. Bolt is currently used in high-load production environments serving databases as large as 1TB. Many companies such as Shopify and Heroku use Bolt-backed services every day."</p> https://tedneward.github.io/Research/storage/cayley/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cayley/index.html <p>"Features:<br> * Built-in query editor, visualizer and REPL<br> * Multiple query languages:<br> * Gizmo: query language inspired by Gremlin<br> * GraphQL-inspired query language<br> * MQL: simplified version for Freebase fans<br> * Modular: easy to connect to your favorite programming languages and back-end stores<br> * Production ready: well tested and used by various companies for their production workloads<br> * Fast: optimized specifically for usage in applications"</p> <p><a href="https://cayley.io/">Website</a> | <a href="https://github.com/cayleygraph/cayley">Source</a></p> https://tedneward.github.io/Research/storage/coherence/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/coherence/index.html <p><a href="http://coherence.java.net/">Website</a> | <a href="https://github.com/oracle/coherence">Source</a></p> https://tedneward.github.io/Research/storage/cratedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cratedb/index.html <p><a href="https://crate.io/">Website</a> | <a href="https://github.com/crate/crate">Source</a></p> <p>Blog:</p> <ul> <li><a href="https://crate.io/blog/handling-dynamic-objects-in-cratedb">"Handling Dynamic Objects in CrateDB"</a></li> <li><a href="https://crate.io/blog/analyzing-and-visualizing-twitter-conversations">"Analyzing and Visualizing Twitter Conversations"</a></li> <li><a href="https://crate.io/blog/indexing-and-storage-in-cratedb">"Indexing and Storage in CrateDB"</a></li> </ul> https://tedneward.github.io/Research/storage/derby/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/derby/index.html <p><a href="https://db.apache.org/derby/">Website</a> | <a href="https://db.apache.org/derby/manuals/index.html">Docs</a> | <a href="https://db.apache.org/derby/papers/derby_web.html">Source</a></p> <p>Full-featured, runs either client-server or embedded. Also supports Java stored procedures.</p> https://tedneward.github.io/Research/storage/documentdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/documentdb/index.html <p><a href="https://documentdb.io/">Website</a> | <a href="https://github.com/documentdb/documentdb">Source</a> | <a href="https://documentdb.io/docs/">Docs</a></p> <p>It offers a native implementation of document-oriented NoSQL database, enabling seamless CRUD operations on BSON data types within a PostgreSQL framework. Beyond basic operations, DocumentDB empowers you to execute complex workloads, including full-text searches, geospatial queries, and vector embeddings on your dataset, delivering robust functionality and flexibility for diverse data management needs.</p> <p>The project comprises of two PostgreSQL primary components, which work together to support document operations.</p> <ul> <li>pg_documentdb_core : PostgreSQL extension introducing BSON datatype support and operations for native Postgres.</li> <li>pg_documentdb : The public API surface for DocumentDB providing CRUD functionality on documents in the store.</li> </ul> <h2>Articles</h2> <ul> <li><a href="https://opensource.microsoft.com/blog/2025/01/23/documentdb-open-source-announcement/">"DocumentDB: Open-Source Announcement"</a></li> <li><a href="https://venturebeat.com/data-infrastructure/aws-microsoft-and-google-unite-behind-linux-foundation-documentdb-database-to-cut-enterprise-costs-and-limit-vendor-lock-in">https://venturebeat.com/data-infrastructure/aws-microsoft-and-google-unite-behind-linux-foundation-documentdb-database-to-cut-enterprise-costs-and-limit-vendor-lock-in</a></li> </ul> https://tedneward.github.io/Research/storage/dybase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dybase/index.html <p><a href="http://www.garret.ru/~knizhnik/dybase.html">Website</a> | <a href="http://www.garret.ru/~knizhnik/dybase-020.zip">Sources (ZIP)</a> | <a href="http://www.garret.ru/~knizhnik/dybase/doc/dybase.html">Docs</a></p> https://tedneward.github.io/Research/storage/edgedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/edgedb/index.html <p><a href="https://www.edgedb.com/">Website</a> | <a href="https://github.com/edgedb/edgedb">Source</a></p> https://tedneward.github.io/Research/storage/extremedb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/extremedb/index.html <p><a href="https://www.mcobject.com/extremedbfamily/">Website</a></p> https://tedneward.github.io/Research/storage/filestash/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/filestash/index.html <p><a href="https://www.filestash.app/">Website</a> | <a href="https://github.com/mickael-kerjean/filestash">Source</a></p> https://tedneward.github.io/Research/storage/geode/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/geode/index.html <p>"Geode pools memory, CPU, network resources, and optionally local disk across multiple processes to manage application objects and behavior. It uses dynamic replication and data partitioning techniques to implement high availability, improved performance, scalability, and fault tolerance. In addition to being a distributed data container, Apache Geode is an in-memory data management system that provides reliable asynchronous event notifications and guaranteed message delivery."</p> <p><a href="https://geode.apache.org/">Website</a> | <a href="http://github.com/apache/geode">Source</a></p> https://tedneward.github.io/Research/storage/hbase/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/hbase/index.html https://tedneward.github.io/Research/storage/implementation/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/implementation/index.html <p><a href="https://www.reddit.com/r/databasedevelopment/">/r/databasedevelopment</a></p> <h2>Build your own....</h2> <ul> <li><a href="https://www.codeproject.com/Articles/7410/Implementation-of-a-B-Tree-Database-Class">B-Tree Implementation</a></li> <li><a href="https://transactional.blog/building-berkeleydb/">Building BerkeleyDB</a></li> <li><a href="https://github.com/amit-davidson/Building-a-NoSQL-database-from-zero">Building a NoSQL database from zero</a></li> <li><a href="https://betterprogramming.pub/build-a-nosql-database-from-the-scratch-in-1000-lines-of-code-8ed1c15ed924">Build a NoSQL database from scratch in 1000 lines of code</a> (in Go)</li> <li><a href="https://blog.rajdeep-das.com/building-a-distributed-database-step-by-step-part-1-abc7d944e52d">Building a Distributed Database</a> <a href="https://github.com/Prashant47/distributed-database">Source</a></li> <li><a href="https://www.nan.fyi/database">Build your own database</a>: build a key-value database from the ground up</li> <li><a href="https://technicaldeft.com/build-a-database-server">Build a Database server</a>: Book ($34, with 14-day refund)</li> <li><a href="https://skyzh.github.io/mini-lsm/">Mini-LSM</a>: Build a simple key-value storage engine in a week. Extend it in the second and third weeks.</li> <li><a href="https://github.com/amit-davidson/LibraDB">LibraDB</a>: "... a simple, persistent key/value store written in pure Go. The project aims to provide a working yet simple example of a working database."</li> <li><a href="https://github.com/awelm/simpledb">simpledb</a>: A simple database built from scratch that has some the basic RDBMS features (SQL query parser, transactions, query optimizer)</li> <li><a href="https://medium.com/swlh/how-to-build-a-relational-database-from-scratch-e208061027c7">"How to build a relational database from scratch"</a> (Medium members only)</li> <li><strong>C</strong>: <a href="https://cstack.github.io/db_tutorial/"><em>Let's Build a Simple Database</em></a> Build a clone of sqlite from scratch (<a href="https://github.com/cstack/db_tutorial">Source</a>)</li> <li><strong>C++</strong>: <a href="https://build-your-own.org/redis"><em>Build Your Own Redis from Scratch</em></a></li> <li><strong>C++</strong>: <a href="https://www.codeproject.com/Articles/7410/Implementation-of-a-B-Tree-Database-Class"><em>Implementation of a B-Tree Class</em></a></li> <li><strong>C#</strong>: <a href="https://www.codeproject.com/Articles/1029838/Build-Your-Own-Database"><em>Build Your Own Database</em></a> (<a href="https://github.com/nam178/FooDB">Source</a>)</li> <li><strong>Clojure</strong>: <a href="http://aosabook.org/en/500L/an-archaeology-inspired-database.html"><em>An Archaeology-Inspired Database</em></a></li> <li><strong>Crystal</strong>: <a href="https://medium.com/@marceloboeira/why-you-should-build-your-own-nosql-database-9bbba42039f5"><em>Why you should build your own NoSQL Database</em></a> (<a href="https://github.com/marceloboeira/bojack">Source</a>, Archived)</li> <li><strong>Go</strong>: <a href="https://build-your-own.org/database/"><em>Build Your Own Database from Scratch: Persistence, Indexing, Concurrency</em></a></li> <li><strong>Go</strong>: <a href="https://www.build-redis-from-scratch.dev/"><em>Build Your Own Redis from Scratch</em></a></li> <li><strong>Go</strong>: <a href="https://github.com/krasun/gosqldb">gosqldb</a>: A key-value persistent database that supports SQL queries over B+ and LSM trees</li> <li><strong>Java</strong>: Electric's <a href="http://megacz.com/software/btree.html">B-Tree</a> (Source JAR file)</li> <li><strong>Java</strong>: <a href="https://github.com/jankotek/JDBM3"><em>JDBM3</em></a>: Work was paused and redirected to JDBM4-renamed-MapDB</li> <li><strong>Java</strong>: <a href="https://mapdb.org/"><em>MapDB</em></a>: MapDB provides concurrent Maps, Sets and Queues backed by disk storage or off-heap-memory. It is a fast and easy to use embedded Java database engine. (<a href="https://github.com/jankotek/mapdb/">Source</a>)</li> <li><strong>Java</strong>: <a href="http://cs.bc.edu/~sciore/simpledb/">The SimpleDB Data System</a>: "... a multi-user transactional database server written in Java, which interacts with Java client programs via JDBC. The system is intended for pedagogical use only. The code is clean and compact. The APIs are straightforward. The learning curve is relatively small. Everything about it is geared towards improving the experience of a database system internals course. Consequently, the system is intentionally bare-bones. It implements only a small fraction of SQL and JDBC, and does little or no error checking. The SimpleDB code is an integral part of my textbook <a href="https://www.amazon.com/dp/3030338355/">Database Design and Implementation</a>, published by Springer."</li> <li><strong>JavaScript</strong>: <a href="http://aosabook.org/en/500L/dagoba-an-in-memory-graph-database.html"><em>Dagoba: an in-memory graph database</em></a></li> <li><strong>Python</strong>: <a href="http://aosabook.org/en/500L/dbdb-dog-bed-database.html"><em>DBDB: Dog Bed Database</em></a></li> <li><strong>Python</strong>: <a href="http://charlesleifer.com/blog/building-a-simple-redis-server-with-python/"><em>Write your own miniature Redis with Python</em></a></li> <li><strong>Ruby</strong>: <a href="https://dineshgowda.com/posts/build-your-own-persistent-kv-store/"><em>Build your own fast, persistent KV store in Ruby</em></a></li> <li><strong>Rust</strong>: <a href="https://tokio.rs/tokio/tutorial/setup"><em>Build your own Redis client and server</em></a></li> <li><strong>Rust</strong>: <a href="https://github.com/yywe/yoursql">YourSQL</a></li> <li><strong>Rust</strong>: <a href="https://github.com/mzinsmeister/OxidSQL">OxidSQL</a></li> <li><strong>Rust</strong>: <a href="https://github.com/radogost/erdb">erdb</a>: An educational relational database</li> </ul> <h2>Reading</h2> <ul> <li><a href="https://news.ycombinator.com/item?id=18410692">Ask HN: Books about database implementation</a></li> <li><a href="https://riak.com/assets/bitcask-intro.pdf">Bitcask paper</a></li> <li><a href="https://en.wikipedia.org/wiki/B-tree">Btree (Wikipedia)</a></li> <li><a href="https://benjamincongdon.me/blog/2021/08/17/B-Trees-More-Than-I-Thought-Id-Want-to-Know/">B-trees: More than I thought I'd want to know</a></li> <li><a href="https://www.programiz.com/dsa/b-tree">B-tree</a></li> <li><a href="https://transactional.blog/blog/2024-data-replication-design-spectrum">Data Replication Design Spectrum</a></li> <li><a href="http://webdam.inria.fr/Alice/">Foundations of Databases</a> (PDF available)</li> <li><a href="https://www.lucavall.in/blog/how-databases-store-and-retrieve-data-with-b-trees">How Databases Store and Retrieve Data with B-Trees</a></li> <li><a href="https://www.geeksforgeeks.org/introduction-of-b-tree-2/">Introduction of B-tree</a></li> </ul> <h3>Transactions</h3> <ul> <li><a href="https://transactional.blog/blog/2025-decomposing-transactional-systems">"Decomposing Transactional Systems"</a>: "Every transactional system does four things: <ul> <li>It <em>executes</em> transactions.</li> <li>It <em>orders</em> transactions.</li> <li>It <em>validates</em> transactions.</li> <li>It <em>persists</em> transactions.</li> </ul> <p><em>Executing</em> a transaction means evaluating the body of the transaction to produce the intended reads and writes. There is still notable variety across systems as to how the body of a transaction is executed. Writes might be applied to storage during this phase, or they might be buffered locally and submitted as a batch at the end. A transaction might be executed more than once for different purposes.</p> <p><em>Ordering</em> a transaction means assigning the transaction some notion of a time at which it occurred. This could be a version, a timestamp, a log sequence number, or a more complex description of transaction IDs it happened before or after. MVCC databases may assign two versions: an initial read version, and a final commit version. In this case, we’re mainly focused on the specific point at which the commit version is chosen — the time at which the database claims all reads and writes occurred atomically.</p> <p><em>Validating</em> a transaction means enforcing concurrency control, or more rarely, domain-specific semantics. If a transaction is going to be rejected for a system defined reason, such as having serializability conflicts with existing transactions, it will happen here. When validation happens after ordering, it checks to see if the assigned order is valid. When validation happens before ordering, it provides a range of acceptable commit versions, and the ordering step chooses one of them.</p> <p><em>Persisting</em> a transaction makes making it durable, generally to disk. Sometimes writes are incrementally made durable during transaction execution, but the key point in persistence is when all writes and the commit record marking the transaction as committed are durable. Often this is noting how the system performs replication and persists the outcome of its atomic commitment protocol. (And sometimes the lines between those two aren’t very clear.)</p> <p>All four of these things must be done before the system may acknowledge a transaction’s result to a client. However, these steps can be done in any order. They can be done concurrently. Different systems achieve different tradeoffs by reordering these steps. ... A classic optimistic concurrency control database will execute a transaction, and record the read and write sets. Once execution finishes, a commit version is allocated, and concurrent transactions are checked for conflicts. If no conflicts were found, the transaction is made durable, and acknowledged as committed. ... A classic pessimistic concurrency control database executes a transaction and acquires locks as it runs to exclude conflicting transactions. When the transaction finishes, it acquires a commit version, and then is persisted to disk. Then it releases the locks.</p> </li> </ul> https://tedneward.github.io/Research/storage/innodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/innodb/index.html <p><a href="https://en.wikipedia.org/wiki/InnoDB">Wikipedia</a> | <a href="https://mariadb.com/kb/en/innodb/">Website</a></p> https://tedneward.github.io/Research/storage/jsondb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/jsondb/index.html <p><a href="https://jsondb.io/">Website</a> | <a href="https://github.com/Jsondb/jsondb-core">Source</a></p> <p>Small memory footprint, runs embedded within your Java program,<br> Provides APIs that are very similar in names and semantics to those of Spring Data for MongoDB,<br> Supports encryption of data, XPath based search/find queries.</p> https://tedneward.github.io/Research/storage/kinto/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/kinto/index.html <p><a href="https://www.kinto-storage.org/">Website</a> | <a href="https://docs.kinto-storage.org/en/latest/overview.html">Docs</a></p> https://tedneward.github.io/Research/storage/libsql/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/libsql/index.html <p><a href="https://libsql.org/">Website</a> | <a href="https://github.com/libsql/libsql">Source</a></p> <p>Related: <a href="https://github.com/libsql/sqld">sqld</a> - networked version of libsql</p> <h2>Articles</h2> <ul> <li><a href="https://blog.logrocket.com/libsql-next-gen-fork-sqlite/">Get started with libSQL, a next-gen fork of SQLite</a></li> </ul> https://tedneward.github.io/Research/storage/lowdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/lowdb/index.html <p><a href="https://github.com/typicode/lowdb">Source</a></p> https://tedneward.github.io/Research/storage/martendb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/martendb/index.html <p><a href="https://martendb.io/">Website</a> | <a href="https://github.com/jasperfx/marten">Source</a></p> <p>Document Database: The Marten library provides .NET developers with the ability to easily use the proven PostgreSQL database engine and its fantastic JSON support as a fully fledged document database. The Marten team believes that a document database has far reaching benefits for developer productivity over relational databases with or without an ORM tool.</p> <p>Event Store: Event Sourcing can be a powerful technique in applications that are workflow centric or have any need for historical queries. Marten utilizes the strong JSONB support to expose an ACID-compliant event store implementation over the PostgreSQL database. Even better yet, Marten takes advantage of PostgreSQL's embedded JavaScript support to enable effective user-defined projections against your event streams.</p> https://tedneward.github.io/Research/storage/actian/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/actian/index.html <p><a href="https://www.actian.com/data-management/nosql-object-database/">Website</a></p> https://tedneward.github.io/Research/storage/arcticdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/arcticdb/index.html <p><a href="https://arcticdb.io/">Website</a> | <a href="https://github.com/man-group/ArcticDB">Source</a></p> <p>ArcticDB is a serverless DataFrame database engine designed for the Python Data Science ecosystem.</p> <p>ArcticDB enables you to store, retrieve and process DataFrames at scale, backed by commodity object storage (S3-compatible storages and Azure Blob Storage).</p> <p>ArcticDB requires zero additional infrastructure beyond a running Python environment and access to object storage and can be installed in seconds.</p> <p>ArcticDB is:</p> <ul> <li> <p>Fast</p> <ul> <li>Process up to 100 million rows per second for a single consumer</li> <li>Process a billion rows per second across all consumers</li> <li>Quick and easy to install: pip install arcticdb</li> </ul> </li> <li> <p>Flexible</p> <ul> <li>Data schemas are not required</li> <li>Supports streaming data ingestion</li> <li>Bitemporal - stores all previous versions of stored data</li> <li>Easy to setup both locally and on cloud</li> <li>Scales from dev/research to production environments</li> </ul> </li> <li> <p>Familiar</p> <ul> <li>ArcticDB is the world's simplest shareable database</li> <li>Easy to learn for anyone with Python and Pandas experience</li> <li>Just you and your data - the cognitive overhead is very low.</li> </ul> </li> </ul> https://tedneward.github.io/Research/storage/brightstardb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/brightstardb/index.html <p><a href="https://brightstardb.com/">Website</a> | <a href="https://github.com/BrightstarDB/BrightstarDB">Source</a></p> <ul> <li> <p><strong>Cross Platform</strong> BrightstarDB runs on Windows, Linux, OSX, Windows Phone, Android and iOS, with a single consistent data file format across all platforms. The core libraries have a small footprint and install with zero configuration for embedded applications.</p> </li> <li> <p><strong>Server or Embedded</strong> Run BrightstarDB embedded inside your own application or connect to a BrightstarDB server using either the RESTful HTTP API or the strongly-typed .NET client API.</p> </li> <li> <p><strong>RDF Flexibility</strong> BrightstarDB is an RDF triplestore. It does not require the definition of a database schema, and with the RDF data model model you can easily add and integrate data of all shapes. We also implement the standard SPARQL query language, update language and protocol so you can use off-the-shelf client tools to connect to your data stores.</p> </li> <li> <p><strong>LINQ Simplicity</strong> BrightstarDB provides an ORM for RDF. A custom LINQ provider maps queries against your application domain model into SPARQL queries. This provides full CRUD and query capability for strongly-typed .NET applications.</p> </li> <li> <p><strong>Our Store or Yours</strong> The ORM for RDF, LINQ to SPARQL and the .NET dynamic objects API all support connecting to any SPARQL endpoint that implements SPARQL 1.1 standards. So you can choose to store your triples in BrightstarDB or any other compliant triplestore and still make use of our advanced, developer-friendly APIs.</p> </li> <li> <p><strong>Open Source</strong> BrightstarDB is developed in the open on GitHub. Our code is licensed under the permissive MIT license, allowing you to use BrightstarDB in commercial and non-commercial applications alike. Community support is available from our GitHub issue tracker. Commercial support is also available from NetworkedPlanet.</p> </li> </ul> https://tedneward.github.io/Research/storage/citus/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/citus/index.html <p><a href="https://www.citusdata.com/">Website</a> | <a href="https://github.com/citusdata/citus">Source</a></p> <p>Citus is a PostgreSQL extension that transforms Postgres into a distributed database—so you can achieve high performance at any scale.</p> <p>With Citus, you extend your PostgreSQL database with new superpowers:</p> <ul> <li>Distributed tables are sharded across a cluster of PostgreSQL nodes to combine their CPU, memory, storage and I/O capacity.</li> <li>References tables are replicated to all nodes for joins and foreign keys from distributed tables and maximum read performance.</li> <li>Distributed query engine routes and parallelizes SELECT, DML, and other operations on distributed tables across the cluster.</li> <li>Columnar storage compresses data, speeds up scans, and supports fast projections, both on regular and distributed tables.</li> <li>Query from any node enables you to utilize the full capacity of your cluster for distributed queries</li> </ul> <p>You can use these Citus superpowers to make your Postgres database scale-out ready on a single Citus node. Or you can build a large cluster capable of handling high transaction throughputs, especially in multi-tenant apps, run fast analytical queries, and process large amounts of time series or IoT data for real-time analytics. When your data size and volume grow, you can easily add more worker nodes to the cluster and rebalance the shards.</p> https://tedneward.github.io/Research/storage/cosmosdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/cosmosdb/index.html <p>"Azure Cosmos DB is a fully managed database service with turnkey global distribution and transparent multi-master replication. Get single-digit millisecond read and write latencies at the 99th percentile, automatic and elastic scaling of throughput and storage worldwide, 99.999-percent high availability, and five well-defined consistency choices—all backed by industry-leading comprehensive SLAs. Multi-model with wire protocol–compatible API endpoints for Cassandra, MongoDB, SQL, Gremlin, Etcd, and Table along with built-in support for Jupyter notebooks."</p> <p><a href="https://azure.microsoft.com/en-us/services/cosmos-db/">Website</a> | <a href="https://docs.microsoft.com/en-us/azure/cosmos-db/">Documentation</a>(<a href="https://docs.microsoft.com/en-us/azure/opbuildpdf/cosmos-db/toc.pdf?branch=live">Generated PDF</a>) | <a href="https://azure.microsoft.com/en-us/resources/developer-s-guide-to-getting-started-with-azure-cosmos-db/">Book</a></p> https://tedneward.github.io/Research/storage/db4o/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/db4o/index.html <p>Used to be product from db4objects, since liquidated.</p> <ul> <li><a href="https://github.com/iboxdb/db4o-gpl">db4o-gpl</a>: GPL'ed db4o code, both Java and .NET</li> <li><a href="https://github.com/lytico/db4o">db4o</a></li> <li><a href="https://github.com/pegurnee/db4o">db4o</a></li> </ul> https://tedneward.github.io/Research/storage/dgraph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dgraph/index.html <p><a href="https://dgraph.io/">Website</a> | <a href="https://github.com/dgraph-io/dgraph">Source</a></p> <p>Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It provides ACID transactions, consistent replication, and linearizable reads. It's built from the ground up to perform for a rich set of queries. Being a native GraphQL database, it tightly controls how the data is arranged on disk to optimize for query performance and throughput, reducing disk seeks and network calls in a cluster.</p> <p>Using the official Dgraph image: <code>docker pull dgraph/dgraph:latest</code></p> <p>Run a Quick Standalone Cluster: <code>docker run -it -p 8080:8080 -p 9080:9080 -v ~/dgraph:/dgraph dgraph/standalone:latest</code></p> https://tedneward.github.io/Research/storage/dolt/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dolt/index.html <p><a href="https://github.com/dolthub/dolt">Source</a> | <a href="https://docs.dolthub.com/introduction/what-is-dolt">Docs</a></p> <p>Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a git repository. Connect to Dolt just like any MySQL database to run queries or update the data using SQL commands. Use the command line interface to import CSV files, commit your changes, push them to a remote, or merge your teammate's changes.</p> <p>All the commands you know for Git work exactly the same for Dolt. Git versions files, Dolt versions tables. It's like Git and MySQL had a baby!</p> <p>We also built <a href="https://www.dolthub.com/">DoltHub</a>, a place to share Dolt databases. We host public data for free!</p> <p>We built Dolt using the following axioms:</p> <ul> <li>Git versions files. Dolt versions table schema and table data.</li> <li>Dolt will copy the Git command line exactly.</li> <li>Dolt will be MySQL compatible.</li> <li>Git features in SQL will extend MySQL SQL. Write operations will be procedures. Read operations will be system tables.</li> </ul> https://tedneward.github.io/Research/storage/dynamo/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/dynamo/index.html <p><a href="https://docs.aws.amazon.com/dynamodb/index.html">Website</a> | <a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/index.html">Developer Guide</a> (<a href="https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/dynamodb-dg.pdf">PDF</a>)</p> https://tedneward.github.io/Research/storage/ese/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/ese/index.html <p><a href="https://github.com/microsoft/Extensible-Storage-Engine">Source</a></p> <p>From the source README:</p> <blockquote> <p>ESE enables applications to store data to, and retrieve data from tables using indexed or sequential cursor navigation. It supports denormalized schemas including wide tables with numerous sparse columns, multi-valued columns, and sparse and rich indexes. ESE enables applications to enjoy a consistent data state using transacted data update and retrieval. A crash recovery mechanism is provided so that data consistency is maintained even in the event of a system crash. ESE provides ACID (Atomic Consistent Isolated Durable) transactions over data and schema by way of a write-ahead log and a snapshot isolation model.</p> <ul> <li>A summary of features and the JET API documentation are up on <a href="https://docs.microsoft.com/en-us/windows/win32/extensible-storage-engine/extensible-storage-engine">Microsoft's official documentation site</a></li> <li>A more extensive list of ESE database features <a href="https://en.wikipedia.org/wiki/Extensible_Storage_Engine">are documented in our Wikipedia entry</a></li> </ul> <p>The library provides many other strongly layered and, thus, reusable sub-facilities as well:</p> <ul> <li>A synchronization and locking library</li> <li>An STL-like data structures library</li> <li>An OS abstraction layer</li> <li>A Block / Cache Manager</li> </ul> </blockquote> <p>All this is in addition to the full-blown database engine itself.</p> <blockquote> <p>Most people do not know that JET was an acronym for an API set, not a specific database format or engine. Just as there is no such thing as "the SQL engine", as there are many implementations of the protocol, there is no "JET engine" or "JET database". It is in the acronym, "Joint Engine Technology". And as such, there are two separate implementations of the JET API. This is the JET Blue engine implementation, see <a href="https://docs.microsoft.com/en-us/windows/win32/extensible-storage-engine/extensible-storage-engine">Notes in here</a>. The origin of the <a href="https://en.wikipedia.org/wiki/Extensible_Storage_Engine#History">colors have an an amusing source</a> by the way. Most people think of the "JET engine" as JET Red, that shipped under Microsoft Access. This is not that "JET engine". We renamed to ESE to try to avoid this confusion, but it seems that the confusion continues to this day.</p> </blockquote> https://tedneward.github.io/Research/storage/factdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/factdb/index.html <p><a href="https://github.com/MadBomber/fact_db">Source</a></p> <p>FactDb implements the Event Clock concept - capturing organizational knowledge through temporal facts with validity periods (valid_at/invalid_at), entity resolution, and audit trails back to source content.</p> <p>Key Features</p> <ul> <li>Temporal Facts - Track facts with validity periods</li> <li>Entity Resolution - Resolve mentions to canonical entities</li> <li>Audit Trails - Every fact links back to source content</li> <li>Multiple Extractors - Extract facts manually, via LLM, or rule-based</li> <li>Semantic Search - PostgreSQL with pgvector</li> <li>Concurrent Processing - Batch process with parallel pipelines</li> <li>Output Formats - JSON, triples, Cypher, or text for LLM consumption</li> <li>Temporal Queries - Fluent API for point-in-time queries and diffs</li> </ul> https://tedneward.github.io/Research/storage/firebirdsql/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/firebirdsql/index.html <p><a href="https://www.firebirdsql.org">Website</a> | <a href="https://github.com/FirebirdSQL/">Source</a> | <a href="https://www.firebirdsql.org/en/firebird-rdbms/">Docs</a></p> https://tedneward.github.io/Research/storage/graphdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/graphdb/index.html <p><a href="https://www.ontotext.com/products/graphdb/">Website</a> | Commercial/closed-source</p> <p>Run GraphDB in Docker: <code>docker run -d -p 7200:7200 ontotext/graphdb:10.0.0</code> (Source: <a href="https://github.com/Ontotext-AD/graphdb-docker">https://github.com/Ontotext-AD/graphdb-docker</a>)</p> https://tedneward.github.io/Research/storage/hsqldb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/hsqldb/index.html <p><a href="http://hsqldb.org/">Website</a> <a href="https://sourceforge.net/projects/hsqldb/">Source</a></p> https://tedneward.github.io/Research/storage/indexeddb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/indexeddb/index.html <p>"IndexedDB is a transactional database system, like an SQL-based RDBMS. However, unlike SQL-based RDBMSes, which use fixed-column tables, IndexedDB is a JavaScript-based object-oriented database. IndexedDB lets you store and retrieve objects that are indexed with a key; any objects supported by the structured clone algorithm can be stored. You need to specify the database schema, open a connection to your database, and then retrieve and update data within a series of transactions."</p> <p><a href="https://w3c.github.io/IndexedDB/">Specification</a> | <a href="https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API">Reference</a></p> <h2>Using</h2> <h3>Open the database</h3> <pre><code>// Let us open our database const request = window.indexedDB.open("MyTestDatabase", 3); </code></pre> <p>The open request doesn't open the database or start the transaction right away. The call to the open() function returns an IDBOpenDBRequest object with a result (success) or error value that you handle as an event. Most other asynchronous functions in IndexedDB do the same thing - return an IDBRequest object with the result or error. The result for the open function is an instance of an IDBDatabase.</p> <p>The second parameter to the open method is the version of the database. The version of the database determines the database schema — the object stores in the database and their structure. If the database doesn't already exist, it is created by the open operation, then an <code>onupgradeneeded</code> event is triggered and you create the database schema in the handler for this event. If the database does exist but you are specifying an upgraded version number, an <code>onupgradeneeded</code> event is triggered straight away, allowing you to provide an updated schema in its handler. More on this later in "Creating or updating the version of the database" below, and the <a href="https://developer.mozilla.org/en-US/docs/Web/API/IDBFactory/open"><code>IDBFactory.open</code></a> page.</p> <p>Warning: The version number is an <code>unsigned long long</code> number, which means that it can be a very big integer. It also means that you can't use a float, otherwise it will be converted to the closest lower integer and the transaction may not start, nor the upgradeneeded event trigger. So for example, don't use 2.4 as a version number: <code>const request = indexedDB.open("MyTestDatabase", 2.4); // don't do this, as the version will be rounded to 2</code></p> <h4>Generating handlers</h4> <pre><code>request.onerror = (event) =&gt; { // Do something with request.error! }; request.onsuccess = (event) =&gt; { // Do something with request.result! }; </code></pre> <p>If the request succeeds, the success event is fired, and the function assigned to onsuccess is called. If the request fails, the error event is fired, and the function assigned to onerror is called.</p> <p>The IndexedDB API is designed to minimize the need for error handling, so you're not likely to see many error events (at least, not once you're used to the API!). In the case of opening a database, however, there are some common conditions that generate error events. The most likely problem is that the user decided not to give your web app permission to create a database. One of the main design goals of IndexedDB is to allow large amounts of data to be stored for offline use. (To learn more about how much storage you can have for each browser, see How much data can be stored? on the Browser storage quotas and eviction criteria page.)</p> <p>Obviously, browsers do not want to allow some advertising network or malicious website to pollute your computer, so browsers used to prompt the user the first time any given web app attempts to open an IndexedDB for storage. The user could choose to allow or deny access. Also, IndexedDB storage in browsers' privacy modes only lasts in-memory until the incognito session is closed.</p> <p>Now, assuming that the user allowed your request to create a database, and you've received a success event to trigger the success callback; What's next? The request here was generated with a call to indexedDB.open(), so request.result is an instance of IDBDatabase, and you definitely want to save that for later.</p> <h4>Handling errors</h4> <p>As mentioned above, error events bubble. Error events are targeted at the request that generated the error, then the event bubbles to the transaction, and then finally to the database object. If you want to avoid adding error handlers to every request, you can instead add a single error handler on the database object, like so:</p> <pre><code>db.onerror = (event) =&gt; { // Generic error handler for all errors targeted at this database's // requests! console.error(`Database error: ${event.target.error?.message}`); }; </code></pre> <p>One of the common possible errors when opening a database is VER_ERR. It indicates that the version of the database stored on the disk is greater than the version that you are trying to open. This is an error case that must always be handled by the error handler.</p> <h4>Creating or updating the version of the database</h4> <p>When you create a new database or increase the version number of an existing database (by specifying a higher version number than you did previously, when Opening a database), the onupgradeneeded event will be triggered and an IDBVersionChangeEvent object will be passed to any onversionchange event handler set up on request.result (i.e., db in the example). In the handler for the upgradeneeded event, you should create the object stores needed for this version of the database:</p> <pre><code>// This event is only implemented in recent browsers request.onupgradeneeded = (event) =&gt; { // Save the IDBDatabase interface const db = event.target.result; // Create an objectStore for this database const objectStore = db.createObjectStore("name", { keyPath: "myKey" }); }; </code></pre> <p>In this case, the database will already have the object stores from the previous version of the database, so you do not have to create these object stores again. You only need to create any new object stores, or delete object stores from the previous version that are no longer needed. If you need to change an existing object store (e.g., to change the keyPath), then you must delete the old object store and create it again with the new options. (Note that this will delete the information in the object store! If you need to save that information, you should read it out and save it somewhere else before upgrading the database.)</p> <p>Trying to create an object store with a name that already exists (or trying to delete an object store with a name that does not already exist) will throw an error.</p> <p>If the onupgradeneeded event exits successfully, the onsuccess handler of the open database request will then be triggered.</p> <h4>Structuring the database</h4> <p>Now to structure the database. IndexedDB uses object stores rather than tables, and a single database can contain any number of object stores. Whenever a value is stored in an object store, it is associated with a key. There are several different ways that a key can be supplied depending on whether the object store uses a key path or a key generator.</p> <p>The following table shows the different ways the keys are supplied:</p> <table> <thead> <tr> <th>Key Path (keyPath) </th> <th> Key Generator (autoIncrement) </th> <th> Description</th> </tr> </thead> <tbody> <tr> <td>No </td> <td> No </td> <td> This object store can hold any kind of value, even primitive values like numbers and strings. You must supply a separate key argument whenever you want to add a new value.</td> </tr> <tr> <td>Yes </td> <td> No </td> <td> This object store can only hold JavaScript objects. The objects must have a property with the same name as the key path.</td> </tr> <tr> <td>No </td> <td> Yes </td> <td> This object store can hold any kind of value. The key is generated for you automatically, or you can supply a separate key argument if you want to use a specific key.</td> </tr> <tr> <td>Yes </td> <td> Yes </td> <td> This object store can only hold JavaScript objects. Usually a key is generated and the value of the generated key is stored in the object in a property with the same name as the key path. However, if such a property already exists, the value of that property is used as key rather than generating a new key.</td> </tr> </tbody> </table> <p>You can also create indices on any object store, provided the object store holds objects, not primitives. An index lets you look up the values stored in an object store using the value of a property of the stored object, rather than the object's key.</p> <p>Additionally, indexes have the ability to enforce simple constraints on the stored data. By setting the unique flag when creating the index, the index ensures that no two objects are stored with both having the same value for the index's key path. So, for example, if you have an object store which holds a set of people, and you want to ensure that no two people have the same email address, you can use an index with the unique flag set to enforce this.</p> <p>That may sound confusing, but this simple example should illustrate the concepts. First, we'll define some customer data to use in our example:</p> <pre><code>// This is what our customer data looks like. const customerData = [ { ssn: "444-44-4444", name: "Bill", age: 35, email: "bill@company.com" }, { ssn: "555-55-5555", name: "Donna", age: 32, email: "donna@home.org" }, ]; </code></pre> <p>Of course, you wouldn't use someone's social security number as the primary key to a customer table because not everyone has a social security number, and you would store their birth date instead of their age, but let's ignore those unfortunate choices for the sake of convenience and move along.</p> <p>Now let's look at creating an IndexedDB to store our data:</p> <pre><code>const dbName = "the_name"; const request = indexedDB.open(dbName, 2); request.onerror = (event) =&gt; { // Handle errors. }; request.onupgradeneeded = (event) =&gt; { const db = event.target.result; // Create an objectStore to hold information about our customers. We're // going to use "ssn" as our key path because it's guaranteed to be // unique - or at least that's what I was told during the kickoff meeting. const objectStore = db.createObjectStore("customers", { keyPath: "ssn" }); // Create an index to search customers by name. We may have duplicates // so we can't use a unique index. objectStore.createIndex("name", "name", { unique: false }); // Create an index to search customers by email. We want to ensure that // no two customers have the same email, so use a unique index. objectStore.createIndex("email", "email", { unique: true }); // Use transaction oncomplete to make sure the objectStore creation is // finished before adding data into it. objectStore.transaction.oncomplete = (event) =&gt; { // Store values in the newly created objectStore. const customerObjectStore = db .transaction("customers", "readwrite") .objectStore("customers"); customerData.forEach((customer) =&gt; { customerObjectStore.add(customer); }); }; }; </code></pre> <p>As indicated previously, onupgradeneeded is the only place where you can alter the structure of the database. In it, you can create and delete object stores and build and remove indices.</p> <p>Object stores are created with a single call to createObjectStore(). The method takes a name of the store, and a parameter object. Even though the parameter object is optional, it is very important, because it lets you define important optional properties and refine the type of object store you want to create. In our case, we've asked for an object store named "customers" and defined a keyPath, which is the property that makes an individual object in the store unique. That property in this example is "ssn" since a social security number is guaranteed to be unique. "ssn" must be present on every object that is stored in the objectStore.</p> <p>We've also asked for an index named "name" that looks at the name property of the stored objects. As with createObjectStore(), createIndex() takes an optional options object that refines the type of index that you want to create. Adding objects that don't have a name property still succeeds, but the objects won't appear in the "name" index.</p> <p>We can now retrieve the stored customer objects using their ssn from the object store directly, or using their name by using the index. To learn how this is done, see the section on using an index.</p> <h4>Using a key generator</h4> <p>Setting up an autoIncrement flag when creating the object store would enable the key generator for that object store. By default this flag is not set.</p> <p>With the key generator, the key would be generated automatically as you add the value to the object store. The current number of a key generator is always set to 1 when the object store for that key generator is first created. Basically the newly auto-generated key is increased by 1 based on the previous key. The current number for a key generator never decreases, other than as a result of database operations being reverted, for example, the database transaction is aborted. Therefore deleting a record or even clearing all records from an object store never affects the object store's key generator.</p> <p>We can create another object store with the key generator as below:</p> <pre><code>// Open the indexedDB. const request = indexedDB.open(dbName, 3); request.onupgradeneeded = (event) =&gt; { const db = event.target.result; // Create another object store called "names" with the autoIncrement flag set as true. const objStore = db.createObjectStore("names", { autoIncrement: true }); // Because the "names" object store has the key generator, the key for the name value is generated automatically. // The added records would be like: // key : 1 =&gt; value : "Bill" // key : 2 =&gt; value : "Donna" customerData.forEach((customer) =&gt; { objStore.add(customer.name); }); }; </code></pre> <p>For more details about the key generator, see Key generators in the spec.</p> <h3>Adding, retrieving, and removing data</h3> <p>Before you can do anything with your new database, you need to start a transaction. Transactions come from the database object, and you have to specify which object stores you want the transaction to span. Once you are inside the transaction, you can access the object stores that hold your data and make your requests. Next, you need to decide if you're going to make changes to the database or if you just need to read from it. Transactions have three available modes: readonly, readwrite, and versionchange.</p> <p>To change the "schema" or structure of the database—which involves creating or deleting object stores or indexes—the transaction must be in versionchange mode. This transaction is opened by calling the IDBFactory.open method with a version specified.</p> <p>To read the records of an existing object store, the transaction can either be in readonly or readwrite mode. To make changes to an existing object store, the transaction must be in readwrite mode. You open such transactions with IDBDatabase.transaction. The method accepts two parameters: the storeNames (the scope, defined as an array of object stores that you want to access) and the mode (readonly or readwrite) for the transaction. The method returns a transaction object containing the IDBIndex.objectStore method, which you can use to access your object store. By default, where no mode is specified, transactions open in readonly mode.</p> <p>You can speed up data access by using the right scope and mode in the transaction. Here are a couple of tips:</p> <p>When defining the scope, specify only the object stores you need. This way, you can run multiple transactions with non-overlapping scopes concurrently.<br> Only specify a readwrite transaction mode when necessary. You can concurrently run multiple readonly transactions with overlapping scopes, but you can have only one readwrite transaction for an object store. To learn more, see the definition for transaction in the IndexedDB key characteristics and basic terminology article.</p> <h4>Adding data to the database</h4> <p>If you've just created a database, then you probably want to write to it. Here's what that looks like:</p> <pre><code>const transaction = db.transaction(["customers"], "readwrite"); // Note: Older experimental implementations use the deprecated constant IDBTransaction.READ_WRITE instead of "readwrite". // In case you want to support such an implementation, you can write: // const transaction = db.transaction(["customers"], IDBTransaction.READ_WRITE); </code></pre> <p>The transaction() function takes two arguments (though one is optional) and returns a transaction object. The first argument is a list of object stores that the transaction will span. You can pass an empty array if you want the transaction to span all object stores, but don't do it because the spec says an empty array should generate an InvalidAccessError. If you don't specify anything for the second argument, you get a read-only transaction. Since you want to write to it here you need to pass the "readwrite" flag.</p> <p>Now that you have a transaction you need to understand its lifetime. Transactions are tied very closely to the event loop. If you make a transaction and return to the event loop without using it then the transaction will become inactive. The only way to keep the transaction active is to make a request on it. When the request is finished you'll get a DOM event and, assuming that the request succeeded, you'll have another opportunity to extend the transaction during that callback. If you return to the event loop without extending the transaction then it will become inactive, and so on. As long as there are pending requests the transaction remains active. Transaction lifetimes are really very simple but it might take a little time to get used to. A few more examples will help, too. If you start seeing TRANSACTION_INACTIVE_ERR error codes then you've messed something up.</p> <p>Transactions can receive DOM events of three different types: error, abort, and complete. We've talked about the way that error events bubble, so a transaction receives error events from any requests that are generated from it. A more subtle point here is that the default behavior of an error is to abort the transaction in which it occurred. Unless you handle the error by first calling stopPropagation() on the error event then doing something else, the entire transaction is rolled back. This design forces you to think about and handle errors, but you can always add a catchall error handler to the database if fine-grained error handling is too cumbersome. If you don't handle an error event or if you call abort() on the transaction, then the transaction is rolled back and an abort event is fired on the transaction. Otherwise, after all pending requests have completed, you'll get a complete event. If you're doing lots of database operations, then tracking the transaction rather than individual requests can certainly aid your sanity.</p> <p>Now that you have a transaction, you'll need to get the object store from it. Transactions only let you have an object store that you specified when creating the transaction. Then you can add all the data you need.</p> <pre><code>// Do something when all the data is added to the database. transaction.oncomplete = (event) =&gt; { console.log("All done!"); }; transaction.onerror = (event) =&gt; { // Don't forget to handle errors! }; const objectStore = transaction.objectStore("customers"); customerData.forEach((customer) =&gt; { const request = objectStore.add(customer); request.onsuccess = (event) =&gt; { // event.target.result === customer.ssn; }; }); </code></pre> <p>The result of a request generated from a call to add() is the key of the value that was added. So in this case, it should equal the ssn property of the object that was added, since the object store uses the ssn property for the key path. Note that the add() function requires that no object already be in the database with the same key. If you're trying to modify an existing entry, or you don't care if one exists already, you can use the put() function, as shown below in the Updating an entry in the database section.</p> <h3>Removing data from the database</h3> <p>Removing data is very similar:</p> <pre><code>const request = db .transaction(["customers"], "readwrite") .objectStore("customers") .delete("444-44-4444"); request.onsuccess = (event) =&gt; { // It's gone! }; </code></pre> <h3>Getting data from the database</h3> <p>Now that the database has some info in it, you can retrieve it in several ways. First, the simple get(). You need to provide the key to retrieve the value, like so:</p> <pre><code>const transaction = db.transaction(["customers"]); const objectStore = transaction.objectStore("customers"); const request = objectStore.get("444-44-4444"); request.onerror = (event) =&gt; { // Handle errors! }; request.onsuccess = (event) =&gt; { // Do something with the request.result! console.log(`Name for SSN 444-44-4444 is ${request.result.name}`); }; </code></pre> <p>That's a lot of code for a "simple" retrieval. Here's how you can shorten it up a bit, assuming that you handle errors at the database level:</p> <pre><code>db .transaction("customers") .objectStore("customers") .get("444-44-4444").onsuccess = (event) =&gt; { console.log(`Name for SSN 444-44-4444 is ${event.target.result.name}`); }; </code></pre> <p>See how this works? Since there's only one object store, you can avoid passing a list of object stores you need in your transaction and just pass the name as a string. Also, you're only reading from the database, so you don't need a "readwrite" transaction. Calling transaction() with no mode specified gives you a "readonly" transaction. Another subtlety here is that you don't actually save the request object to a variable. Since the DOM event has the request as its target you can use the event to get to the result property.</p> <p>Updating an entry in the database<br> Now we've retrieved some data, updating it and inserting it back into the IndexedDB is pretty simple. Let's update the previous example somewhat:</p> <pre><code>const objectStore = db .transaction(["customers"], "readwrite") .objectStore("customers"); const request = objectStore.get("444-44-4444"); request.onerror = (event) =&gt; { // Handle errors! }; request.onsuccess = (event) =&gt; { // Get the old value that we want to update const data = event.target.result; // update the value(s) in the object that you want to change data.age = 42; // Put this updated object back into the database. const requestUpdate = objectStore.put(data); requestUpdate.onerror = (event) =&gt; { // Do something with the error }; requestUpdate.onsuccess = (event) =&gt; { // Success - the data is updated! }; }; </code></pre> <p>So here we're creating an objectStore and requesting a customer record out of it, identified by its ssn value (444-44-4444). We then put the result of that request in a variable (data), update the age property of this object, then create a second request (requestUpdate) to put the customer record back into the objectStore, overwriting the previous value.</p> <p>Note: In this case we've had to specify a readwrite transaction because we want to write to the database, not just read from it.</p> <p>Using a cursor<br> Using get() requires that you know which key you want to retrieve. If you want to step through all the values in your object store, then you can use a cursor. Here's what it looks like:</p> <pre><code>const objectStore = db.transaction("customers").objectStore("customers"); objectStore.openCursor().onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { console.log(`Name for SSN ${cursor.key} is ${cursor.value.name}`); cursor.continue(); } else { console.log("No more entries!"); } }; </code></pre> <p>The openCursor() function takes several arguments. First, you can limit the range of items that are retrieved by using a key range object that we'll get to in a minute. Second, you can specify the direction that you want to iterate. In the above example, we're iterating over all objects in ascending order. The success callback for cursors is a little special. The cursor object itself is the result of the request (above we're using the shorthand, so it's event.target.result). Then the actual key and value can be found on the key and value properties of the cursor object. If you want to keep going, then you have to call continue() on the cursor. When you've reached the end of the data (or if there were no entries that matched your openCursor() request) you still get a success callback, but the result property is undefined.</p> <p>One common pattern with cursors is to retrieve all objects in an object store and add them to an array, like this:</p> <pre><code>const customers = []; objectStore.openCursor().onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { customers.push(cursor.value); cursor.continue(); } else { console.log(`Got all customers: ${customers}`); } }; </code></pre> <p>Note: Alternatively, you can use getAll() to handle this case (and getAllKeys()). The following code does precisely the same thing as above:</p> <pre><code>objectStore.getAll().onsuccess = (event) =&gt; { console.log(`Got all customers: ${event.target.result}`); }; </code></pre> <p>There is a performance cost associated with looking at the value property of a cursor, because the object is created lazily. When you use getAll() for example, the browser must create all the objects at once. If you're just interested in looking at each of the keys, for instance, it is much more efficient to use a cursor than to use getAll(). If you're trying to get an array of all the objects in an object store, though, use getAll().</p> <h3>Using an index</h3> <p>Storing customer data using the SSN as a key is logical since the SSN uniquely identifies an individual. (Whether this is a good idea for privacy is a different question, and outside the scope of this article.) If you need to look up a customer by name, however, you'll need to iterate over every SSN in the database until you find the right one. Searching in this fashion would be very slow, so instead you can use an index.</p> <pre><code>// First, make sure you created index in request.onupgradeneeded: // objectStore.createIndex("name", "name"); // Otherwise you will get DOMException. const index = objectStore.index("name"); index.get("Donna").onsuccess = (event) =&gt; { console.log(`Donna's SSN is ${event.target.result.ssn}`); }; </code></pre> <p>The "name" index isn't unique, so there could be more than one entry with the name set to "Donna". In that case you always get the one with the lowest key value.</p> <p>If you need to access all the entries with a given name you can use a cursor. You can open two different types of cursors on indexes. A normal cursor maps the index property to the object in the object store. A key cursor maps the index property to the key used to store the object in the object store. The differences are illustrated here:</p> <pre><code>// Using a normal cursor to grab whole customer record objects index.openCursor().onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // cursor.key is a name, like "Bill", and cursor.value is the whole object. console.log( `Name: ${cursor.key}, SSN: ${cursor.value.ssn}, email: ${cursor.value.email}`, ); cursor.continue(); } }; // Using a key cursor to grab customer record object keys index.openKeyCursor().onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // cursor.key is a name, like "Bill", and cursor.primaryKey is the SSN. // No way to directly get the rest of the stored object. console.log(`Name: ${cursor.key}, SSN: ${cursor.primaryKey}`); cursor.continue(); } }; </code></pre> <h3>Specifying the range and direction of cursors</h3> <p>If you would like to limit the range of values you see in a cursor, you can use an IDBKeyRange object and pass it as the first argument to openCursor() or openKeyCursor(). You can make a key range that only allows a single key, or one that has a lower or upper bound, or one that has both a lower and upper bound. The bound may be "closed" (i.e., the key range includes the given value(s)) or "open" (i.e., the key range does not include the given value(s)). Here's how it works:</p> <pre><code>// Only match "Donna" const singleKeyRange = IDBKeyRange.only("Donna"); // Match anything past "Bill", including "Bill" const lowerBoundKeyRange = IDBKeyRange.lowerBound("Bill"); // Match anything past "Bill", but don't include "Bill" const lowerBoundOpenKeyRange = IDBKeyRange.lowerBound("Bill", true); // Match anything up to, but not including, "Donna" const upperBoundOpenKeyRange = IDBKeyRange.upperBound("Donna", true); // Match anything between "Bill" and "Donna", but not including "Donna" const boundKeyRange = IDBKeyRange.bound("Bill", "Donna", false, true); // To use one of the key ranges, pass it in as the first argument of openCursor()/openKeyCursor() index.openCursor(boundKeyRange).onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // Do something with the matches. cursor.continue(); } }; </code></pre> <p>Sometimes you may want to iterate in descending order rather than in ascending order (the default direction for all cursors). Switching direction is accomplished by passing prev to the openCursor() function as the second argument:</p> <pre><code>objectStore.openCursor(boundKeyRange, "prev").onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // Do something with the entries. cursor.continue(); } }; </code></pre> <p>If you just want to specify a change of direction but not constrain the results shown, you can just pass in null as the first argument:</p> <pre><code>objectStore.openCursor(null, "prev").onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // Do something with the entries. cursor.continue(); } }; </code></pre> <p>Since the "name" index isn't unique, there might be multiple entries where name is the same. Note that such a situation cannot occur with object stores since the key must always be unique. If you wish to filter out duplicates during cursor iteration over indexes, you can pass nextunique (or prevunique if you're going backwards) as the direction parameter. When nextunique or prevunique is used, the entry with the lowest key is always the one returned.</p> <pre><code>index.openKeyCursor(null, "nextunique").onsuccess = (event) =&gt; { const cursor = event.target.result; if (cursor) { // Do something with the entries. cursor.continue(); } }; </code></pre> <p>Please see "IDBCursor Constants" for the valid direction arguments.</p> <h3>Version changes while a web app is open in another tab</h3> <p>When your web app changes in such a way that a version change is required for your database, you need to consider what happens if the user has the old version of your app open in one tab and then loads the new version of your app in another. When you call open() with a greater version than the actual version of the database, all other open databases must explicitly acknowledge the request before you can start making changes to the database (an onblocked event is fired until they are closed or reloaded). Here's how it works:</p> <pre><code>const openReq = mozIndexedDB.open("MyTestDatabase", 2); openReq.onblocked = (event) =&gt; { // If some other tab is loaded with the database, then it needs to be closed // before we can proceed. console.log("Please close all other tabs with this site open!"); }; openReq.onupgradeneeded = (event) =&gt; { // All other databases have been closed. Set everything up. db.createObjectStore(/* … */); useDatabase(db); }; openReq.onsuccess = (event) =&gt; { const db = event.target.result; useDatabase(db); }; function useDatabase(db) { // Make sure to add a handler to be notified if another page requests a version // change. We must close the database. This allows the other page to upgrade the database. // If you don't do this then the upgrade won't happen until the user closes the tab. db.onversionchange = (event) =&gt; { db.close(); console.log( "A new version of this page is ready. Please reload or close this tab!", ); }; // Do stuff with the database. } </code></pre> <p>You should also listen for VersionError errors to handle the situation where already opened apps may initiate code leading to a new attempt to open the database, but using an outdated version.</p> https://tedneward.github.io/Research/storage/janusgraph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/janusgraph/index.html <p><a href="https://janusgraph.org/">Website</a> | <a href="https://github.com/janusgraph/janusgraph">Website</a></p> https://tedneward.github.io/Research/storage/junodb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/junodb/index.html <p><a href="https://github.com/paypal/junodb">Source</a></p> https://tedneward.github.io/Research/storage/kvass/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/kvass/index.html <p><a href="https://github.com/maxmunzel/kvass">Source</a></p> <p>As in, run it from the CLI to store any binaries of desire.</p> <h2>Simple usage</h2> <pre><code>$ kvass set hello world $ kvass get hello world # enumerate keys $ kvass ls hello # store arbitrary files $ kvass set logo &lt; kvass.jpg $ kvass get logo &gt; kvass.jpg # Its trivial to set up and operate kvass across multiple devices $ ssh you@yourserver.com kvass config show Encryption Key: 5abf59f5f1a2f3c998a4f592ce081a23e14a68fd8a792259c6ec0fc1e8fb1246 # &lt;- copy this for the next step ProcessID: 752176921 Remote: (None) $ kvass config key 5abf59f5f1a2f3c998a4f592ce081a23e14a68fd8a792259c6ec0fc1e8fb1246 # set the same key for all your devices $ kvass config remote yourserver.com:8000 # tell kvass where to find the server instance # Run "kvass serve" on your server using systemd, screen or the init system of your choice (runit, anyone?). You can specify the interface and port to host at with [--bind]. $ kvass serve --bind="0.0.0.0:80" # host on the default HTTP port (which means you can generate cleaner URLs - just set your remote no port) # every set will now be broadcasted to the server $ kvass set "hello from the other side" hello $ ssh you@yourserver kvass get "hello from the other side" hello # and every get will check the server for updates $ ssh you@yourserver kvass set hello 👋 $ kvass get hello 👋 # Good to know: All communication between the client and server is authenticated and encrypted using AES-256 GCM. # remember the file we stored earlier? Let's get a shareable url for it! $ kvass url logo http://demo.maxnagy.com:8000/get?q=OQMwTQmFCz6xiWxFxt4Mkw # you can also print the corresponding qr code directly to your terminal kvass qr logo </code></pre> https://tedneward.github.io/Research/storage/limbo/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/limbo/index.html <p><a href="https://github.com/tursodatabase/limbo">Source</a> | <a href="https://avi.im/blag/2024/faster-sqlite/">Paper: “Serverless Runtime / Database Co-Design With Asynchronous I/O”</a></p> <h2>Articles/Blog Posts</h2> <p><a href="https://avi.im/blag/2024/faster-sqlite/">"In search of a faster SQLite"</a></p> https://tedneward.github.io/Research/storage/mapdb/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/mapdb/index.html <p><a href="https://mapdb.org/">Website</a> | <a href="https://github.com/jankotek/mapdb/">Source</a> | <a href="https://jankotek.gitbooks.io/mapdb/content/">Docs</a></p> <p>Hello world:</p> <pre><code>//import org.mapdb.* DB db = DBMaker.memoryDB().make(); ConcurrentMap map = db.hashMap("map").make(); map.put("something", "here"); </code></pre> https://tedneward.github.io/Research/storage/memgraph/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/memgraph/index.html <p><a href="https://memgraph.com/">Website</a> | <a href="https://github.com/memgraph/memgraph">Source</a> | <a href="https://memgraph.com/docs">Docs</a></p> <p>Docker run instructions:</p> <pre><code>docker run -it -p 7687:7687 -p 3000:3000 memgraph/memgraph-platform </code></pre> <p><a href="https://memgraph.com/docs/memgraph/getting-started">Getting Started</a> instructions. Seems relatively standard.</p> https://tedneward.github.io/Research/speaking/speakingio/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/speakingio/index.html <p><a href="https://speaking.io/">Website</a></p> <h3>Outline</h3> <ol> <li> <p>First: plan out your talk.</p> <ol> <li><a href="https://speaking.io/plan/writing-a-cfp/">Writing the CFP</a>: The conference will have a Call For Papers. You'll need to write one.</li> <li><a href="https://speaking.io/plan/repetition/">Repetition</a>: Here's a trick of the trade: repetition. Repetition is a trick of the trade.</li> <li><a href="https://speaking.io/plan/number-of-slides/">The Number of Slides</a>: How many slides should you plan for?</li> <li><a href="https://speaking.io/plan/an-outline/">Outlining</a>: How do I get started on my talk?</li> <li><a href="https://speaking.io/plan/an-idea/">The Idea</a>: What should you talk about? What's a good topic for you?</li> <li><a href="https://speaking.io/plan/finding-a-talk/">Finding Where to Talk</a>: How do you even get started, anyway? Where can I give a talk?</li> <li><a href="https://speaking.io/plan/talks-are-entertainment/">Talks are Entertainment</a>: Public speaking's dirty little secret.</li> </ol> </li> <li> <p>Design and build your slides.</p> <ol> <li><a href="https://speaking.io/design/typography/">Typography</a>: What do the words on your slides look like? Hint: they're probably a disaster.</li> <li><a href="https://speaking.io/design/transitions/">Slide Transitions and Animations</a>: What should your animations look like?</li> <li><a href="https://speaking.io/design/software/">Software</a>: What software should I build my slide deck with?</li> <li><a href="https://speaking.io/design/dimensions/">Dimensions</a>: Should your slides be 4:3 or widescreen?</li> <li><a href="https://speaking.io/design/color/">Color</a>: How do I choose a color scheme?</li> <li><a href="https://speaking.io/design/using-images/">Using Images</a>: Photos and clip art and images, oh my.</li> <li><a href="https://speaking.io/design/duplicating-objects/">Duplicating Objects</a>: Learn this one weird trick to build your slides quickly.</li> </ol> </li> <li> <p>Prep for the big day.</p> <ol> <li><a href="https://speaking.io/prep/scoping-out-the-room/">Scoping out the Room</a>: You're just about to go up on stage. Scope the room out.</li> <li><a href="https://speaking.io/prep/recording-your-talk/">Recording Your Talk</a>: Think about recording your talk as practice, and for others.</li> <li><a href="https://speaking.io/prep/practicing-it/">Practicing It</a>: At some point you should practice. Quit putting it off, you know I'm right.</li> <li><a href="https://speaking.io/prep/live-demos/">Live Tech Demos</a>: What's the best way to prepare a live demo in front of an audience?</li> <li><a href="https://speaking.io/prep/presenter-remotes/">The Best Presenter Remotes</a>: Thinking about controlling your slides with a remote?</li> </ol> </li> <li> <p>Deliver and do your thing.</p> <ol> <li><a href="https://speaking.io/deliver/your-voice/">Your Voice</a>: How fast should you talk? How slow? How can you command your voice?</li> <li><a href="https://speaking.io/deliver/what-they-dont-tell-you-about-public-speaking/">What They Don't Tell You</a>: What are the things to think about that you won't think about?</li> <li><a href="https://speaking.io/deliver/remotes/">Remotes</a>: A quick little guide on navigating your deck with a remote.</li> <li><a href="https://speaking.io/deliver/nervousness/">Dealing with Nervousness</a>: Everyone gets nervous.</li> <li><a href="https://speaking.io/deliver/fucking-up/">Fucking up</a>: All the preparation in the world won't save you from a brainfart.</li> </ol> </li> <li> <p>React and reflect on what just happened.</p> <ol> <li><a href="https://speaking.io/react/sharing-your-talk-online/">Sharing Your Talk Online</a>: Turns out your biggest audience wasn't even in the room with you.</li> <li><a href="https://speaking.io/react/improving/">Improving a Talk</a>: C'mon, you're not perfect. You probably messed something up.</li> <li><a href="https://speaking.io/react/handling-questions/">Handling Questions</a>: What's the trick behind Q&amp;A after a talk?</li> </ol> </li> </ol> https://tedneward.github.io/Research/speaking/kick-ass-guide-to-presentations/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/kick-ass-guide-to-presentations/index.html <p><em>(by Brian Stampfl)</em></p> <p><strong>Kick Ass Key Factor #1:</strong> Know Your Stuff</p> <p><strong>Kick Ass Key Factor #2:</strong> Build Your Presentation So You Can't Forget Your Stuff</p> <h1>Chapter 1: Becoming Bulletproof</h1> <p>The difference between a good presentation and an awesome one is a small amount of extra work, attention to detail, and a willingness to take a chance.</p> <h1>Chapter 2: Overcoming Fear</h1> <p>Some basic generalizations of things that scare us when we set foot in front of a live audience:<br> 1. Participation is not an option; you're mandated to speak.<br> 2. Screwing up is equal to death. Embarrassment is a uniquely human emotion.<br> 3. Forgetting your own name.<br> 4. Stinking at presentations. This is a self-fulfilling prophecy.<br> 5. Generally unidentifiable fear.</p> <p>Fear is a clue we need to prepare better.</p> <p>Realizations:<br> * We're all friends here. Your classmates are your friends; everyone wants you to succeed; nobody cares if you make a mistake; they are worried about their own presentations, not you.<br> * Yeah, but I don't know these people. Strangers in the audience are one handshake from being your friend; the audience wants you to succeed; they don't care if you make a mistake; any audience wants to learn, be entertained, and be supportive.<br> * But I'm worried what people will think.</p> <h1>Chapter 3: Entertaining a Room Full of Lawyers</h1> <p>The best presentations are actually performances.</p> <h1>Chapter 4: Studying the Greats -- Develop Your Style</h1> <p>Watch the pros, figure out what they're doing right. When you see something that grabs your attention, figure out a way to take that thing with you.</p> <h1>Chapter 5: Where do I start?</h1> <p>Answer three critical questions:<br> 1. How much time are you given for your presentation?<br> 2. What is your method of delivery? Flash cards; reading from something written; memorization; PowerPoint; group presentation<br> 3. What is your topic?</p> <h1>Chapter 6: Brainstorming 101</h1> <p>Brainstorming steps:<br> * Have pen and paper or whiteboard at the ready<br> * Agree that there are no dumb ideas; if an idea pops into your head, write it down<br> * As quickly as you can write, ark out ideas related to the general subject<br> * Use abbreviations if possible<br> * Write ideas down for at least ten minutes straight<br> * At the end of the brainstorm (ten minutes or end of the brain dump), pick out five topics that most closely fit what you need to speak about</p> <p>Now split the topic down into smaller chunks. Narrowing will make it feel less daunting and more interesting.</p> <h1>Chapter 7: Technically Speaking</h1> <p>(Technical discussion of using PowerPoint for PC, Mac, etc)</p> <h1>Chapter 8: The PowerPoint Build Begins</h1> <p>Your PowerPoint is not the presentation</p> <p>You'll never improve a poorly thought-out presentation by projecting garbage onto the screen.</p> <h1>Chapter 9: The Rule of Three</h1> <p>Expand each segment of your outline into three parts.</p> <p>Example:</p> <ul> <li>Beginning <ul> <li>Introduction: Yourself and the Topic <ul> <li>Name</li> <li>Short Bio</li> <li>Topic Name</li> </ul> </li> <li>Preview: Tell the Audience What to Expect <ul> <li>Describe the Problem</li> <li>Explain Popular Opinions</li> <li>Why to Think My Way</li> </ul> </li> <li>Historical Background: Ensures Everybody Knows the Backstory <ul> <li>In 1945, Thing Occurred</li> <li>Company "X" Responded This Way</li> <li>Today, the Problem is This</li> </ul> </li> </ul> </li> <li>Middle <ul> <li>Description of What is Happening in the World <ul> <li>This Thing is Occurring</li> <li>This is the Result</li> <li>Long-Term Ramifications</li> </ul> </li> </ul> </li> </ul> <p>The Three Types of Learners: Auditory, Visual, Hands-On</p> <p>The Three Types of Delivery: Bullet Points, Stories, Combination</p> <h1>Chapter 10: How to Create Kick Ass Content</h1> <p>Start with an Explosion: Grab attention right away. The Question; The Promise; The Unfinished Story; the Fascinating Facts.</p> <h1>Chapter 11: How Many Slides Should I Create?</h1> <p>You shouldn't be asking how many slides you should create, but rather how long will it take to go through the slides you've created.</p> <p>As a guideline, one slide for each minute you have to speak.</p> <p>Building slides takes much longer than most people think. Be prepared to put in the time.</p> <h1>Chapter 12: Building the Framework</h1> <p>Ge the framework of your presentation built:<br> 1. Create a folder on your computer that contains all the pictures, logos, videos, and a copy of your outline. Importing all the information from one place makes things easier.<br> 2. Create as many blank slides as you need. Don't worry about making too many or not enough.<br> 3. Import all your photos. Don't worry about the order in which you bring them in, and don't worry about sizing or fine-tuning. Just get them into the slides.<br> 4. Import your video. Make sure that the original video is stored and remains in the same folder as Step 1.<br> 5. Copy and paste the lines of text from the outline into each slide.<br> 6. Arrange the slides.</p> <h1>Chapter 13: Text</h1> <p>DO NOT paste large blocks of text into your presentation. No audience wants to read a book projected onto a screen.</p> <p>Text is not a way of conveying information to your audience, but visual cues and reminders of the information you're sharing with them.</p> <p>PowerPoint is designed to support you, not carry you</p> <p>Font Color: easiest color combo is black text on white background; most dark colored fonts should work well on a white background. Second best combo is white text on a black background--there isn't as much reflection from the screen.</p> <h1>Chapter 14: Photograph File Size and Resolution</h1> <p>Best bet is to adjust the images resolution so that the longest dimension is around 1k pixels. Use high quality photos in your presentation. Brighten up photos; projectors never project the same level of brightness as an LCD screen. Highlight details on a picture; insert objects such as circles or arrows into your slides to highlight the detail you want your audience to focus on.</p> <h1>Chapter 15: Video</h1> <p>Top four ways video can kill your presentation:<br> 1. Poor quality<br> 2. Lost in transfer<br> 3. Linking problems<br> 4. Not embedding videos</p> <h1>Chapter 16: Transitions and Animations</h1> <p>No amount of smoke and mirrors can make up for bad content. Avoid them.</p> <h1>Chapter 17: Object Alignment</h1> <p>It's important that the text on the slide aligns vertically into the same place as the text on the slide previous to it. Otherwise the text (or logo or whatever) will appear to be "jumping" from one slide to the next.</p> <p>Visual balance: fill the frame. Think of composing the slide the same way we compose a photo.</p> <h1>Chapter 18: Relevancy</h1> <p>It may be time to remove some non-vital content. The reason presenters include irrelevant (crowd-pleasing) content is because the presenter is worried what they have to say is not interesting and they're afraid they might not please the crowd. Revisit your delivery. Are you presenting to the audience, or merely talking at them? Often the topic may not be particularly interesting in and of itself, but the passion, knowledge of the material, and delivery that makes it worth listening to.</p> <h1>Chapter 19: Props</h1> <p>Performance props: yet another means of relaying your message. It will remain in your control the entire time. While you may draw attention to the item you're using, you should not have to describe what the item is. The performance prop is a tool, and it becomes an extension of your story. No matter what your performance prop is, knowing ahead of time how you will hold the item, move with it, how you'll introduce it, and even when you might set the item down should all be thought out ahead of time. At a minimum, practice with your performance prop at least once or twice.</p> <p>Show and Tell props: those items relevant to your topic that you will hold up to the audience and actually describe. The benefit of show-and-tell props is that you can present items to the audience that might otherwise be difficult to describe. Or the item might be so unique or personal that your mere possession of the item makes it special. You'll want to know ahead of time how you'll actually handle the item while you're speaking.</p> <h1>Chapter 20: All the Gadgets</h1> <p>Slide advancement: remote control or keyboard?</p> <p>Microphones and podiums.</p> <p>Laser pointers. Don't use them.</p> <h1>Chapter 21: Room Configuration</h1> <p>If you have any control where your audience will sit, you'll want them as far forward and close together as possible. By doing so, you'll improve the odds of a group dynamic occurring; for example, we know that laughter is contagious. What you want with your audience is connection: laughter is just one benefit of connection. Obtaining that connection with your audience is a major key to success.</p> <p>Lights on or off? Ensure the screen image is not washed out. Keep a few lights on so that those taking notes can see.</p> <p>Handouts. Pass them out early, keep them physically linked (stapled, bound, etc).</p> <h1>Chapter 22: What Comes Out of Your Mouth</h1> <p>Command Presence. Knowledge of Material, Solid Delivery, Passion. Command Presence is not a loud voice mowing over your audience. It's an attitude.</p> <p>Know Your Stuff--But Don't Memorize Your Stuff. Memorizing your presentation is the equivalent of turning your presentation into a school play. Anyone who memorizes their script tends to sound as if they have memorized a script; it's very easy to detect and a turn-off for listeners. There is no built-in emotion, inflection, or motivation that would drive the characters; despite the speakers' best efforts, the result is a monotone delivery. Those who memorize their material will often show signs of trying to recall the information. This usually presents itself in the form of the speaker looking up and to the left in an attempt to recall their next line. This will generally override any attempt to connect with your audience.</p> <h1>Chapter 23: The Screen Is Not Your Audience</h1> <p>While looking up/back at the screen, speakers will sometimes become mesmerized by their own images, or are simply too dependent on the notes they've created for themselves. The speaker becomes drawn to the screen and forgets to turn back around and address their audience. The audience then has to look at the presenter's backside for the entire presentation. Glancing up to the screen for a few moments, and even directing your audience's attention to specific details is completely acceptable. But don't forget where your audience is sitting and be sure to speak to them.</p> <h1>Never Read To Your Audience.</h1> <ul> <li>The Audience Knows How to Read; if you must put large amounts of text on the screen, let the audience read it themselves--direct their attention to the text, then give them time to do so.</li> <li>Most People Stink at Reading Out Loud.</li> <li>Large Amounts of Text Screams Laziness</li> </ul> <p>Crutch words (ah, um, you know, etc)</p> <h1>Chapter 24: Practice for Time</h1> <p>Speed Kills: most people tend to speak faster than they realize when they're first learning the craft of public speaking.</p> <p>Efficiency of Words: What you plan to say to your audience and what you actually say may turn out to be two different things. Your brain starts streamlining your message and speaking in your natural voice, abandoning all the extra words that you don't use in normal speech.</p> <p>One More Question: Q&amp;A can be the best parts of your presentation, but unless you are experienced with adjust your presentation time on the fly, hold off on all questions until the end.</p> <h1>Chapter 25: Humor</h1> <p>Humor comes with great risk!</p> <p>Humor either works or it doesn't.</p> <p>Know your audience</p> <p>Stop the bleeding</p> <h1>Chapter 26: Body Talk</h1> <p>Your body needs to deliver your message as well as your words. The best way to counteract what your body might be saying outside your awareness is to take control of what your body is doing.</p> <p>If public speaking scares us, our bodies are likely doing a fight-or-flight response.</p> <p>The Distance to Here. Where you stand in relation to your audience will reflect your level of comfort and confidence. Closing the gap between the audience and ourselves is the opposite of what fear would have us do; the nervous person may find themselves creating distance from the audience as they search for some unidentified comfort zone. Luckily for us, most speaking environments limit the speaker's ability to step too far away.</p> <p>Building Barriers. Placing an object between yourself and the audience is a primal response designed to protect us from a perceived threat. Shielding can be accomplished by standing behind the podium, a microphone stand, a table, or any object available to the speaker. Less obvious methods of shielding involve the speaker putting their hands into their pockets, the fig leaf pose, or folding their arms across their chest.</p> <p>What Do I Do With My Hands? For Men: avoid standing with your hands folded over one another in front of your crotch--the fig leaf look is generally reserved for wedding photos; avoid contact with your body that could be described by the words picking or digging; refrain from any attempt to adjust your underwear; keep your hands away from your nose. For Women: women tend to overuse their hands and body to communicate--while the movements are generally natural, the line between conveying thought and aerobic workout is not always clear.</p> <p>So what should you do with your hands? Let them move freely as you speak. Gestures will allow you to put emphasis on points, relay emotions, and even create imagery for your audience. Forcing your hands to remain at your sides or some other place will feel and look unnatural; once you allow yourself the freedom of movement, while avoiding the previously labeled pitfalls, you'll find that your audience will see you as being comfortable, and you'll feel more relaxed yourself.</p> <p>Focus on Your Eyes. Scan Wide, Scan Deep; see the full width and depth of the crowd, so visualize the entire seating area divided into sections, and focus your attention on one section at a time, then move to the next. What Should I See? Pick out people that are looking directly at you; make eye contact with them, hold it for a few moments, then move on to the next person. Once that momentary contact is made, you'll know instinctively what impact you are having on that person.</p> <p>Control Your Body. Be careful that walking around doesn't transition into pacing. Adjust your environment if necessary (within reason).</p> <h1>Chapter 27: What Not To Wear</h1> <p>Old People and Teachers Want You to Dress Nice. Look Like You Know What You're Talking About.</p> <h1>Chapter 28: The Art of Audience Engagement</h1> <p>Three Types of Crowds<br> * <em>The Hot Crowd:</em> these are the fun ones, and require little or no work on your part to get them to participate.<br> * <em>The Warm Crowd:</em> just good folks who are interested in hearing what you have to say, but didn't necessarily plan on being part of the presentation. You'll need to establish a rapport with these people before you can have a running dialogue with them.<br> * <em>The Cold Corpse Crowd:</em> this is the type of audience that exists when attendance is mandated and the topics are of little interest to them. When asking a question of the crowd, be prepared that you may not get any responses. When this happens, simply be patient, stand quietly, and wait for a response. Scan the room and make eye contact with as many people as you can and wait. The waiting will serve three purposes: first, it will serve as a message to your audience that you will be posing questions to them and that you'll be standing by for answers; second, your pause will create an awkward tension in the room, and someone may say something just to break the tension; third, the pause will allow you to take advantage of the human brain's inability to not answer questions (the brain simply can't NOT answer a question if it's able to). Your job is to give the audience permission to say what they're thinking. Waiting for their answer is that permission.<br> * <em>Help! I can't crack this nut!</em> If the audience refuses to interact with you, you'll simply need to interact with yourself.<br> Taming a Wild Audience. There's always a chance you'll encounter that one person who is dying to participate. Others are awkwardly inquisitive, some can't sit quietly for ten minutes, a few are obnoxious, and once in a while, the audience simply takes on a life of its own.<br> * <em>The Green Gorilla Guy.</em> This is the person that asks the impossible hypothetical questions about your topic. (The term refers to the absurdity of the question itself.) Resolution: Keep in mind that although the question might sound goofy to you, the person may be sincere in asking. Not all things obvious to you are obvious to others. Sensitivity is key.<br> * <em>The Private Conversation Guy.</em> This person will raise their hand not to ask questions about your topic, but rather to ask questions that are of personal interest to themselves only. Resolution: This one is easy since the person appears to have some interest in your expertise. Simply give a short, quick answer to their question and then invite them to speak with you at the end of the presentation.<br> * <em>I Have a Question, and a Better Answer Guy.</em> Asking a question designed to offer an opportunity to show off the questioners' own esoteric knowledge. Resolution: Some people are just dying for attention. The best approach to this problem is to publicly acknowledge the questioner's knowledge and redirect back to the presentation.<br> * <em>The Rowdy Crowd Takeover.</em> Once in a while you'll come across a group that gets caught up in their own self-generated entertainment, cracking jokes back and forth. Resolution: Luckily, this silliness usually lasts for less than a minute. Have fun with it. Let it play out until the laughter dies down. Then break back in and continue on. If you experience repeat offenses to the point of disruption, let the audience know you have a limited amount of time and need to continue on.<br> * <em>The Non-Stop Question Guy.</em> Resolution: Once you see the pattern, redirect that person's questions to the end of the presentation.<br> * <em>The Confronter.</em> This person takes advantage of having the attention of a large group of people to address their personal needs or complaints. Resolution: acknowledge the confronter's concerns as legitimate, bring it back around to the larger topic in general (leaving open room for discussion or dissent), and invite the confronter to discuss the issue privately following the event.</p> <h1>Chapter 29: Crash and Burn--When Things Go Wrong</h1> <p>What the crowd don't know, the crowd don't know. Fix or dismiss the problem. Move on. The audience doesn't care if you make a mistake. They want you to succeed.</p> https://tedneward.github.io/Research/speaking/ultimate-public-speaking-survival-guide/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/ultimate-public-speaking-survival-guide/index.html <p><em>(by Ramakrishna Reddy)</em></p> <h1>Chapter 1: Psychological Barriers</h1> <p>Public speaking is overwhelming; how can I understand it in simple terms? Did you know how the drive the first time you drove a car? You start off learning primary activities such as acceleration, braking, changing gears, and so on; in speaking, you learn primary activities such as content creation, eye contact, voice modulation, hand gestures, facial expressions, stance and so on. In order to make the drive interesting and enjoyable, you can equip your vehicle with stereo system, adjustable seats, etc; in speaking, you can tell stories, create humor and use visual aids. When driving, unless you know where you are going, you waste yours and your passengers' time; if you do not have a purpose in mind when you speak, you do the same.</p> <p>Fear strikes me whenever I want to give a presentation; what should I do? Fear strikes everyone. You must start by accepting that the fear of public speaking is the fear of the unknown and fear of ridicule. Write down your questions/fears and examine them.</p> <p>What will other people think if I make a mistake during my presentation? People respect you when you speak in front of an audience. People only think about you as much as you think they think of you. Two secrets: Speak as often as possible and at every given opportunity; take the time to reflect on every experience by asking what went right, what went wrong, and for suggestions on how to do it better next time.</p> <p>Can I really sizzle on stage even though I don't have an eye-catching look? Yes.</p> <p>Even though speaking comes naturally to me, do I still need to prepare my presentation? Yes. Do not assume that you need not prepare because you have notes; you have to be comfortable with your content. Your words must roll smoothly off your tongue.</p> <p>Can I still give an effective presentation without a strong vocabulary? Using simple words that we speak in day-to-day conversations is the secret to a well-received presentation. Your audience will judge you based on the value of what you say, rather than the glamor of your words.</p> <h1>Chapter 2: Speech Creation</h1> <p>I know my topic but I feel stuck; what am I missing? Feeling stuck will stop you from taking action. What is the purpose of the presentation (general-purpose (inform, entertain, persuade or inspire), and specific)? A specific purpose should be clear, short and add value to your audience. It is very important to determine your specific purpose. A simple way to formulate your specific purpose is by knowing what you want your audience to feel, think or do after your presentation.</p> <p>How do I start creating content for my presentation? Basic skeletal structure: an opening, context setting, key points, support points, application, summary, and conclusion.</p> <ul> <li>Opening is what you say in the first 30/60 seconds.</li> <li>Context setting is explaining what you are going to tell them. Sometimes this part gets merged into your opening.</li> <li>Key points help to reinforce the specific purpose of your talk. 3-4 key points per talk.</li> <li>Supporting points substantiate or prove a key point. Use stories, statistics, illustrations, or facts for support.</li> <li>Application tells the audience how to apply the key point.</li> <li>Summary drives home what you have told them and recaps your presentation.</li> <li>Conclusion is the showstopper for your presentation. Final 30/60 seconds of your talk.</li> </ul> <p>How do I create a killer opening for my talk? The first 30/60 seconds of your presentation determines how well the rest of the presentation will go. Four keys for a killer opening:</p> <ol> <li>Start with a current observation about your surroundings and the occasion for which you are speaking.</li> <li>Start with a question.</li> <li>Start with a startling statistic.</li> <li>Start with a personal story or experience.</li> </ol> <p>Some specific strategies: take a few seconds to scan your audience before uttering the opening words; do not start with fillers like "the weather is good" or "what a sunny day"; build trust, be open to your audience, by using open palms and a gentle voice to open up and start building trust; listen to your audience and acknowledge any response; be relaxed and comfortable; make them like you by not showing a know-it-all attitude; tone down the entertainment factor for a business presentation.</p> <p>What is a context setting and how do I create one? Telling your audience what you are going to tell them; a promise you will eventually fulfill or a problem that you can solve during your speech. Specific steps:</p> <ol> <li>Context should be audience-focused; the audience should feel it is about them.</li> <li>Context should be clear and specific; the audience should be able to clearly understand your agenda.</li> <li>Go even further and add a specificity factor.</li> </ol> <p>How do I select the key points for my presentation? Interview and audience analysis.</p> <p>How do I support the key points of my presentation? Aristotle says that an effective speech has three key elements: ethos (the credibility of the speaker), logos (the reasoning behind the point), and pathos (the emotional attachment between the speaker and the audience). Use stories, statistics, facts, illustration (PowerPoint, Props, Writing board, Questions and facts).</p> <p>How to create a compelling story? Stories have the ability to transfer emotions. Elements to a powerful story:</p> <ul> <li> <p>Must-have elements</p> <ul> <li>Conflict</li> <li>A character with a desire (hero/heroine)</li> <li>Obstacles in that path</li> <li>Overcoming obstacles</li> <li>The point</li> </ul> </li> <li> <p>Good-to-have elements</p> <ul> <li>Event setting -- the time and place</li> <li>Character building -- what the characters look like, their vulnerabilities, etc</li> <li>Dialogue -- what the characters say to each other</li> <li>Humor -- optional, but humor makes a story more compelling</li> <li>Emotion -- knowing what the characters are feeling can make the difference between a good story and a great story</li> <li>Sensory words -- words that evoke the senses of touch, sound, smell, vision, taste</li> </ul> </li> </ul> <p>How do I best use transitions in my speech? Transitions maintain a smooth and clear flow between points in the speech. If the speech is a journey, transitions are the signboards. Transitions can be verbal, non-verbal (movement, pause, prompted gestures), or a combination of the two.</p> <p>How can I have an effective summary? Do's and Don'ts for effective Q&amp;A:</p> <ul> <li>Do take a pause and answer straight to the point.</li> <li>Do clarify any question which isn't clear. Reiterate the question to the audience.</li> <li>Do confirm your answer to all questions by saying, "Does that make sense?" or "Does that answer your question?"</li> <li>Do act confident. You might not know all the answers but you can get the answers.</li> <li>Don't try to be a know-it-all. If you are not sure, say so; but you can always go find out and get back to them.</li> </ul> <p>How do I create a memorable conclusion for my speech? Seven ideas to a memorable conclusion:</p> <ol> <li>Best Wishes Close</li> <li>Gratitude Close</li> <li>Answer Close -- if your presentation opens with a question, answer it</li> <li>Action Close -- call to action</li> <li>Dialogue Close</li> <li>Poem Close</li> <li>Circular Close -- bring the audience back to the starting point</li> </ol> <p>How do I refine my speech content? Four tools to refine your content:</p> <ol> <li>Choose simple words with the fewest syllables</li> <li>Use active voice</li> <li>Repeat key phrases</li> <li>Use rhetorical devices (similes, metaphors, alliterations)</li> </ol> <h1>Chapter 3: Speech Delivery</h1> <p>I know my content very well, so do I really need to care about anything else? Yes; what are your miscommunications? Bad (nervous) habits?</p> <p>How should I move during my presentation? Posture--straighten your back and head, broaden your shoulders, stand with firm footing. Movement--speakers should move only when they want to communicate an idea. Movement helps in the following areas:</p> <ul> <li>Transition: transition from one point to another</li> <li>Drama: depict the actual movement that happens in a scene</li> <li>Increase impact: putting a strong foot forward to convey an important and strong point, etc</li> </ul> <p>How do I make effective eye contact? While practicing your speech, close your eyes. Imagine your audience in your mind. Divide into quadrants. Imagine talking to one person in the first quadrant, looking them in the eye, for 5 to 10 seconds. Move to the second quadrant: find a person, look them in the eye for 5 to 10 seconds. Third, fourth, repeat.</p> <p>Is there a simple way to improve the quality of my voice? How to...</p> <ul> <li>... sound clear: speak every word with clarity. Make sure you pronounce every syllable of each word. Complete every word before you go on to the next word. Make sure you enunciate the endings of words properly. Make sure you maintain breath and energy throughout the sentence.</li> <li>... find your optimum tone: Practice your speech in a tone that you would use with a friend. Then gradually increase your tone from soft to loud. When you feel that your voice feels natural and resonant, that's your optimum tone.</li> <li>... control the pace of your speech: the human ear can grasp words easily when spoken at a range of 120 to 160 words per minute. Find your current pace of speech (start reading content and stop the timer after 1 minute) and adjust accordingly. Note that certain parts of the speech work great if spoken slowly.</li> <li>... pause effectively: practice pausing. Pauses help you connect with the audience, build anticipation before you deliver the punch line, gives time for the audience to laugh after you deliver the punch line, acts as a signal that you are going to say something very important, gives time for the audience to reflect on an important point, and acts as a transition tool for moving from one point to another.</li> <li>... use emotion: Feel the emotion in your heart before you express it through words.</li> <li>... say dialogue or monologue: use the emotion and attitude of the person speaking in the dialogue/monologue. You have to clearly distinguish the characters in a dialogue.</li> </ul> <p>How do I use my hands during the presentation? Two forms of gestures: emphatic and non-emphatic. Emphatic gestures are gestures to emphasize any point. Non-emphatic gestures help you describe an object/character, prompt the audience or depict the scene. Practice gestures until you internalize or have your speech in muscle memory. Video-record your speech; observe your gestures. Be comfortable enough on stage that you don't need to gesture--not gesturing (keeping your hands to your side) can add impact to certain parts of your presentation. Learning to speak without any gesture is a great skill to master.</p> <p>How to create correct facial expressions during the presentation? Facial expressions communicate a lot more than what you say verbally. Practice the content, then consciously allow the emotions to flow on your face.</p> <h1>Chapter 4: Preparation Steps</h1> <p>How can I create and maintain my connection with the audience? Tools for creating a connection:</p> <ul> <li>Your introduction. Plays a crucial role to build likability and trust. What expertise you bring to the table; why are you qualified to speak, without ego-boosting. How you can help your audience. An interest apart from the topic or any light remark.</li> <li>Use the pronouns you/we. Have as many "you"-focused sentences as possible. Frame content in such a way that you transition and involve the audience using "you" or "we" so that they understand that you care for them.</li> <li>Speak in the third person. Minimize the use of "I". Unless it is your own original thought, realization, finding, experience or learning, do not use "I". If the idea or teaching is someone else's, use him/her in the reference. My (your) job is to simplify and spread the information. This helps in establishing your credibility.</li> <li>Be sincere.</li> <li>Maintain genuine eye contact. Eyes are doorways to form connection. While presenting, there is a tendency for your eyes to glaze over the audience, but the moment you talk with an audience, you develop a deeper bonding.</li> <li>Engage your audience. Ask interactive questions. Invite a few of them to the stage. Create a group activity.</li> </ul> <p>Is there a particular strategy to ensure that the audience will continue to listen? If you are looking for just one strategy in particular that will ensure an audience will continue to listen to you, then, ENTERTAIN! If you entertain well, the audience can't stop loving you. Try self-deprecating humor; the audience will respect your more because you are ready to let your guard down and poke fun at yourself.</p> <p>What is the final checkpoint, before I freeze my content? The most important part in your content preparation is the clarity of your presentation.</p> <p>Can the venue affect my presentation? Yes, in various ways:</p> <ul> <li>Venue type: board room? Classroom-style? Auditorium?</li> <li>Time limit: are you going to be timed by someone else or will you manage it by yourself?</li> <li>Projector: check if the venue has a projector</li> <li>Audience seating position: audience is seated for making connection through eye contact; audience is concentrated together and not scattered (Never leave empty chairs for a presentation; energy dissipates in emptiness.)</li> <li>Acoustics</li> <li>Audibility: verify audibility with a friend/colleague by having them sit in all corners of the room</li> </ul> <p>Can you give me the exact steps to practice my speech? Steps for practice:</p> <ul> <li>Stand straight with firm footing. Keep your spine and head straight.</li> <li>Keep your hands at your sides with your fingers slightly curled.</li> <li>Keep your feet at around 8-12 inches distant from each other</li> <li>Don't worry about eye contact yet</li> <li>Practice your speech content until it smoothly rolls off your tongue. Refactor content if necessary.</li> <li>Check for any distracting mechanisms in your current speech.</li> <li>Next, practice by speaking your content in loud, soft, slow and at least in a fast tone. Make sure to enunciate the words clearly. Complete every word before going to the next.</li> <li>Check if you are starting the sentences in high energy and ending them in low energy.</li> <li>Start focusing on internalizing your speech. You should be able to tell the content even if somebody wakes you up in the middle of the night.</li> <li>If you are planning to pause at certain places in your talk, practice those pauses deliberately.</li> <li>Feel the emotion in your heart and practice.</li> <li>Ask a friend to give feedback on the relevance of your facial expressions to the intent of your speech</li> <li>Work on your eye contact; use the quadrant strategy in your imagination</li> <li>Let your hand gestures flow naturally. Practice the following rules of thumb: use open palms; do not point at the audience; move hands from shoulder level; stroke at the right word</li> <li>Work on your movements. Check your footwork; determine at what point of the stage you would be at different parts of your presentation.</li> <li>If you are using PPT, practice by sequencing the slides. Practice by asking your friend to navigate the slides for you. Or use a remote control for it.</li> <li>Rehearse by incorporating all the steps.</li> <li>Give a complete rehearsal to your close friends, family or others.</li> <li>Keep repeating the steps</li> </ul> <p>Can you give me a strategy for not going blank on stage? If the speech is in muscle memory, then you will be excited and ready. You will not blank.</p> <h1>Chapter 5: Presentation Day Steps</h1> <p>How to dress for my presentation? Groom well.</p> <p>Is there a checklist for the presentation day? Suggestions:</p> <p>Before going to the venue:</p> <ul> <li>Do eat light food. Breads and salads; you need energy to show energy in your presentation</li> <li>Do charge your laptop beforehand. Take your charger. If you have a PPT, have a backup.</li> <li>Do take a cab/car instead of public transport. If you are going to drive, start early to avoid traffic jams and parking problems</li> <li>Do hang around with a supportive friend. You need someone to bounce your content off. You will feel less stressful if there is a dependable person with you.</li> <li>Do reach the venue early.</li> </ul> <p>After going to the venue:</p> <ul> <li>Do chat with the presentation planner and check the venue setting</li> <li>Do check if the projector is working. Try to display your PPT.</li> <li>Do check if you have a proper writing board and a working marker (if you are going to use one)</li> <li>Do check for arrangement of chairs. Check if the number of chairs is in accordance with the number of people who have confirmed their presence.</li> <li>Do not forget to check the temp for the A/C.</li> <li>Do test the microphone</li> <li>Do video-record your presentation. Ask a friend/colleague to record. Or ask the presentation planner to record.</li> <li>Do check for any distraction because of the lighting. Are the lights falling on your eyes? Take a call on whether you want the lights on or off during your presentation.</li> <li>Do remind the presentation planner or the anchor to ask the audience members to keep their cell phones on silent</li> <li>Do not forget to hand over a copy of your introduction to the planner or anchor</li> <li>Do not forget to reach out and greet your audience members as they enter the venue</li> <li>Do converse with your audience after your greetings. Ask questions on what brings them there. Keep it quick and move on; it has to be natural</li> </ul> <p>What if my heart starts pounding, ears get heated and hands become cold, 10 minutes before the presentation? Steps to tackle last-minute anxiety:</p> <ul> <li>Rub your left palm with your right hand for 10-15 seconds. Reverse.</li> <li>Slowly, breathe in and breathe out.</li> <li>If necessary, go to a private corner where no one is around and do some vigorous punching and kicking in the air. This boosts energy and excitement.</li> <li>Read (and answer) the following questions: What is my intent? Am I present? Will I have fun? How would I give this presentation if I knew this was the one ever?</li> </ul> <p>How do I carry myself from seat to stage, once the anchor calls my name? Get up from your seat and spring to the stage; show enthusiasm because it is contagious. Wear a smile while you walk. As soon as you take position on stage, do not start off right away; instead, attract everyone's attention and create the connection before you start your speech. Stand and scan the audience for a good 4 to 5 seconds--your audience will stop doing other things and start focusing on you. Then begin.</p> <p>How do I handle myself during the actual presentation? Be present. Be passionate. Be energetic.</p> <p>How do I handle any unexpected problems during the actual presentation? Do not get flustered. Know that your audience will empathize if there is a problem and always want you to do well. Some example scenarios:</p> <ul> <li>Losing train of thought. Strategies for bouncing back: have some stock lines in place that can be used in case you lose your train of thought; even if you forget, do not try to go back and share the missed portion; remember the structure, instead of the words.</li> <li>Power shuts down. Just be cool.</li> <li>Cell phone ringing. Keep a few lines ready to create humor: "Tell your friend that I will call him later."</li> <li>Latecomer to your presentation. Ignore it, or be ready with some humor if it disrupts the talk.</li> </ul> <p>How do I carry myself off the stage after my presentation? After you say the last words, pause, smile, say goodbye internally, the audience will understand. Pass control to the anchor or MC. While you turn and walk off the stage, show genuine enthusiasm.</p> <p>What should I do after the presentation? Reflect. Get audience feedback. Give your own feedback. Analyze your video.</p> https://tedneward.github.io/Research/speaking/presentation-patterns/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/presentation-patterns/index.html <p><em>(by Neal Ford, Matthew McCullough, Nate Schutta)</em></p> <h1>Part 1: Prepare</h1> <h2>Chapter 1: Presentation Prelude Patterns</h2> <h3>Pattern: Know Your Audience</h3> <p>Preparing a great presentation is lost if you present it to the “wrong” audience. Seek data on your attendees; their occupations, relative ages, comfort level with the material, and general background. Tune your material to match as many of the attending demographics as possible.</p> <h3>Pattern: Social Media Advertising</h3> <p>Our current digital age has shifted the focus of marketing presentations from printed flyers and bulletin boards to Twitter, Facebook, blog posts, and e-mails. Presenters should aggressively utilize free and low-cost avenues to advertise their talk in an effort to increase attendance, ticket sales, or just general awareness, all of which can manifest as greater credibility for the presenter.</p> <h3>Pattern: Required</h3> <p>A command performance presents unique challenges and added stress. Unless the topic is near and dear to your heart, you may have to manufacture motivation and interest.</p> <h3>Pattern: The Big Why</h3> <p>You should be clear with yourself and to your audience as to why you’re giving this talk. It is a critical foundation for the vector you are using to address this talk’s topic. Clarifying your motivation does not rule out giving a talk for money, as advertising, to educate, or just to inspire.</p> <h3>Pattern: Proposed</h3> <p>Most presenters are required to submit a talk title, abstract, topic outline, and perhaps even samples of past work in video or PDF form. The most common ways to get into a conference—paying attention to the details and presenting a polished proposal—increase your odds your talk will be accepted.</p> <h3>Antipattern: Abstract Attorney</h3> <p>An abstract attorney is someone who takes the description of the presentation from the formal conference brochure way too seriously. They’ll often comment (negatively) on any and all deviations from the abstract either during the talk (in the form of an often condescending question) or afterward on session evaluations or on social media.</p> <h2>Chapter 2: Creativity Patterns</h2> <h3>Pattern: Narrative Arc</h3> <p>Presentations are a form of storytelling; don’t ignore a few thousand years of oratory history. A Narrative Arc is a common trope; organizing your presentation in a similar way leverages your audience’s lifetime of story listening experience.</p> <h3>Pattern: Fourthought</h3> <p>Don’t rush to use a presentation tool when building a presentation. The four stages of creating a presentation are ideation, capture, organize, and design.</p> <h3>Pattern: Crucible</h3> <p>The act of constructing the presentation differs from presenting the presentation. The presentation will change (sometimes drastically) under the pressure of presenting it.</p> <h3>Pattern: Concurrent Creation</h3> <p>Presentation materials don’t need to be created in the same order as they are found in the presentation. Craft the material in the order that fits; no one will know you did things out of sequence.</p> <h3>Pattern: Triad</h3> <p>Your audience can only absorb a certain amount of material in a short time. If you limit your presentation to three main talking points, it allows you to cover them thoroughly without overwhelming your audience.</p> <h3>Pattern: Expansion Joints</h3> <p>Building a presentation for one (and only one) length is a missed opportunity. Crafting short-, medium-, and full-length versions of your talk provides additional opportunities for delivery and simplifies adapting to shorter windows if the promised time slot is cut short.</p> <h3>Pattern: Talklet</h3> <p>Instead of doing an hour-long presentation, do three semirelated 20-minute talks. This allows great flexibility for time, content, and narrative flow.</p> <h3>Pattern: Unifying Visual Theme</h3> <p>Use a common, repeating visual element to tie together the disparate parts of your presentation.</p> <h3>Pattern: Brain Breaks</h3> <p>Research suggests that the average adult attention span is about 20 minutes. Plan something that breaks the concentration coma: humor, a story, something titillating, audience participation, and so on roughly every 10 to 20 minutes. Reengage your audience or lose them to their smartphones.</p> <h3>Antipattern: Alienating Artifact</h3> <p>An alienating artifact is something that disenfranchises an audience member or audience members from the topic at hand. This can take the form of an image, a quote, or offensive language.</p> <h3>Antipattern: Celery</h3> <p>Just like the vegetable, a celery talk is one that expends more effort chewing (processing) on the part of the audience member than they get back in calories or value and new materials learned.</p> <h3>Pattern: Leet Grammars</h3> <p>Leet Grammars refers to correctly using appropriate slang, jargon, and other “insider” colloquialisms in your presentation to bond with your audience.</p> <h3>Pattern: Lightning Talk</h3> <p>Lightning Talk is a timed presentation, usually five minutes long and optionally constrained by aspecific number of slides. In many formats, the slides advance automatically.</p> <h3>Pattern: Takahashi</h3> <p>Takahashi is a stylized talk format originated by Masoyoshi Takahashi (and popularized in the West by Laurence Lessig) that uses one or two words per slide but transitions through them very quickly.</p> <h3>Pattern: Cave Painting</h3> <p>Use a huge sectioned canvas with your presentation laid out linearly, zooming in on the constituent sections as you proceed through your presentation.</p> <h1>Part 2: Build</h1> <h2>Chapter 3: Slide Construction Patterns</h2> <h3>Antipattern: Cookie Cutter</h3> <p>Ideas don’t have a predetermined word count and accordingly you shouldn’t artificially pad content to make it appear to fill a slide. No law says that every thought worth having will fit on a single slide, so stop trying.</p> <h3>Pattern: Coda</h3> <p>The coda is the concluding piece of a slide deck. It provides a partitioned place to put topic-relevant material that isn’t delivered in the spoken portion of the presentation.</p> <h3>Antipattern: Injured Outlines</h3> <p>Outlines should never have a single point at an indent level; don’t create bullet points that have only one sub-bullet. Bullets in presentations mimic outlines and it “looks” grammatically incorrect to have orphan bullets.</p> <h3>Pattern: Peer Review</h3> <p>Your first draft of a presentation won’t be your best, nor should it be your last. Find a well-informed colleague who knows the topic and have them critique your first draft.</p> <h3>Pattern: Foreshadowing</h3> <p>Like foreshadowing in literature, this pattern adds elements early in a talk to seed an idea that will be resolved (hopefully with more resonance) later.</p> <h3>Antipattern: Bullet-riddled corpse</h3> <p>A Bullet-Riddled Corpse is a presentation where every slide is a long series of dull bullet points. Typically, these slides will then be read to the audience insinuating they can’t read.</p> <h3>Pattern: Greek Chorus</h3> <p>Ancient Greek plays often featured a chorus, a group of players who stood aside from the action, making occasional comments and interludes during the play. Adding a Greek Chorus means you seed the audience with some partisans to interject comments, enthusiasm, or to help defend your case if you are outnumbered.</p> <h3>Antipattern: Ant Fonts</h3> <p>Don’t use tiny fonts in a desire to cram more information on a slide. Slide size is completely arbitrary and has no relationship to the proper size of the content. If you ever think (or worse, say) “You probably can’t read this in the back,” you’ve failed your audience. Slides with unreadable fonts often result from “helpful” software that modifies font size as you add more content. You don’t get bonus points for using fewer slides.</p> <h3>Antipattern: Fontaholic</h3> <p>A Fontaholic is someone who believes using a cacophony of fonts will “jazz up” a presentation. Disharmonic fonts are jarring. They make presentations gruesome and more difficult to read.</p> <h3>Antipattern: Floodmarks</h3> <p>Floodmarks are marketing and branding headers, footers, and watermarks that invade the content area of the slide. Your audience won’t forget who you work for or the name of the conference during the course of your talk. Egregious instances of this antipattern lead to several other antipatterns by occupying valuable real estate.</p> <h3>Antipattern: Photomaniac</h3> <p>A Photomaniac picks up random pieces of clipart, stock photography, and other tchotchkes, littering them about a presentation, filling any and all empty spaces.</p> <h3>Pattern: Composite Animation</h3> <p>A composite animation is when more than one simple animation provided by the presentation tool is applied to an object for a more impressive and educationally helpful combinatory effect.</p> <h3>Pattern: A la carte Content</h3> <p>Creating a presentation “on the fly” using the interest of the crowd to guide the presentation. Typically, this approach begins with a set of related slides (often structured as a Talklet series) that can be alternatively arranged. This technique works well if you have more content than the time slot allows or the talk naturally breaks along several subtopic lines.</p> <h3>Pattern: Vacation Photos</h3> <p>A picture is worth a thousand words, but does that translate into an effective presentation? Carefully selected images can stand in place of wordy slides. With this approach, the presenter becomes a greater focus of attention, supplying the full verbal channel and allowing the well-chosen images to amplify the spoken words.</p> <h3>Pattern: Defy Defaults</h3> <p>Don’t settle for the defaults or stock templates from any tool. If you do use a stock template, at least customize it so that it’s as unique as a signature.</p> <h3>Antipattern: Borrowed Shoes</h3> <p>From time to time, you may be called upon to deliver someone else’s presentation. You are speaking about something that you’re not sincere about, that doesn’t quite fit you, and that smells funny the entire time you’re delivering it. While it can be done in a pinch, the results are rarely awe-inspiring.</p> <h2>Chapter 4: Temporal Patterns</h2> <h3>Antipattern: Slideuments</h3> <p>A Slideument is a presentation that is trying to be both a presentation aide and an attractive printed version of the presentation. An entire chapter is devoted to dealing with the issues raised by this commonly required antipattern.</p> <h3>Pattern: Infodeck</h3> <p>Infodeck is an emerging form of presentation intended to be consumed in solitude rather than to be presented to an audience. It is an instance of the Slideuments antipattern that actually works for its intended purpose.</p> <h3>Pattern: Gradual Consistency</h3> <p>Reveal content bit by bit, only showing the full connection between all the ideas as the conclusion of segments or modules.</p> <h3>Pattern: Charred Trail</h3> <p>The slide shows out-of-date content by graying it out as the presenter progresses through the slide. This prevents the audience from reading ahead and results in a more printable artifact.</p> <h3>Pattern: Exuberant Title Top</h3> <p>A technique that allows you to place a core idea on the center of the slide and then add supporting elements underneath as the main idea migrates to the top of the slide. The Slideuments pattern allows you to use fewer slides yet separate important points.</p> <h3>Pattern: Invisibility</h3> <p>This pattern uses invisible elements that don’t appear on printed versions of the slides but are revealed throughout your talk. This preserves a sense of surprise when you are forced to provide handouts (Slideuments?) in advance of your talk.</p> <h3>Pattern: Context Keeper</h3> <p>An organizational device within a presentation that reveals the structure of the talk, either temporally, by subject matter, or by some other contextually meaningful manner.</p> <h3>Pattern: Breadcrumbs</h3> <p>Create an agenda trail throughout your presentation to provide context on progress. In addition to providing context, this gives the audience a familiar element to ground them.</p> <h3>Pattern: Bookends</h3> <p>Use distinct slides throughout the presentation to supply agenda context. Often visually distinct from the standard slides, they provide a sense of progress as well. Bookend slides indicate the end of one section and the start of a new one.</p> <h3>Pattern: Soft Transitions</h3> <p>When you have stale content, one way to subtly transition away from it uses an almost-imperceptible fade transition for either the slide or elements on the slide. It should happen so gradually that the audience doesn’t even realize it’s happening. Add both slide and element transitions to soften the transition between slides.</p> <h3>Pattern: Intermezzi</h3> <p>Presentations should be made up of logical parts in the form of a Triad or Narrative Arc shape. Sometimes, these sections necessitate a color change, thematic shift, or outline introduction to clearly signal the beginning or ending of an act. This pattern is an implementation of the Context Keeper pattern and is similar to Bookends but is used during the course of the presentation to wrap sections, as opposed to wrapping the entire presentation.</p> <h3>Pattern: Backtracking</h3> <p>Backtracking is the intentional repetition of material for the purpose of reestablishing context. This helps the audience regain their conceptual footing as the narrative moves forward.</p> <h3>Pattern: Preroll</h3> <p>Prior to the start time of your talk, displaying the topic and presenter on screen can be informative and clarifying. If the information cannot comfortably fit on one slide, a series of slides can be recorded as a video and then embedded into the actual presentation and set to loop. When the presenter is ready to give the talk, he or she clicks the remote to advance past the animation to the first content slide.</p> <h3>Pattern: Crawling Credits</h3> <p>As pioneered by the movie Star Wars (with the later addition to the title Episode IV: A New Hope) in 1977, credits can be calmly and progressively shown in a seemingly infinite bottom-to-top slide. This gives each name, company, or URL equal but limited screen time without seeming hurried.</p> <h2>Chapter 5: Demonstrations vs Presentations</h2> <h3>Pattern: Live Demo</h3> <p>Live Demo is running the product live in front of the audience. Thought of as a way to gain credibility and actively promote the product, this approach carries the significant risk of a failing demo. The return on investment is rarely worth the risk.</p> <h3>Antipattern: Dead Demo</h3> <p>This antipattern uses a live demonstration as a time filler when the presenter is short on expositional content.</p> <h3>Pattern: Lipsync</h3> <p>Rather than giving a live demonstration, record the interaction with the tool and play it back as part of the presentation. This approach reduces stress, prevents errors, and allows you to use the heads-up display to see slide metadata. You can also make additional points while the demonstration is playing.</p> <h3>Pattern: Traveling Highlights</h3> <p>Use highlighting (either intrinsic in the tool or a third-party add-on like OmniDazzle) to draw attention to something on the slide such as a picture or screenshot of another tool or application. Traveling Highlights implies that you use transitions to highlight different parts of the slide as you walk through the details.</p> <h3>Pattern: Crawling Code</h3> <p>Code is difficult to display inside a presentation. Simply pasting monochrome text onto a slide is not only ugly, but it often doesn’t fit on one screen. Crawling code is a means of showing just a portion of the code at a time while offering context of the other previous and upcoming lines. The out-of-attention lines are shaded to focus attention on the lines being discussed.</p> <h3>Pattern: Emergence</h3> <p>Emergence suggests that the big thing you’re ultimately going to show (tool, giant diagram, source code, etc.) isn’t a static display. Use motion, transitions, highlights, and other presentation effects to gradually reveal pieces or details.</p> <h3>Pattern: Live on Tape</h3> <p>This term refers to a recorded version of your entire presentation provided electronically. More effective and interactive than the standard “saving your presentation as a PDF.”</p> <h1>Part 3: Deliver</h1> <h2>Chapter 6: Stage Prep</h2> <h3>Pattern: Preparation</h3> <p>Murphy’s Law applies to presentations; nothing trumps being prepared. Plan for as many eventualities of as possible, be it a hardware failure, a room change, or an unruly crowd. It may seem prosaic, but consider a checklist: supported projector resolutions, techniques to manage the audience, display dongles, presenter remotes, notes on your topic, a bottle of water, and other needed items reduce the risk of common mistakes that less prepared presenters might make.</p> <h3>Pattern: Posse</h3> <p>Having team members or friends in the audience not only provides a visual focal point of some likely-to-be-smiling faces but also can act as assistants in times of need. They can also be seeded with useful questions and they can clap and nod at appropriate times; the audience symbiotically follows suit and has a more positive impression of the talk and you as the presenter. Friendly faces reduce stress and put you more at ease.</p> <h3>Pattern: Seeding Satisfaction</h3> <p>Even the friendliest of audiences benefits from some warm-up. Simple greetings, handshakes, and sincere questions, prior to the start of your presentation, break the ice and prime your audience to be more receptive to your message.</p> <h3>Pattern: Display of High Value</h3> <p>As a presenter, you should demonstrate to the audience that there is a difference between the speaker and audience members; the speaker is bringing new knowledge and discoveries and the audience is there to absorb, question, and clarify these new points.</p> <h3>Antipattern: Shortchanged</h3> <p>Dealing with a last-minute reduction in presentation time is unfortunately one of the skills frequent presenters have to hone. Coping well with the situation leaves the audience none the wiser. Managing this situation poorly leaves you rushing, cutting important material, and delivering a less meaningful presentation.</p> <h2>Chapter 7: Performance antipatterns</h2> <h3>Antipattern: Hiccup Words</h3> <p>Hiccup Words are involuntary exclamations (“Ummm,” “Ahhh,” etc.) that distract and detract from the presentation. Dead air isn’t a cardinal sin, and it’s unlikely anyone in your audience will break into a presentation just because you paused.</p> <h3>Antipattern: Disowning Your Topic</h3> <p>If you think a topic is rudimentary, expect it is for your audience as well. Accelerate if you think the audience is already familiar with the topic. Don’t treat your topic as if it was the most interesting thing in the world—it’s not. Assume you’re speaking to a group of smart people who are terribly bored of hearing about your topic; ad-lib and go off the prepared path at will.</p> <h3>Antipattern: Lipstick on a Pig</h3> <p>This antipattern are talks with pointless or weak content dressed up with special effects. Contrary to the wishful practitioner’s hopes and dreams, audience members will see through the decorative facade to the fundamental lack of useful material or insight.</p> <h3>Antipattern: Tower of Babble</h3> <p>Presenters are inclined to use arcane jargon irrespective of the skill level of their audience. Follow that inclination and use three-letter acronyms and technospeak at liberty. Feel confident your audience understands those phrases even if they are giving you puzzled looks.</p> <h3>Antipattern: Bunker</h3> <p>Hiding behind something doesn’t engender trust with your audience. You speak to a group of people; making a more personal connection with them adds nuance and extra meaning to your presentation. Avoid podiums; walk around and engage with your audience. Stationary presenters often give very static presentations.</p> <h3>Antipattern: Hecklers</h3> <p>An audience of any substantial size will have individuals that aim to gain recognition through attacks on the presenter’s premise, approach, or discoveries. Coping with them requires preparation, a cool head, and decisive action. Dealing decisively with hecklers keeps the presentation on track and protects the congenial members of the audience.</p> <h3>Antipattern: Going Meta</h3> <p>Never talk about the talk itself. The audience isn’t interested in mechanics; they came to hear about the interesting topic that you are uniquely equipped to expound upon. This would be akin to a theatrical (not a director’s cut) release of Star Wars Episode IV: A New Hope spending time discussing how George Lucas used a certain kind of lens, the temperature of lighting on the set, and the brand of makeup on the actors.</p> <h3>Antipattern: Backchannel</h3> <p>The backchannel encompasses all the ways people can chat about your presentation while it’s occurring, either via text messages during a business meeting or tweeting during an event.</p> <h3>Antipattern: Laser Weapons</h3> <p>Laser pointers are generally a crutch for identifying content elements on a slide that could otherwise be emphasized by Traveling Highlights or as an attempt to explain a slide that “you probably can’t read in the back.”</p> <h3>Antipattern: Negative Ignorance</h3> <p>Never pose a question to the audience in the form of “Who here is not familiar with X?” People dislike admitting ignorance, especially in a circle of their peers. And turning it to a positive question isn’t much better.</p> <h3>Antipattern: Dual-Headed Monster</h3> <p>This antipattern delivers a presentation to both a live and a remote audience via some desktop sharing or video streaming tool. It reduces the value and richness of the presentation by trying to serve two audiences, each with disparate and competing format and delivery requirements.</p> <h2>Chapter 8: Performance patterns</h2> <h3>Pattern: Carnegie Hall</h3> <p>The only true preparation for unusual occurrences is practice. This pattern defines the types and number of practice sessions you need.</p> <h3>Pattern: Emotional state</h3> <p>Presenters should be attuned to the condition and situation of their audience. Adapt the talk to fit the mood and needs of the audience.</p> <h3>Pattern; Breathing Room</h3> <p>Talking too fast and not leaving enough time for the audience to fully understand the deeper implications of the talk is a common antipattern many many presenters exhibit. While difficult, breathing room is the purposeful insertion of quiet to allow important concepts time to settle and germinate. While dead air might be a cardinal sin in radio, moments of pause allow your audience to integrate your message.</p> <h3>Pattern: Shoeless</h3> <p>Presenter creature comforts, while possibly perceived as odd or eccentric, can put the speaker at ease, enhancing the delivery. Such comforts include the removal of shoes, certain placement of the laptop, comfortable clothes, a favorite beverage, or stepping away from the lectern to mingle among the audience.</p> <h3>Pattern: Mentor</h3> <p>The audience wants to be led but not by the nose. Take on the role of guide and, via the presentation, provide your audience with new skills, ideas, and techniques as if you were their private tutor in your field.</p> <h3>Pattern: Weatherman</h3> <p>TV weathermen use green screens and monitors to see and gesture toward map details. Never turn your back on the audience. If you need to point to something on the screen, do it like a weatherman: Stand off to the side so that you can see both the screen and the audience as you gesture.</p> <h3>Pattern: Seeding the First Question</h3> <p>Purposely placing an obvious question for your audience to ask somewhere in the presentation as a way to break the ice for question-and-answer sessions.</p> <h3>Pattern: Make It Rain</h3> <p>Use props like chairs, paper airplanes, candy (typically thrown at/to audience members), and other devices to add flair to your presentation and encourage interaction, especially with preternaturally shy groups.</p> <h3>Pattern: Entertainment</h3> <p>Entertaining the audience is an important device to capture the attention but should not be used to excess. Like any seasoning, a little bit goes a long way; don’t confuse your message with your method.</p> <h3>Pattern: The Stakeout</h3> <p>Arriving on time for your talk seems too simple to even mention. However, chaos is conspiring to prevent you from doing just that. If the venue isn’t amenable to corporate camping, locate a nearby coffee shop or other work-conducive location. You can arrive at this staging area with ample time to spare but still have the opportunity to use your cushion productively should mayhem take a holiday. It also avoids aimlessly wandering the venue hallways.</p> <h3>Pattern: Lightsaber</h3> <p>When used minimally in the hands of a thoughtful presenter, a laser pointer can be a useful on-demand teaching tool.</p> <h3>Pattern: Echo Chamber</h3> <p>When an attendee asks a question, always repeat the question before answering. Restating the question assures you’ve heard the query and gives you time to formulate a response.</p> <h3>Pattern: Red, Yellow, Green</h3> <p>Several conferences, in order to simplify the voting process and increase participation, put red, yellow, and green cards near the entrance to the room. Upon exit, attendees drop a card into the voting bucket indicating their opinion of the talk. The scores can be tallied immediately.</p> <h3>Analog Noise</h3> <p>Computers make it possible to create visually flawless imagery and text. However, in sci-fi movies, even apparently digital communication has jitter, breakups, blockiness, and noise. The imperfection is a cue that the communications are happening over a long distance. “Analog noise” can be an endearing quality that makes the situation more believable for the audience. Such noise can be intentionally introduced via handwritten fonts, film grain, uneven lines, monochrome colors, and “amateur” photograph errors. Adding some noise to your slides adds visual interest and garners greater attention.</p> https://tedneward.github.io/Research/storage/4store/index.html Mon, 13 Apr 2026 14:04:56 +0000 storage/4store/index.html <p><a href="https://github.com/4store/4store">Source</a></p> <p>Looks like it's abandoned; the website <a href="http://4store.org">http://4store.org</a> no longer appears viable and the last commit to GitHub was 3 years ago.</p> https://tedneward.github.io/Research/speaking/public-speaking-topic-secrets/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/public-speaking-topic-secrets/index.html <p><em>(by Ramakrishna Reddy)</em></p> <h1>Ch1: Blueprint for Speech Topic Ideas</h1> <p>Step 1: LARGER PURPOSE of the speech</p> <ul> <li>Why are you giving the speech?</li> <li>What is the occasion for the speech?</li> <li>What type of speech will suit my larger purpose?</li> </ul> <p>Step 2: Topics that interest YOU</p> <ul> <li>What do you know?</li> <li>What are you enthusiastic about?</li> <li>What are your life experiences?</li> </ul> <p>Step 3: Topics that interest your AUDIENCE</p> <ul> <li>What are your audience's needs?</li> <li>Do they belong to a certain age group?</li> <li>What are their shared experiences?</li> </ul> <p>Toastmasters classification:</p> <ul> <li>Informational</li> <li>Entertaining</li> <li>Persuasive</li> <li>Inspirational</li> </ul> <h1>Ch2: How to find Informational Speech Topic Ideas</h1> <p>Things you like (Books, Gardening, Exercise, Sports, Cooking, Psych, Internet, Expert)</p> <p>Observations</p> <p>Problem/Solutions</p> <h1>Ch3: How to find Entertaining Speech Topic Ideas</h1> <p>Humor is largely based on two principles:</p> <ul> <li>Truth</li> <li>Surprise</li> </ul> <p>Premise and punchline. As a rule of thumb, the premise should be true and should not be funny. The punchline should have the surprise. It can have exaggeration as well.</p> <p>Self-Deprecating (Large relatable unexpected events, Unusual physical traits, Your current life situation, Your identity)</p> <p>Characters in your life (Family, Your professional life, Social setting)</p> <h1>Ch4: How to find Persuasive Speech Topic ideas</h1> <p>Ideally, every persuasive/inspiring speech should have these three elements:</p> <ul> <li>Ethos: relating to credibility--being fair, using proper language, using correct grammar, using appropriate vocabulary</li> <li>Pathos: relating to emotional--emotional tone, stories of emotional events</li> <li>Logos: relating to logic--facts or figures, citing history, logical arguments</li> </ul> <h1>Ch5: How to find Inspiring Speech Topic ideas</h1> <p> <nothing></nothing></p> https://tedneward.github.io/Research/reading/processes/kanban/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/kanban/index.html <p><a href="http://www.infoq.com/minibooks/kanban-scrum-minibook">Kanban and Scrum - making the most of both</a></p> <p><a href="https://leanpub.com/kanbanforskeptics">Kanban for skeptics</a> - Nick Oostvogels <em>(Leanpub account or valid email requested)</em></p> https://tedneward.github.io/Research/reading/processes/ship-it/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/ship-it/index.html <p><em>(by Jared Richardson, William A Gwaltney)</em></p> <ol> <li>Choose your habits</li> <li>Stay in the sandbox</li> <li>If you need it, check it in</li> <li>Script builds on day one</li> <li>Any machine can be a build machine</li> <li>Build continuously</li> <li>Test continuously</li> <li>Avoid collective memory loss</li> <li>Exercise your product--automate your tests</li> <li>Use a common, flexible test harness</li> <li>Use the best tool for the job</li> <li>Use open formats to integrate tools</li> <li>Keep critical path technologies familiar</li> <li>Work to the list</li> <li>Let a tech lead</li> <li>Use daily meetings for frequent course corrections</li> <li>It's OK to say "later"</li> <li>Always review all code</li> <li>The goal is software, not compliance</li> <li>Architect as a group</li> <li>If production uses it, you should too</li> <li>Solve the hardest programs first</li> <li>An encapsulated architecture is a scalable architecture</li> <li>You can't steer a boat unless its moving</li> <li>Don't change legacy code until you can test it</li> <li>Use test-driven refactoring to clean up untestable code</li> <li>Mock client tests do the most with the least</li> <li>Continuously test changing code</li> <li>It has to work for everyone</li> <li>Integrate often, build and test continuously</li> <li>Deliver live demos early &amp; often</li> <li>Publicize what you're doing and why</li> <li>Face time builds teamwork</li> <li>Only fix what needs fixing</li> <li>Disruptive best practices aren't</li> <li>Innovate from the bottom up</li> <li>Show, don't just tell</li> <li>Cultivate management buy-in</li> <li>Test continuously</li> <li>Test where the bugs are</li> <li>The list is a living document</li> <li>If it's not on the list, it's not part of the project</li> <li>Always give feedback fast</li> </ol> https://tedneward.github.io/Research/security/endpoint/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/endpoint/index.html <h3>Anti-Virus / Anti-Malware</h3> <ul> <li><a href="https://github.com/codeyourweb/fastfinder">Fastfinder</a> - Fast customisable cross-platform suspicious file finder. Supports md5/sha1/sha256 hashs, litteral/wildcard strings, regular expressions and YARA rules. Can easily be packed to be deployed on any windows / linux host.</li> <li><a href="https://www.rfxn.com/projects/linux-malware-detect/">Linux Malware Detect</a> - A malware scanner for Linux designed around the threats faced in shared hosted environments.</li> <li><a href="https://github.com/Neo23x0/Loki">LOKI</a> - Simple Indicators of Compromise and Incident Response Scanner</li> <li><a href="http://rkhunter.sourceforge.net/">rkhunter</a> - A Rootkit Hunter for Linux</li> <li><a href="http://www.clamav.net/">ClamAv</a> - ClamAV® is an open-source antivirus engine for detecting trojans, viruses, malware &amp; other malicious threats.</li> </ul> <h3>Content Disarm &amp; Reconstruct</h3> <ul> <li><a href="https://github.com/docbleach/DocBleach">DocBleach</a> - An open-source Content Disarm &amp; Reconstruct software sanitizing Office, PDF and RTF Documents.</li> </ul> <h3>Configuration Management</h3> <ul> <li><a href="https://github.com/fleetdm/fleet">Fleet device management</a> - Fleet is the lightweight, programmable telemetry platform for servers and workstations. Get comprehensive, customizable data from all your devices and operating systems.</li> <li><a href="http://www.rudder-project.org/">Rudder</a> - Rudder is an easy to use, web-driven, role-based solution for IT Infrastructure Automation &amp; Compliance. Automate common system administration tasks (installation, configuration); Enforce configuration over time (configuring once is good, ensuring that configuration is valid and automatically fixing it is better); Inventory of all managed nodes; Web interface to configure and manage nodes and their configuration; Compliance reporting, by configuration and/or by node.</li> </ul> <h3>Authentication</h3> <ul> <li><a href="https://github.com/google/google-authenticator">google-authenticator</a> - The Google Authenticator project includes implementations of one-time passcode generators for several mobile platforms, as well as a pluggable authentication module (PAM). One-time passcodes are generated using open standards developed by the Initiative for Open Authentication (OATH) (which is unrelated to OAuth). These implementations support the HMAC-Based One-time Password (HOTP) algorithm specified in RFC 4226 and the Time-based One-time Password (TOTP) algorithm specified in RFC 6238. <a href="http://xmodulo.com/two-factor-authentication-ssh-login-linux.html">Tutorials: How to set up two-factor authentication for SSH login on Linux</a></li> <li><a href="https://github.com/kurolabs/stegcloak">Stegcloak</a> - Securely assign Digital Authenticity to any written text</li> </ul> <h3>Mobile / Android / iOS</h3> <ul> <li><a href="https://github.com/ashishb/android-security-awesome">android-security-awesome</a> - A collection of android security related resources. A lot of work is happening in academia and industry on tools to perform dynamic analysis, static analysis and reverse engineering of android apps.</li> <li><a href="http://wiki.secmobi.com/">SecMobi Wiki</a> - A collection of mobile security resources which including articles, blogs, books, groups, projects, tools and conferences. *</li> <li><a href="https://github.com/OWASP/owasp-mstg">OWASP Mobile Security Testing Guide</a> - A comprehensive manual for mobile app security testing and reverse engineering.</li> <li><a href="https://github.com/kai5263499/osx-security-awesome">OSX Security Awesome</a> - A collection of OSX and iOS security resources</li> <li><a href="https://github.com/cossacklabs/themis">Themis</a> - High-level multi-platform cryptographic framework for protecting sensitive data: secure messaging with forward secrecy and secure data storage (AES256GCM), suits for building end-to-end encrypted applications.</li> <li><a href="https://mobilesecuritywiki.com/">Mobile Security Wiki</a> - A collection of mobile security resources.</li> <li><a href="https://github.com/iBotPeaches/Apktool">Apktool</a> - A tool for reverse engineering Android apk files.</li> <li><a href="https://github.com/skylot/jadx">jadx</a> - Command line and GUI tools for produce Java source code from Android Dex and Apk files.</li> <li><a href="https://github.com/Storyyeller/enjarify">enjarify</a> - A tool for translating Dalvik bytecode to equivalent Java bytecode.</li> <li><a href="https://github.com/51j0/Android-Storage-Extractor">Android Storage Extractor</a> - A tool to extract local data storage of an Android application in one click.</li> <li><a href="https://github.com/quark-engine/quark-engine">Quark-Engine</a> - An Obfuscation-Neglect Android Malware Scoring System.</li> <li><a href="https://www.jetbrains.com/decompiler/">dotPeek</a> - Free-of-charge standalone tool based on ReSharper's bundled decompiler.</li> <li><a href="https://github.com/GrapheneOS/hardened_malloc">hardened_malloc</a> - Hardened allocator designed for modern systems. It has integration into Android's Bionic libc and can be used externally with musl and glibc as a dynamic library for use on other Linux-based platforms. It will gain more portability / integration over time.</li> <li><a href="https://github.com/ir193/AMExtractor">AMExtractor</a> - AMExtractor can dump out the physical content of your Android device even without kernel source code.</li> <li><a href="https://github.com/frida/frida">frida</a> - Dynamic instrumentation toolkit for developers, reverse-engineers, and security researchers.</li> <li><a href="https://github.com/UDcide/udcide">UDcide</a> - Android Malware Behavior Editor.</li> <li><a href="https://github.com/ptswarm/reFlutter">reFlutter</a> - Flutter Reverse Engineering Framework</li> </ul> <h3>Forensics</h3> <ul> <li><a href="https://github.com/google/grr">grr</a> - GRR Rapid Response is an incident response framework focused on remote live forensics.</li> <li><a href="https://github.com/volatilityfoundation/volatility">Volatility</a> - Python based memory extraction and analysis framework.</li> <li><a href="http://mig.mozilla.org/">mig</a> - MIG is a platform to perform investigative surgery on remote endpoints. It enables investigators to obtain information from large numbers of systems in parallel, thus accelerating investigation of incidents and day-to-day operations security.</li> <li><a href="https://github.com/diogo-fernan/ir-rescue">ir-rescue</a> - <em>ir-rescue</em> is a Windows Batch script and a Unix Bash script to comprehensively collect host forensic data during incident response.</li> <li><a href="https://github.com/dogoncouch/logdissect">Logdissect</a> - CLI utility and Python API for analyzing log files and other data.</li> <li><a href="https://github.com/TonyPhipps/Meerkat">Meerkat</a> - PowerShell-based Windows artifact collection for threat hunting and incident response.</li> <li><a href="https://github.com/google/rekall">Rekall</a> - The Rekall Framework is a completely open collection of tools, implemented in Python under the Apache and GNU General Public License, for the extraction and analysis of digital artifacts computer systems.</li> <li><a href="https://github.com/504ensicsLabs/LiME.git">LiME</a> - Linux Memory Extractor</li> <li><a href="https://github.com/soxoj/maigret">Maigret</a> - Maigret collect a dossier on a person by username only, checking for accounts on a huge number of sites and gathering all the available information from web pages.</li> </ul> https://tedneward.github.io/Research/security/openpolicyagent/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/openpolicyagent/index.html <p><a href="https://www.openpolicyagent.org/">Website</a> | <a href="https://github.com/open-policy-agent">Source</a></p> <ul> <li>uses <a href="/languages/rego">Rego</a> for its policy declarations/definitions</li> </ul> https://tedneward.github.io/Research/speaking/49-habits-for-public-speaking-success/index.html Mon, 13 Apr 2026 14:04:56 +0000 speaking/49-habits-for-public-speaking-success/index.html <p><em>(by Mark Davis)</em></p> <ul> <li><strong>Can we create a great talk?</strong> Be willing to accept feedback; have a positive attitude; mental and physical preparation before the talk</li> <li><strong>Open with a great first line</strong> Audience does not want to be manipulated into sharing their feelings. Don't ask, "How are we tonight?" Why would we ask a question that leads to anything other than a positive outcome? Use a guaranteed opening, like telling a story, asking a question, or doing an activity. Get the audience leaning forward with compelling statements. We want the audience focused on wanting to hear more.</li> <li><strong>Get your speed right</strong> Practice speaking slowly.</li> <li><strong>Love the microphone</strong> Practice holding hairbrush/shampoo bottle. Hold it close to your mouth, just a few inches away. (Rest on chin.)</li> <li><strong>Manage your water intake</strong></li> <li><strong>Stay on time!</strong></li> <li><strong>Mind your gaps and pauses Ums/ahs/well/so crutch words.</strong></li> <li><strong>You can do better than "Hello"</strong> Plan a strong opening</li> <li><strong>Do not drink alcohol before or during a talk!</strong></li> <li><strong>Avoid dehydration</strong> Particularly for hot climates/situations</li> <li><strong>Get the volume right</strong></li> <li><strong>Put variety into the speech</strong> No monotones</li> <li><strong>Trust the technicians, but test beforehand</strong> Prepare for the worst; what will you do when you are outside, and no slides or computers are available?</li> <li><strong>Dress for success</strong></li> <li><strong>Keep the food in the green room</strong></li> <li><strong>Relax and do your best</strong> Nervous speakers think the audience wants to know they are nervous; audience is now focusing on the speaker's nerves and issues, which removes the focus from the speech. Only say what we want the audience to focus on.</li> <li><strong>Never apologize</strong> Some speakers apologize because they want to downplay the audience's expectations. Do not apologize. No one wants to hear it. A confident start gives the audience permission to enjoy our talk.</li> <li><strong>The weather doesn't matter</strong> Do not talk about the weather unless it is relevant to the talk.</li> <li><strong>Location, location, location</strong></li> <li><strong>Being funny is an art form</strong></li> <li><strong>Be well rested</strong></li> <li><strong>Control the body</strong> Try not to be tied to a podium; if we can walk, we will not get as tired.</li> <li><strong>Pause. Breathe. Continue.</strong> Best presentation has moments of silence. This is for the audience to reflect and consider what was said. If we talk too fast or do not pause, we can provide information overload, and then no one remembers anything.</li> <li><strong>Variety, not monotone</strong> Monotones can be cured by increasing the volume. Watch TV news reporters and listen for the melody in their voices. Watch cartoons and listen to the changes in tone, pitch and volume of each character.</li> <li><strong>Don't hurry up, just wrap it up</strong> Prepare a plan for when the talk goes off its original path. Know what can be cut from the presentation so you can finish on time.</li> <li><strong>Close well</strong> Practice closing/ending. Tell the audience what you would like them to do at the end (call to action). Show them what additional resources are available.</li> <li><strong>Panel discussions (if you have to)</strong> Have clear and specific topics for the panel discussion. Everyone must know the guidelines and stick to them. Have a moderator who will keep things on track. Engage the audience with questions they can respond to. Know the rules for interacting with the other panelists and respect when the other panelists are talking. Be prepared to answer questions from the audience and give concise answers. Every speaker should have equal time talking.</li> <li><strong>Don't let the small stuff stop you starting well</strong> Tell a joke. Open with a story. Ask a question. Do an activity. Ask for a volunteer. Share something personal. Tell a secret. (Get the opening right.)</li> <li><strong>Your attitude matters</strong> Keep it positive with all management, organizing staff, and anyone talking to the audience.</li> <li><strong>Use the microphone</strong></li> <li><strong>Take the feedback</strong></li> <li><strong>One introduction is enough</strong> Getting started is better than sharing 20 years of your past experience. In a speech/TedTalk, the value you bring is your content, not your name. You are just the delivery vehicle for your message.</li> <li><strong>Be clear</strong></li> <li><strong>Focus on the audience</strong> Focus on delivering good material right away. Realize when we are wasting people's time, and what is valuable. Audience wants solutions right away.</li> <li><strong>Spare us the name-dropping</strong> People trust in the reality of a personal story; they do not believe us when we have to refer to other people all the time. When we have the confidence to share original material, there is a better connection.</li> <li><strong>Talk with the audience</strong> Every talk needs to be made in such a way that it is a conversation. The most effective presentation is focused on the audience, and a speech that engages the audience's values and interests will be applauded.</li> <li><strong>Make your facts real</strong> Never use a fact as a fact without research.</li> <li><strong>Speak well of others</strong></li> <li><strong>Beware your dominant thoughts</strong> Never mention a topic you do not want to talk about, and try not to think about anything that could distract. Stay on topic.</li> <li><strong>Don't point</strong></li> <li><strong>Talk towards the audience</strong> Try to always keep your back to the side or back of the room. Aim to make eye contact with someone every five to seven seconds. If you have a big screen behind you, do not look at it. Use the laptop or monitors instead.</li> <li><strong>They are already listening--talk already</strong> Observe what the audience looks like in the beginning when they are all paying attention. If the body language and facial expressions stay the same, you are doing OK; if not, you have to adjust.</li> <li><strong>Make PowerPoint work for you</strong></li> <li><strong>Use questions well</strong> Have prepared questions that you can ask to allow you to repeat and recap key points. This also helps add material to support the initial presentation.</li> <li><strong>Speak to the group</strong> Focus on the happy people--focus on where the smiles are. Everyone in the room is still listening to us, even if they do not look like it. Those who aren't looking directly at us can still see with their peripheral vision. We cannot read the minds of the audience, even if we can guess at their body language.</li> <li><strong>Use the equipment like a pro</strong> Be a professional and just get started; trust that it all works. (Particularly if you did a tech check.)</li> <li><strong>Be yourself</strong> Don't claim ownership of unoriginal material.</li> <li><strong>Suggest, don't force</strong> Everything a speaker presents is still just an idea/opinion/theory. People can always choose what to do with the information. Remember that no matter how knowledgeable we are, we do not know everything. Humility needs to be at the essence of why we teach, share, and educate our audiences. If we do not have the mindset to accept feedback, we will never improve.</li> <li><strong>Be humble</strong> Be honest about our background, our area of study, our success. Being open about our qualifications and experience will allow us to be trusted by our audience. We can build a strong rapport by being specific in our knowledge and not trying to be the expert in all areas.</li> <li><strong>Always be patient with your crowd</strong></li> <li><strong>Speak well to everyone</strong> Audiences want to be there; want to learn; deserve respect; deserve a professional presentation; is not our friend until we communicate well and make them one; is not an enemy until we disrespect them and make them one.</li> <li><strong>Keep your language G-rated</strong></li> <li><strong>Focus on the positive</strong> Everything we say directs the audience's focus and attention. Do not say anything you do not want the audience to focus on.</li> <li><strong>Are you really that funny?</strong> It is not the audience's fault if we are not funny.</li> <li><strong>Phones off</strong></li> <li><strong>Plant positive seeds</strong></li> </ul> https://tedneward.github.io/Research/reading/processes/old-agile-vs-new-agile/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/old-agile-vs-new-agile/index.html <p><a href="https://dzone.com/articles/old-versus-new-agile">DZone source</a> citing Cliff Berg as original source, but link appears broken?</p> <p>Agile has changed. In the <a href="https://produxlabs.com/product-thinking-blog/episode-15-jeff-patton">words of Jeff Patton</a>,</p> <blockquote> <p>“When people say Agile today, they mean something different today than they did in 2001... it's come to mean something else.”</p> </blockquote> <p>Unfortunately, some members of the Agile community are stuck in the past. They still believe early Agile myths and dogma such as:</p> <ul> <li>Every team can and should self organize.</li> <li>Teams can and should be fully autonomous.</li> <li>Before Agile there was only waterfall.</li> <li>Programming is a mostly social activity, and is analogous to sports.</li> <li>Agile coaches do not need to take an interest in the technical aspects of how the work is done, such as DevOps issues.</li> <li>Facilitators do not need to understand the subject being discussed.</li> <li>Good leadership usually emerges.</li> <li>The team usually knows best.</li> <li>The human side of Agile is where the focus should be: the team will take care of the technical side in the course of their work if you merely empower them.</li> <li>A team needs a methodology or framework.</li> <li>The problems that need solving are mostly at the team level.</li> <li>Scrum and Kanban are the only choices.</li> <li>Popular Agile practices such as face-to-face communication, a team room, standups, TDD, and pair programming are best for everyone and all teams.</li> </ul> <p>Granted, not everyone bought into the above extremes, but a great many did, and these became assumed standard views across the Agile community during its early years. Yet if one looks at a range of compelling books and blogs that have come out in the past ten years from new authors, they paint a very different picture, and it is remarkably consistent. These arguably define a “new Agile”. Who are these new authors and what do they say? What is the “new Agile”?</p> <p>One of my favorites is Klaus Leopold, and his book <em>Rethinking Agile: Why Agile Teams Have Nothing To Do With Business Agility</em>. It is a non-technical book, so you won’t find DevOps in it, but despite that he brilliantly paints a vivid picture of why old-school Agile’s obsessive focus on teams causes us to lose sight of the product. He connects the dots on just how much has to happen to get a product to market and why an Agile team is only a tiny slice of that, and why all those involved need to understand the big picture and not just their little slice.</p> <p>Another favorite is Mathew Skelton and Manuel Pais’s book <a href="../management/team-topologies"><em>Team Topologies</em></a>. In it they expertly explain how teams exist in an ecosystem, and that the structures around those teams and how they interact is what is most important of all. Unlike old-school Agile’s tunnel vision about “the team”, Skelton and Pais explain why the entire product creation ecosystem is the wicked problem that must be solved.</p> <p>Some old-school Agilists might react with, “But Agile has always pointed to the need for the organization to be Agile.” Well, I was around in the early days, and that is not true. The claim that the organization needs to be Agile really became loud during the late 2000s, after it became clear that inserting Scrum into an existing organization (which Scrum proponents happily supported) did not work, and we saw the rise of the mantra “Don’t do Agile, be Agile.” But by and large, Agilists had no answer for what it looked like for an organization to “be Agile”. Eventually the message became that an organization needs to have an “Agile mindset”, but again that is ill-defined. It is a catch-all for the unknown. In fact, this ambiguity is what led me and others in 2014 to launch Transition2Agile.com, for which we assembled a team that included people who had actually studied organizational behavior.</p> <p>The books and blogs of new Agile are too numerous to list, but I need to mention some more so that you can see the pattern. Of particular note is the acclaimed book <em>Accelerate</em> by Dr. Nicole Forsgren (now VP of Research and Strategy at GitHub), Jez Humble (arguably the “father” of continuous delivery), and Gene Kim (perhaps the “father” of DevOps). In that book they have an entire chapter about leadership, which states (in the Kindle Edition, pp. 238–239),</p> <blockquote> <p>“Why have technology practitioners continuously sought to improve the approach to software development and deployment as well as the stability and security of infrastructure and platforms, yet, in large part, have overlooked (or are unclear about) the way to lead, manage, and sustain these endeavors?. . .we must improve the way we lead and manage IT.”</p> </blockquote> <p>They are pointing to the need for explicit leadership: self organization is not sufficient. In their book, they also provide statistically validated evidence that technical practices pertaining to DevOps strongly and causally correlate with business performance.</p> <p>In the community of thinkers who write about business agility, the topic of leadership is central. The obsessive emphasis on self organization and autonomy are not present: the focus is on having the right kinds of leadership who can create the right climate and support so that people can be empowered. (“Empowered” is not the same as “autonomous.”) Academic writings on organization culture and leadership theory also emphasize the need for the right kinds of leadership.</p> <p>(A note about team autonomy: Among people who really know how this works, “autonomous” is actually shorthand for “mostly autonomous”. For a team to be mostly autonomous, they need to have the right training, skills, experience, support teams, and self-service infrastructure; and even then, they are seldom 100% autonomous.)</p> <p>There are countless other authors who have written powerful books about how humans work best together, including David Marquet, Dr. Daniel Kahneman, Eric Ries, MIrco Hering, Dr. Marta Wilson, Daniel Pink, Patrick Lencioni, Titus Winters (“Software Engineering at Google”), Mik Kersten, Greg McKeown, Jeff Dalton, and many others. Importantly, there are also authors who dismantle the extrovert-favoring and non-neurodiverse bias of the Agile community, including Susan Cain, Cal Newport, and Daniel Goleman. There are also bloggers such as Maarten Dalmijn and countless others who have pointed to myths and dysfunctions with some original Agile ideas and traditions.</p> <h4>The “New Agile” Ideas</h4> <p>And then there is <a href="https://agile2.net/">Agile 2</a>. Agile 2 is new in that it aggregates the ideas of these new thinkers, and integrates these ideas into a cohesive system of thought, while adding missing pieces. Agile 2 interprets these many writings and translates them into a common and holistically integrated shared narrative.</p> <p>But what is that narrative? Agile 2 is complex because humans are complex. It is not a set of bumper sticker maxims asserted without supporting explanation and rationale. Agile 2 is nuanced and broad, and is published with the thought that went into it. But I will summarize it, to give you a sense.</p> <p>Agile 2 is defined by its <a href="https://agile2.net/agile-2/the-values-and-principles-of-agile-2/">Values and Principles</a>. Most of those principles could be summarized as described <a href="https://agile2.net/more-resources/agile-2-in-a-nutshell/">here</a>. Basically, Agile 2 says that extremes don’t usually work well, and that judgment is called for when applying any practice. It also emphasizes the critical importance of having the right kinds of leadership for each situation. Note that “kinds of leadership” is plural. Agile 2 favors emergent leadership and autonomy, but it views those as aspirations rather than assumptions, and includes the theory that senior leaders need to be intentional about the kinds of leadership needed within their organization, as well as take an experimental yet thoughtful approach to the problem of leadership.</p> <p>Another main thesis of Agile 2 is that it embraces neurodiversity, and that effective collaboration about complex topics is not as simple as having a face-to-face conversation. It also recognizes that people not only need to collaborate, but they also need to focus and be able to attain deep thought—something that many people cannot do in a group setting.</p> <p>Another key element of Agile 2 is focus on the product and the ecosystem, instead of on the team. The hard issues tend to be how collections of teams work together and within their ecosystem. Individuals are called out as well, as being important to mentor, develop and recognize just as much as teams need those things, pushing back on old-school Agile’s obsession with “the team”. Again, it is about balance rather than old-school Agile’s extreme focus on the team and minimization of the individual.</p> <h4>Conclusion</h4> <p>Agile needs to evolve — and it has. We all need to let go of early partially formed ideas and elevate our thinking. Much of the thinking is not new in the history of human discourse: many if not most Agile ideas, both new and old, have been expressed many times before; but what changes is the balance and the collections of ideas that link together into a narrative. We need to advance the narrative.</p> <p>More recent Agile ideas are a more evolved expression of Agile. They are more “grown up” in a sense, because they reflect reality instead of a simplistic ideal. Recent ideas are more inclusive, diverse, and rich. Gone is the insistence that everyone works best a certain way: new Agile embraces neurodiversity and helping people to work together and collaborate even when they do not all work best and communicate best in the same way.</p> <p>Finally, this grown-up new Agile is not just about teams: it is about ecosystems and teams, and it addresses head-on the need for leadership in many forms and in many directions, both outward-facing and inward-facing. New Agile builds on and resonates better with the thinking in other communities of thought, including business agility, DevOps, leadership, PeopleOps, and organization design.</p> <p>It is long overdue. If you largely agree with what the aforementioned “new Agile” authors have been writing, then old-school Agile is dead. Long live new Agile, and long live Agile 2!</p> https://tedneward.github.io/Research/reading/the-blank-sheet/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/the-blank-sheet/index.html <p><em>(from <a href="https://fs.blog/blank-sheet/">https://fs.blog/blank-sheet/</a>)</em></p> <p><em>The blank sheet method primes your brain for what you’re about to read and shows you what you’re learning.</em></p> <h2>The Method</h2> <p>Before you start reading a new book, take out a blank sheet of paper. Write down what you know about the book/subject you’re about to read — a mind map.</p> <p>After you finish a reading session, spend a few minutes adding to the map with a different color pen.</p> <p>Before you start your next reading session, review the page.</p> <p>When you’re done reading, put these ‘blank sheets’ into a binder that you periodically review.</p> <h2>Why does it work?</h2> <p>The blank sheet method primes your brain for what you’re about to read, offers structure, and reinforces that you’re learning.</p> <p>When you first start with a blank sheet, you’re forced to search your memory and put on paper what you know (or what you think you know) about a subject. As you read, you literally see the growth.</p> <p>Not only will you add new knowledge, but equally valuable, you’ll remove things you thought you knew that turned out not to be so.</p> <p>Reviewing what you know about a subject, as well as what you have already learned before a reading session, not only improves memory and recall but helps layer and connect ideas.</p> <p>Most of the early connections come from putting the authors’ raw material onto your foundation. If you don’t know anything about the subject before you start, don’t worry. You can borrow the scaffolding in the book to get you started.</p> <p>As your fluency in a subject grows, you’ll start connecting ideas across disciplines, disagreeing with authors about specific points, and even developing your own ideas.</p> <p>This final step is essential for establishing deep fluency and connecting ideas across disciplines. When you finish the book, put the page into a binder. Review the binder every few months. Spaced repetition offers exponential returns for very little additional effort.</p> https://tedneward.github.io/Research/security/fusionauth/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/fusionauth/index.html <h2>Docker setup</h2> <pre><code>``` version: '3' services: db: image: postgres:12.9 environment: PGDATA: /var/lib/postgresql/data/pgdata POSTGRES_USER: ${POSTGRES_USER} POSTGRES_PASSWORD: ${POSTGRES_PASSWORD} # Un-comment to access the db service directly # ports: # - 5432:5432 networks: - db restart: unless-stopped volumes: - db_data:/var/lib/postgresql/data fusionauth: image: fusionauth/fusionauth-app:latest depends_on: - db environment: DATABASE_URL: jdbc:postgresql://db:5432/fusionauth # Prior to version 1.19.0, use this deprecated name # DATABASE_ROOT_USER: ${POSTGRES_USER} DATABASE_ROOT_USERNAME: ${POSTGRES_USER} DATABASE_ROOT_PASSWORD: ${POSTGRES_PASSWORD} # Prior to version 1.19.0, use this deprecated name # DATABASE_USER: ${DATABASE_USER} DATABASE_USERNAME: ${DATABASE_USERNAME} DATABASE_PASSWORD: ${DATABASE_PASSWORD} # Prior to version 1.19.0, use this deprecated names # FUSIONAUTH_MEMORY: ${FUSIONAUTH_MEMORY} # FUSIONAUTH_SEARCH_ENGINE_TYPE: database # FUSIONAUTH_URL: http://fusionauth:9011 # FUSIONAUTH_RUNTIME_MODE: development FUSIONAUTH_APP_MEMORY: ${FUSIONAUTH_APP_MEMORY} FUSIONAUTH_APP_RUNTIME_MODE: development FUSIONAUTH_APP_URL: http://fusionauth:9011 SEARCH_TYPE: database networks: - db restart: unless-stopped ports: - 9011:9011 volumes: - fa_config:/usr/local/fusionauth/config networks: db: driver: bridge volumes: db_data: fa_config: ``` coupled with a .env file: ``` POSTGRES_USER=postgres POSTGRES_PASSWORD=postgres # Prior to version 1.19.0, using DATABASE_USER DATABASE_USER=fusionauth # &gt;= 1.19.0, using DATABASE_USERNAME DATABASE_USERNAME=fusionauth DATABASE_PASSWORD=hkaLBM3RVnyYeYeqE3WI1w2e4Avpy0Wd5O3s3 ES_JAVA_OPTS="-Xms512m -Xmx512m" # Prior to version 1.19.0, using FUSIONAUTH_MEMORY FUSIONAUTH_MEMORY=512M # &gt;= 1.19.0, using FUSIONAUTH_APP_MEMORY FUSIONAUTH_APP_MEMORY=512M ``` </code></pre> https://tedneward.github.io/Research/security/patterns/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/patterns/index.html <ul> <li><a href="https://securitypatterns.distrinet-research.be/">https://securitypatterns.distrinet-research.be/</a></li> <li><a href="https://hillside.net/plop/2022/papers/G1_P1.pdf">https://hillside.net/plop/2022/papers/G1_P1.pdf</a></li> <li><a href="https://dl.acm.org/doi/10.5555/3631672.3631674">https://dl.acm.org/doi/10.5555/3631672.3631674</a></li> <li><a href="http://coresecuritypatterns.websecuritypatterns.com/patterns.htm">http://coresecuritypatterns.websecuritypatterns.com/patterns.htm</a></li> <li><a href="https://opensecurityarchitecture.org/">https://opensecurityarchitecture.org/</a></li> <li><a href="https://archiv.infsec.ethz.ch/intranet_secured/education/as09/seceng/course_material_secured/patterns.pdf/patterns.pdf">https://archiv.infsec.ethz.ch/intranet_secured/education/as09/seceng/course_material_secured/patterns.pdf/patterns.pdf</a></li> </ul> https://tedneward.github.io/Research/reading/processes/extreme-programming-explained/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/extreme-programming-explained/index.html <h1>Principles:</h1> <ul> <li> <p>Rapid Feedback</p> </li> <li> <p>Assume Simplicity</p> </li> <li> <p>Incremental Change</p> </li> <li> <p>Embracing Change</p> </li> <li> <p>Quality Work</p> </li> <li> <p>Teach Learning</p> </li> <li> <p>Small Initial Investment</p> </li> <li> <p>Play to Win</p> </li> <li> <p>Concrete Experiments</p> </li> <li> <p>Open, Honest Communication</p> </li> <li> <p>Work with people's instincts, not against them</p> </li> <li> <p>Accepted Responsibility</p> </li> <li> <p>Local Adaptation</p> </li> <li> <p>Travel Light</p> </li> <li> <p>Honest Measurement</p> </li> </ul> <h1>Practices:</h1> <ul> <li> <p>The Planning Game</p> </li> <li> <p>Small Releases</p> </li> <li> <p>Metaphor</p> </li> <li> <p>Simple Design; do the simplest thing that could possibly work</p> </li> <li> <p>Testing</p> </li> <li> <p>Refactoring</p> </li> <li> <p>Pair Programming</p> </li> <li> <p>Collective Ownership</p> </li> <li> <p>Continuous Integration</p> </li> <li> <p>40-hour week; you can't work a second week of overtime</p> </li> <li> <p>On-site Customer</p> </li> <li> <p>Coding Standards; once and only once rule (DRY)</p> </li> </ul> https://tedneward.github.io/Research/reading/processes/release-it/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/release-it/index.html <p><em>(by Michael Nygard (Pragmatic Bookshelf, 2007, ISBN-13 978-0-9787392-1-8))</em></p> <h1>Stability Antipatterns</h1> <ul> <li>Integration Points</li> <li>Chain Reactions</li> <li>Cascading Failures</li> <li>Users</li> <li>Blocked Threads</li> <li>Attacks of Self-Denial</li> <li>Scaling Effects</li> <li>Unbalanced Capacities</li> <li>Slow Responses</li> <li>SLA Inversion</li> <li>Unbounded Result Sets</li> </ul> <h1>Stability Patterns</h1> <ul> <li>Use Timeouts</li> <li>Circuit Breaker</li> <li>Bulkheads</li> <li>Steady State</li> <li>Fail Fast</li> <li>Handshaking</li> <li>Test Harness</li> <li>Decoupling Middleware</li> </ul> <h1>Capacity Antipatterns</h1> <ul> <li>Resource Pool Contention</li> <li>Excessive JSP Fragments</li> <li>AJAX Overkill</li> <li>Overstaying Sessions</li> <li>Wasted Space in HTML</li> <li>The Reload Button</li> <li>Handcrafted SQL</li> <li>Database Eutrophication</li> <li>Integration Point Latency</li> <li>Cookie Monsters</li> </ul> <h1>Capacity Patterns</h1> <ul> <li>Pool Connections</li> <li>Use Caching Carefully</li> <li>Precompute Content</li> <li>Tune the Garbage Collector</li> </ul> <h1>Networking</h1> <ul> <li>Multihomed Servers</li> <li>Routing</li> <li>Virtual IP Addresses</li> </ul> <h1>Security</h1> <ul> <li>Principle of Least Privilege</li> <li>Configured Passwords</li> </ul> <h1>Availability</h1> <ul> <li>Load Balancing</li> <li>Clustering</li> </ul> <h1>Administration</h1> <ul> <li>Does QA Match Production?</li> <li>Configuration Files</li> <li>Start-up and Shutdown</li> <li>Administrative Interfaces</li> </ul> <h1>Operations</h1> https://tedneward.github.io/Research/security/argus/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/argus/index.html <p><a href="https://github.com/jasonxtn/argus.git">Source</a></p> <table> <thead> <tr> <th> Category </th> <th> Module Count </th> <th> Key Examples </th> </tr> </thead> <tbody> <tr> <td> Network &amp; Infrastructure </td> <td> 52 </td> <td> DNS Records, Open Ports Scan, WHOIS Lookup </td> </tr> <tr> <td> Web Application Analysis </td> <td> 50 </td> <td> Directory Finder, Technology Stack Detection, GraphQL Probe </td> </tr> <tr> <td> Security &amp; Threat Intelligence </td> <td> 33 </td> <td> Censys Recon, Subdomain Enumeration, JS Malware Scanner </td> </tr> </tbody> </table> <h2>Install</h2> <table> <thead> <tr> <th> Option </th> <th> Commands </th> <th> Best For </th> </tr> </thead> <tbody> <tr> <td> No Install </td> <td> git clone <a href="https://github.com/jasonxtn/argus.git">https://github.com/jasonxtn/argus.git</a>; cd argus; pip install -r requirements.txt; python -m argus </td> <td> Quick tests </td> </tr> <tr> <td> Pip </td> <td> pip install argus-recon; argus </td> <td> Minimal setup </td> </tr> <tr> <td> Full </td> <td> git clone …; chmod +x install.sh; ./install.sh; python -m argus </td> <td> Development </td> </tr> <tr> <td> Docker </td> <td> git clone …; docker build -t argus-recon:latest .; docker run … </td> <td> Containers </td> </tr> </tbody> </table> https://tedneward.github.io/Research/security/network/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/network/index.html <h2>Network architecture</h2> <ul> <li><a href="https://github.com/sergiomarotco/Network-segmentation-cheat-sheet">Network-segmentation-cheat-sheet</a> - This project was created to publish the best practices for segmentation of the corporate network of any company. In general, the schemes in this project are suitable for any company.</li> </ul> <h2>Scanning / Pentesting</h2> <ul> <li><a href="http://www.openvas.org/">OpenVAS</a> - OpenVAS is a framework of several services and tools offering a comprehensive and powerful vulnerability scanning and vulnerability management solution.</li> <li><a href="https://github.com/rapid7/metasploit-framework">Metasploit Framework</a> - A tool for developing and executing exploit code against a remote target machine. Other important sub-projects include the Opcode Database, shellcode archive and related research.</li> <li><a href="https://www.kali.org/">Kali</a> - Kali Linux is a Debian-derived Linux distribution designed for digital forensics and penetration testing. Kali Linux is preinstalled with numerous penetration-testing programs, including nmap (a port scanner), Wireshark (a packet analyzer), John the Ripper (a password cracker), and Aircrack-ng (a software suite for penetration-testing wireless LANs).</li> <li><a href="https://tsurugi-linux.org/">tsurugi</a> - heavily customized Linux distribution that designed to support DFIR investigations, malware analysis and OSINT activities. It is based on Ubuntu 20.04(64-bit with a 5.15.12 custom kernel)</li> <li><a href="https://github.com/rafael-santiago/pig">pig</a> - A Linux packet crafting tool.</li> <li><a href="https://github.com/gpotter2/awesome-scapy">scapy</a> - Scapy: the python-based interactive packet manipulation program &amp; library.</li> <li><a href="https://github.com/rfunix/Pompem">Pompem</a> - Pompem is an open source tool, which is designed to automate the search for exploits in major databases. Developed in Python, has a system of advanced search, thus facilitating the work of pentesters and ethical hackers. In its current version, performs searches in databases: Exploit-db, 1337day, Packetstorm Security...</li> <li><a href="https://nmap.org">Nmap</a> - Nmap is a free and open source utility for network discovery and security auditing.</li> <li><a href="https://github.com/caffix/amass">Amass</a> - Amass performs DNS subdomain enumeration by scraping the largest number of disparate data sources, recursive brute forcing, crawling of web archives, permuting and altering names, reverse DNS sweeping and other techniques.</li> <li><a href="https://github.com/rozgo/anevicon">Anevicon</a> - The most powerful UDP-based load generator, written in Rust.</li> <li><a href="https://github.com/isgasho/finshir">Finshir</a> - A coroutines-driven Low &amp; Slow traffic generator, written in Rust.</li> <li><a href="https://github.com/GoVanguard/legion">Legion</a> - Open source semi-automated discovery and reconnaissance network penetration testing framework.</li> <li><a href="https://github.com/aboul3la/Sublist3r">Sublist3r</a> - Fast subdomains enumeration tool for penetration testers</li> <li><a href="https://github.com/RustScan/RustScan">RustScan</a> - Faster Nmap scanning with Rust. Take a 17 minute Nmap scan down to 19 seconds.</li> <li><a href="https://github.com/jtpereyda/boofuzz">Boofuzz</a> - Fuzzing engine and fuzz testing framework.</li> <li><a href="https://github.com/RedTeamPentesting/monsoon">monsoon</a> - Very flexible and fast interactive HTTP enumeration/fuzzing.</li> <li><a href="https://github.com/spectralops/netz">Netz</a>- Discover internet-wide misconfigurations, using zgrab2 and others.</li> <li><a href="https://github.com/deepfence/ThreatMapper">Deepfence ThreatMapper</a> - Apache v2, powerful runtime vulnerability scanner for kubernetes, virtual machines and serverless.</li> <li><a href="https://github.com/deepfence/SecretScanner">Deepfence SecretScanner</a> - Find secrets and passwords in container images and file systems.</li> <li><a href="https://github.com/padok-team/cognito-scanner">Cognito Scanner</a> - CLI tool to pentest Cognito AWS instance. It implements three attacks: unwanted account creation, account oracle and identity pool escalation</li> </ul> <h2>Monitoring / Logging</h2> <ul> <li><a href="https://github.com/retracedhq/retraced">BoxyHQ</a> - Open source API for security and compliance audit logging.</li> <li><a href="http://justniffer.sourceforge.net/">justniffer</a> - Justniffer is a network protocol analyzer that captures network traffic and produces logs in a customized way, can emulate Apache web server log files, track response times and extract all "intercepted" files from the HTTP traffic.</li> <li><a href="http://dumpsterventures.com/jason/httpry/">httpry</a> - httpry is a specialized packet sniffer designed for displaying and logging HTTP traffic. It is not intended to perform analysis itself, but to capture, parse, and log the traffic for later analysis. It can be run in real-time displaying the traffic as it is parsed, or as a daemon process that logs to an output file. It is written to be as lightweight and flexible as possible, so that it can be easily adaptable to different applications.</li> <li><a href="http://ngrep.sourceforge.net/">ngrep</a> - ngrep strives to provide most of GNU grep's common features, applying them to the network layer. ngrep is a pcap-aware tool that will allow you to specify extended regular or hexadecimal expressions to match against data payloads of packets. It currently recognizes IPv4/6, TCP, UDP, ICMPv4/6, IGMP and Raw across Ethernet, PPP, SLIP, FDDI, Token Ring and null interfaces, and understands BPF filter logic in the same fashion as more common packet sniffing tools, such as tcpdump and snoop.</li> <li><a href="https://github.com/gamelinux/passivedns">passivedns</a> - A tool to collect DNS records passively to aid Incident handling, Network Security Monitoring (NSM) and general digital forensics. PassiveDNS sniffs traffic from an interface or reads a pcap-file and outputs the DNS-server answers to a log file. PassiveDNS can cache/aggregate duplicate DNS answers in-memory, limiting the amount of data in the logfile without loosing the essens in the DNS answer.</li> <li><a href="http://sagan.quadrantsec.com/">sagan</a> - Sagan uses a 'Snort like' engine and rules to analyze logs (syslog/event log/snmptrap/netflow/etc).</li> <li><a href="http://www.ntop.org/products/traffic-analysis/ntop/">ntopng</a> - Ntopng is a network traffic probe that shows the network usage, similar to what the popular top Unix command does.</li> <li><a href="https://github.com/rabbitstack/fibratus">Fibratus</a> - Fibratus is a tool for exploration and tracing of the Windows kernel. It is able to capture the most of the Windows kernel activity - process/thread creation and termination, file system I/O, registry, network activity, DLL loading/unloading and much more. Fibratus has a very simple CLI which encapsulates the machinery to start the kernel event stream collector, set kernel event filters or run the lightweight Python modules called filaments.</li> <li><a href="https://github.com/evilsocket/opensnitch">opensnitch</a> - OpenSnitch is a GNU/Linux port of the Little Snitch application firewall</li> <li><a href="https://github.com/wazuh/wazuh">wazuh</a> - Wazuh is a free and open source platform used for threat prevention, detection, and response. It is capable of monitoring file system changes, system calls and inventory changes.</li> <li><a href="https://github.com/matanolabs/matano">Matano</a>: Open source serverless security lake platform on AWS that lets you ingest, store, and analyze petabytes of security data into an Apache Iceberg data lake and run realtime Python detections as code.</li> <li><a href="https://falco.org/">Falco</a> - The cloud-native runtime security project and de facto Kubernetes threat detection engine now part of the CNCF.</li> <li><a href="https://github.com/tenzir/vast">VAST</a> - Open source security data pipeline engine for structured event data, supporting high-volume telemetry ingestion, compaction, and retrieval; purpose-built for security content execution, guided threat hunting, and large-scale investigation.</li> <li><a href="https://github.com/brexhq/substation">Substation</a> - Substation is a cloud native data pipeline and transformation toolkit written in Go.</li> </ul> <h2>IDS / IPS / Host IDS / Host IPS</h2> <ul> <li><a href="https://www.snort.org/">Snort</a> - Snort is a free and open source network intrusion prevention system (NIPS) and network intrusion detection system (NIDS)created by Martin Roesch in 1998. Snort is now developed by Sourcefire, of which Roesch is the founder and CTO. In 2009, Snort entered InfoWorld's Open Source Hall of Fame as one of the "greatest [pieces of] open source software of all time".</li> <li><a href="https://zeek.org/">Zeek</a> - Zeek is a powerful network analysis framework that is much different from the typical IDS you may know.</li> <li><a href="https://github.com/corelight/zeek2es">zeek2es</a> - An open source tool to convert Zeek logs to Elastic/OpenSearch. You can also output pure JSON from Zeek's TSV logs!</li> <li><a href="https://drkeithjones.com">DrKeithJones.com</a> - A blog on cyber security and network security monitoring.</li> <li><a href="https://ossec.github.io/">OSSEC</a> - Comprehensive Open Source HIDS. Not for the faint of heart. Takes a bit to get your head around how it works. Performs log analysis, file integrity checking, policy monitoring, rootkit detection, real-time alerting and active response. It runs on most operating systems, including Linux, MacOS, Solaris, HP-UX, AIX and Windows. Plenty of reasonable documentation. Sweet spot is medium to large deployments.</li> <li><a href="http://suricata-ids.org/">Suricata</a> - Suricata is a high performance Network IDS, IPS and Network Security Monitoring engine. Open Source and owned by a community run non-profit foundation, the Open Information Security Foundation (OISF). Suricata is developed by the OISF and its supporting vendors.</li> <li><a href="http://blog.securityonion.net/">Security Onion</a> - Security Onion is a Linux distro for intrusion detection, network security monitoring, and log management. It's based on Ubuntu and contains Snort, Suricata, Zeek, OSSEC, Sguil, Squert, Snorby, ELSA, Xplico, NetworkMiner, and many other security tools. The easy-to-use Setup wizard allows you to build an army of distributed sensors for your enterprise in minutes!</li> <li><a href="https://github.com/marshyski/sshwatch">sshwatch</a> - IPS for SSH similar to DenyHosts written in Python. It also can gather information about attacker during the attack in a log.</li> <li><a href="https://fbb-git.gitlab.io/stealth/">Stealth</a> - File integrity checker that leaves virtually no sediment. Controller runs from another machine, which makes it hard for an attacker to know that the file system is being checked at defined pseudo random intervals over SSH. Highly recommended for small to medium deployments.</li> <li><a href="https://bitbucket.org/camp0/aiengine">AIEngine</a> - AIEngine is a next generation interactive/programmable Python/Ruby/Java/Lua packet inspection engine with capabilities of learning without any human intervention, NIDS(Network Intrusion Detection System) functionality, DNS domain classification, network collector, network forensics and many others.</li> <li><a href="http://denyhosts.sourceforge.net/">Denyhosts</a> - Thwart SSH dictionary based attacks and brute force attacks.</li> <li><a href="http://www.fail2ban.org/wiki/index.php/Main_Page">Fail2Ban</a> - Scans log files and takes action on IPs that show malicious behavior.</li> <li><a href="http://www.sshguard.net/">SSHGuard</a> - A software to protect services in addition to SSH, written in C</li> <li><a href="https://cisofy.com/lynis/">Lynis</a> - an open source security auditing tool for Linux/Unix.</li> <li><a href="https://github.com/crowdsecurity/crowdsec">CrowdSec</a> - CrowdSec is a free, modern &amp; collaborative behavior detection engine, coupled with a global IP reputation network. It stacks on Fail2Ban's philosophy but is IPV6 compatible and 60x faster (Go vs Python), uses Grok patterns to parse logs and YAML scenario to identify behaviors. CrowdSec is engineered for modern Cloud / Containers / VM based infrastructures (by decoupling detection and remediation). Once detected, you can remedy threats with various bouncers (firewall block, nginx http 403, Captchas, etc.) while the aggressive IPs can be sent to CrowdSec for curation before being shared among all users to further strengthen the community</li> <li><a href="https://github.com/wazuh/wazuh">wazuh</a> - Wazuh is a free and open source XDR platform used for threat prevention, detection, and response. It is capable of protecting workloads across on-premises, virtualized, containerized, and cloud-based environments. Great tool foor all kind of deployments, it includes SIEM capabitilies (indexing + searching + WUI).</li> </ul> <h2>Honey Pot / Honey Net</h2> <ul> <li><a href="https://github.com/paralax/awesome-honeypots">awesome-honeypots</a> - The canonical awesome honeypot list.</li> <li><a href="https://github.com/foospidy/HoneyPy">HoneyPy</a> - HoneyPy is a low to medium interaction honeypot. It is intended to be easy to: deploy, extend functionality with plugins, and apply custom configurations.</li> <li><a href="http://conpot.org/">Conpot</a> - ICS/SCADA Honeypot. Conpot is a low interactive server side Industrial Control Systems honeypot designed to be easy to deploy, modify and extend. By providing a range of common industrial control protocols we created the basics to build your own system, capable to emulate complex infrastructures to convince an adversary that he just found a huge industrial complex. To improve the deceptive capabilities, we also provided the possibility to server a custom human machine interface to increase the honeypots attack surface. The response times of the services can be artificially delayed to mimic the behaviour of a system under constant load. Because we are providing complete stacks of the protocols, Conpot can be accessed with productive HMI's or extended with real hardware. Conpot is developed under the umbrella of the Honeynet Project and on the shoulders of a couple of very big giants.</li> <li><a href="https://github.com/zeroq/amun">Amun</a> - Amun Python-based low-interaction Honeypot.</li> <li><a href="http://glastopf.org/">Glastopf</a> - Glastopf is a Honeypot which emulates thousands of vulnerabilities to gather data from attacks targeting web applications. The principle behind it is very simple: Reply the correct response to the attacker exploiting the web application.</li> <li><a href="https://github.com/desaster/kippo">Kippo</a> - Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.</li> <li><a href="http://kojoney.sourceforge.net/">Kojoney</a> - Kojoney is a low level interaction honeypot that emulates an SSH server. The daemon is written in Python using the Twisted Conch libraries.</li> <li><a href="https://github.com/tnich/honssh">HonSSH</a> - HonSSH is a high-interaction Honey Pot solution. HonSSH will sit between an attacker and a honey pot, creating two separate SSH connections between them.</li> <li><a href="http://sourceforge.net/projects/bifrozt/">Bifrozt</a> - Bifrozt is a NAT device with a DHCP server that is usually deployed with one NIC connected directly to the Internet and one NIC connected to the internal network. What differentiates Bifrozt from other standard NAT devices is its ability to work as a transparent SSHv2 proxy between an attacker and your honeypot. If you deployed an SSH server on Bifrozt’s internal network it would log all the interaction to a TTY file in plain text that could be viewed later and capture a copy of any files that were downloaded. You would not have to install any additional software, compile any kernel modules or use a specific version or type of operating system on the internal SSH server for this to work. It will limit outbound traffic to a set number of ports and will start to drop outbound packets on these ports when certain limits are exceeded.</li> <li><a href="http://bruteforce.gr/honeydrive">HoneyDrive</a> - HoneyDrive is the premier honeypot Linux distro. It is a virtual appliance (OVA) with Xubuntu Desktop 12.04.4 LTS edition installed. It contains over 10 pre-installed and pre-configured honeypot software packages such as Kippo SSH honeypot, Dionaea and Amun malware honeypots, Honeyd low-interaction honeypot, Glastopf web honeypot and Wordpot, Conpot SCADA/ICS honeypot, Thug and PhoneyC honeyclients and more. Additionally it includes many useful pre-configured scripts and utilities to analyze, visualize and process the data it can capture, such as Kippo-Graph, Honeyd-Viz, DionaeaFR, an ELK stack and much more. Lastly, almost 90 well-known malware analysis, forensics and network monitoring related tools are also present in the distribution.</li> <li><a href="http://www.cuckoosandbox.org/">Cuckoo Sandbox</a> - Cuckoo Sandbox is an Open Source software for automating analysis of suspicious files. To do so it makes use of custom components that monitor the behavior of the malicious processes while running in an isolated environment.</li> <li><a href="http://dtag-dev-sec.github.io/mediator/feature/2017/11/07/t-pot-17.10.html">T-Pot Honeypot Distro</a> - T-Pot is based on the network installer of Ubuntu Server 16/17.x LTS. The honeypot daemons as well as other support components being used have been containerized using docker. This allows us to run multiple honeypot daemons on the same network interface while maintaining a small footprint and constrain each honeypot within its own environment. Installation over vanilla Ubuntu - <a href="https://github.com/dtag-dev-sec/t-pot-autoinstall">T-Pot Autoinstall</a> - This script will install T-Pot 16.04/17.10 on a fresh Ubuntu 16.04.x LTS (64bit). It is intended to be used on hosted servers, where an Ubuntu base image is given and there is no ability to install custom ISO images. Successfully tested on vanilla Ubuntu 16.04.3 in VMware.</li> </ul> <h2>Full Packet Capture / Forensic</h2> <ul> <li><a href="https://github.com/simsong/tcpflow">tcpflow</a> - tcpflow is a program that captures data transmitted as part of TCP connections (flows), and stores the data in a way that is convenient for protocol analysis and debugging. Each TCP flow is stored in its own file. Thus, the typical TCP flow will be stored in two files, one for each direction. tcpflow can also process stored 'tcpdump' packet flows.</li> <li><a href="https://github.com/deepfence/PacketStreamer">Deepfence PacketStreamer</a> - High-performance remote packet capture and collection tool, distributed tcpdump for cloud native environments.</li> <li><a href="http://www.xplico.org/">Xplico</a> - The goal of Xplico is extract from an internet traffic capture the applications data contained. For example, from a pcap file Xplico extracts each email (POP, IMAP, and SMTP protocols), all HTTP contents, each VoIP call (SIP), FTP, TFTP, and so on. Xplico isn’t a network protocol analyzer. Xplico is an open source Network Forensic Analysis Tool (NFAT).</li> <li><a href="https://github.com/aol/moloch">Moloch</a> - Moloch is an open source, large scale IPv4 packet capturing (PCAP), indexing and database system. A simple web interface is provided for PCAP browsing, searching, and exporting. APIs are exposed that allow PCAP data and JSON-formatted session data to be downloaded directly. Simple security is implemented by using HTTPS and HTTP digest password support or by using apache in front. Moloch is not meant to replace IDS engines but instead work along side them to store and index all the network traffic in standard PCAP format, providing fast access. Moloch is built to be deployed across many systems and can scale to handle multiple gigabits/sec of traffic.</li> <li><a href="http://www.openfpc.org">OpenFPC</a> - OpenFPC is a set of tools that combine to provide a lightweight full-packet network traffic recorder &amp; buffering system. It's design goal is to allow non-expert users to deploy a distributed network traffic recorder on COTS hardware while integrating into existing alert and log management tools.</li> <li><a href="https://github.com/USArmyResearchLab/Dshell">Dshell</a> - Dshell is a network forensic analysis framework. Enables rapid development of plugins to support the dissection of network packet captures.</li> <li><a href="https://github.com/google/stenographer">stenographer</a> - Stenographer is a packet capture solution which aims to quickly spool all packets to disk, then provide simple, fast access to subsets of those packets.</li> </ul> <h2>Sniffer</h2> <ul> <li><a href="https://www.wireshark.org">wireshark</a> - Wireshark is a free and open-source packet analyzer. It is used for network troubleshooting, analysis, software and communications protocol development, and education. Wireshark is very similar to tcpdump, but has a graphical front-end, plus some integrated sorting and filtering options.</li> <li><a href="http://netsniff-ng.org/">netsniff-ng</a> - netsniff-ng is a free Linux networking toolkit, a Swiss army knife for your daily Linux network plumbing if you will. Its gain of performance is reached by zero-copy mechanisms, so that on packet reception and transmission the kernel does not need to copy packets from kernel space to user space and vice versa.</li> <li><a href="https://addons.mozilla.org/en-US/firefox/addon/http-header-live/">Live HTTP headers</a> - Live HTTP headers is a free firefox addon to see your browser requests in real time. It shows the entire headers of the requests and can be used to find the security loopholes in implementations.</li> </ul> <h2>Security Information &amp; Event Management</h2> <ul> <li><a href="https://www.prelude-siem.org/">Prelude</a> - Prelude is a Universal "Security Information &amp; Event Management" (SIEM) system. Prelude collects, normalizes, sorts, aggregates, correlates and reports all security-related events independently of the product brand or license giving rise to such events; Prelude is "agentless".</li> <li><a href="https://www.alienvault.com/open-threat-exchange/projects">OSSIM</a> - OSSIM provides all of the features that a security professional needs from a SIEM offering – event collection, normalization, and correlation.</li> <li><a href="https://github.com/certsocietegenerale/FIR">FIR</a> - Fast Incident Response, a cybersecurity incident management platform.</li> <li><a href="https://github.com/dogoncouch/LogESP">LogESP</a> - Open Source SIEM (Security Information and Event Management system).</li> <li><a href="https://github.com/wazuh/wazuh">wazuh</a> -Wazuh is a free, open source and enterprise-ready security monitoring solution for threat detection, integrity monitoring, incident response and compliance. It works with tons of data supported by an OpenSearch fork and custom WUI.</li> <li><a href="https://github.com/tenzir/vast">VAST</a> - Open source security data pipeline engine for structured event data, supporting high-volume telemetry ingestion, compaction, and retrieval; purpose-built for security content execution, guided threat hunting, and large-scale investigation.</li> <li><a href="https://github.com/matanolabs/matano">Matano</a> - Open source serverless security lake platform on AWS that lets you ingest, store, and analyze petabytes of security data into an Apache Iceberg data lake and run realtime Python detections as code.</li> </ul> <h2>VPN</h2> <ul> <li><a href="https://openvpn.net/">OpenVPN</a> - OpenVPN is an open source software application that implements virtual private network (VPN) techniques for creating secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It uses a custom security protocol that utilizes SSL/TLS for key exchange.</li> <li><a href="https://github.com/firezone/firezone">Firezone</a> - Open-source VPN server and egress firewall for Linux built on WireGuard that makes it simple to manage secure remote access to your company’s private networks. Firezone is easy to set up (all dependencies are bundled thanks to Chef Omnibus), secure, performant, and self hostable.</li> </ul> <h2>Fast Packet Processing</h2> <ul> <li><a href="http://dpdk.org/">DPDK</a> - DPDK is a set of libraries and drivers for fast packet processing.</li> <li><a href="https://github.com/pfq/PFQ">PFQ</a> - PFQ is a functional networking framework designed for the Linux operating system that allows efficient packets capture/transmission (10G and beyond), in-kernel functional processing and packets steering across sockets/end-points.</li> <li><a href="http://www.ntop.org/products/packet-capture/pf_ring/">PF_RING</a> - PF_RING is a new type of network socket that dramatically improves the packet capture speed.</li> <li><a href="http://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/">PF_RING ZC (Zero Copy)</a> - PF_RING ZC (Zero Copy) is a flexible packet processing framework that allows you to achieve 1/10 Gbit line rate packet processing (both RX and TX) at any packet size. It implements zero copy operations including patterns for inter-process and inter-VM (KVM) communications.</li> <li><a href="https://elixir.bootlin.com/linux/latest/source/Documentation/networking/packet_mmap.rst">PACKET_MMAP/TPACKET/AF_PACKET</a> - It's fine to use PACKET_MMAP to improve the performance of the capture and transmission process in Linux.</li> <li><a href="http://info.iet.unipi.it/~luigi/netmap/">netmap</a> - netmap is a framework for high speed packet I/O. Together with its companion VALE software switch, it is implemented as a single kernel module and available for FreeBSD, Linux and now also Windows.</li> </ul> <h2>Firewall</h2> <ul> <li><a href="https://www.pfsense.org/">pfSense</a> - Firewall and Router FreeBSD distribution.</li> <li><a href="https://opnsense.org/">OPNsense</a> - is an open source, easy-to-use and easy-to-build FreeBSD based firewall and routing platform. OPNsense includes most of the features available in expensive commercial firewalls, and more in many cases. It brings the rich feature set of commercial offerings with the benefits of open and verifiable sources.</li> <li><a href="https://www.cipherdyne.org/fwknop/">fwknop</a> - Protects ports via Single Packet Authorization in your firewall.</li> </ul> <h2>Anti-Spam</h2> <ul> <li><a href="https://github.com/spamscanner">Spam Scanner</a> - Anti-Spam Scanning Service and Anti-Spam API by <a href="https://github.com/niftylettuce">@niftylettuce</a>.</li> <li><a href="https://github.com/rspamd/rspamd">rspamd</a> - Fast, free and open-source spam filtering system.</li> <li><a href="https://spamassassin.apache.org/">SpamAssassin</a> - A powerful and popular email spam filter employing a variety of detection technique.</li> <li><a href="https://scammerlist.now.sh/">Scammer-List</a> - A free open source AI based Scam and Spam Finder with a free API</li> </ul> <h2>Docker Images for Penetration Testing &amp; Security</h2> <ul> <li><code>docker pull kalilinux/kali-linux-docker</code> <a href="https://hub.docker.com/r/kalilinux/kali-linux-docker/">official Kali Linux</a></li> <li><code>docker pull owasp/zap2docker-stable</code> - <a href="https://github.com/zaproxy/zaproxy">official OWASP ZAP</a></li> <li><code>docker pull wpscanteam/wpscan</code> - <a href="https://hub.docker.com/r/wpscanteam/wpscan/">official WPScan</a></li> <li><code>docker pull remnux/metasploit</code> - <a href="https://hub.docker.com/r/remnux/metasploit/">docker-metasploit</a></li> <li><code>docker pull citizenstig/dvwa</code> - <a href="https://hub.docker.com/r/citizenstig/dvwa/">Damn Vulnerable Web Application (DVWA)</a></li> <li><code>docker pull wpscanteam/vulnerablewordpress</code> - <a href="https://hub.docker.com/r/wpscanteam/vulnerablewordpress/">Vulnerable WordPress Installation</a></li> <li><code>docker pull hmlio/vaas-cve-2014-6271</code> - <a href="https://hub.docker.com/r/hmlio/vaas-cve-2014-6271/">Vulnerability as a service: Shellshock</a></li> <li><code>docker pull hmlio/vaas-cve-2014-0160</code> - <a href="https://hub.docker.com/r/hmlio/vaas-cve-2014-0160/">Vulnerability as a service: Heartbleed</a></li> <li><code>docker pull opendns/security-ninjas</code> - <a href="https://hub.docker.com/r/opendns/security-ninjas/">Security Ninjas</a></li> <li><code>docker pull diogomonica/docker-bench-security</code> - <a href="https://hub.docker.com/r/diogomonica/docker-bench-security/">Docker Bench for Security</a></li> <li><code>docker pull ismisepaul/securityshepherd</code> - <a href="https://hub.docker.com/r/ismisepaul/securityshepherd/">OWASP Security Shepherd</a></li> <li><code>docker pull danmx/docker-owasp-webgoat</code> - <a href="https://hub.docker.com/r/danmx/docker-owasp-webgoat/">OWASP WebGoat Project docker image</a></li> <li><code>docker-compose build &amp;&amp; docker-compose up</code> - <a href="https://github.com/owasp/nodegoat#option-3---run-nodegoat-on-docker">OWASP NodeGoat</a></li> <li><code>docker pull citizenstig/nowasp</code> - <a href="https://hub.docker.com/r/citizenstig/nowasp/">OWASP Mutillidae II Web Pen-Test Practice Application</a></li> <li><code>docker pull bkimminich/juice-shop</code> - <a href="https://hub.docker.com/r/bkimminich/juice-shop">OWASP Juice Shop</a></li> <li><code>docker pull jeroenwillemsen/wrongsecrets</code>- <a href="https://hub.docker.com/r/jeroenwillemsen/wrongsecrets">OWASP WrongSecrets</a></li> <li><code>docker run -dit --name trd -p 8081:80 cylabs/cy-threat-response</code> - <a href="https://hub.docker.com/r/cylabs/cy-threat-response">Cyware Threat Response Docker</a></li> <li><code>docker-compose -d up</code> - <a href="https://github.com/cider-security-research/cicd-goat">cicd-goat</a></li> </ul> https://tedneward.github.io/Research/security/supertokens/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/supertokens/index.html <p><a href="https://supertokens.com/">Website</a> | <a href="https://github.com/supertokens/supertokens-core">Source</a> | <a href="https://supertokens.io/docs/community/introduction">Docs</a></p> <p>Supertokens architecture is optimized to add secure authentication for your users without compromising on user and developer experience</p> <p>Three building blocks of SuperTokens architecture</p> <ul> <li>Frontend SDK: Manages session tokens and renders login UI widgets</li> <li>Backend SDK: Provides APIs for sign-up, sign-in, signout, session refreshing, etc. Your Frontend will talk to these APIs</li> <li>SuperTokens Core: The HTTP service for the core auth logic and database operations. This service is used by the Backend SDK</li> </ul> <p>SuperTokens is an open-core alternative to proprietary login providers like Auth0 or AWS Cognito. We are different because we offer:</p> <ul> <li>Open source: SuperTokens can be used for free, forever, with no limits on the number of users.</li> <li>An on-premises deployment so that you control 100% of your user data, using your own database.</li> <li>An end-to-end solution with login, sign-ups, user and session management, without all the complexities of OAuth protocols.</li> <li>Ease of implementation and higher security.</li> <li>Extensibility: Anyone can contribute and make SuperTokens better!</li> </ul> <h3>Philosophy</h3> <p>Authentication directly affects the UX, dev experience, and security of any app. We believe that current solutions cannot optimize for all three "pillars", leading to many applications hand-rolling their own auth. This not only leads to security issues but is also a massive time drain.</p> <p>We want to change that - we believe the only way is to provide a solution that has the right level of abstraction gives you maximum control, is secure, and is simple to use - just like if you build it yourself, from scratch (minus the time to learn, build, and maintain).</p> <p>We also believe in the principle of least vendor lock-in. Your having full control of your user's data means that you can switch away from SuperTokens without forcing your existing users to logout, reset their passwords, or in the worst case, sign up again.</p> <hr> <h1>SuperTokens and "vanilla" JS</h1> <p><a href="https://supertokens.com/docs/web-js/index.html">Docs</a> | <a href="https://github.com/supertokens/supertokens-web-js">Source</a></p> <p>Using SuperTokens with vanilla JS involves two main parts: the frontend using the <code>supertokens-web-js</code> SDK and a backend that uses a SuperTokens Backend SDK (for Node.js, Python, Go, etc.).</p> <p>Installation: <code>npm i -s supertokens-website</code></p> <p>Usage:</p> <pre><code>supertokens.init({ apiDomain: "&lt;URL to your auth backend&gt;" }); </code></pre> https://tedneward.github.io/Research/reading/processes/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/index/index.html <p><a href="https://age-of-product.com/sprint-anti-patterns/">"28 Sprint Anti-Patterns"</a></p> <p><a href="https://uxdesign.cc/the-age-of-agile-must-end-bc89c0f084b7">The Age of Agile Must End</a>: "As far as manifestos go, this is one of the more pathetic documents ever created. It lists four "Values" and twelve "Principles." They may have been well-intentioned at their inception — that intention was to treat developers as humans, not cogs in a machine — but now we’ve gone full circle. These principles have devolved into something caustic for organizations. Viewed now, they’re variations on a theme of how a scrum pod can absolve themselves of any accountability.</p> <ul> <li>Individuals and interactions over processes and tools. <em>(Yeah, we’re gonna do things however we want)</em></li> <li>Welcome changing requirements, even in late development. <em>(We can also change our minds whenever)</em></li> <li>Projects are built around motivated individuals, who should be trusted. <em>(Leave us alone, take what you get)</em></li> <li>Working software is the primary measure of progress. <em>(The fact that we produced something, anything, is all that matters)</em></li> <li>Simplicity — the art of maximizing the amount of work not done — is essential. <em>(We are going to do as little as possible)</em></li> <li>Best architectures, requirements, and designs emerge from self-organizing teams. <em>(Don’t tell us what to do)</em></li> </ul> <p>"The combination of Agile principles and Scrum practices is disastrous for startups. These are operational directives from management; designers, PM’s, and engineers are not self-organizing and choosing to work this way. It’s all in the name of an "MVP" and time to market; this is what happens. <em>Every. Time.</em>"</p> <p><a href="https://www.infoq.com/articles/real-options-enhance-agility/">"Real options" underlie agile practices</a>: Whether people realise it or not, "freedom to choose" is the underlying principle behind many of the agile practices. We call this principle Real Options. An understanding of Real Options allows us to develop and refine new agile practices and take agile into directions it hasn't gone before. Real Options also help us understand why some people resist some of the practices. <em>(I disagree with the premise, but the rest of the article has some good weight to it.)</em></p> <p><a href="https://www.researchgate.net/publication/313360479_Software_Development_Waste">"Software Development Waste"</a>: "The purpose of this paper is to identify and describe different kinds of waste in software development."</p> <p><a href="https://launchschool.com/books/agile_planning">Agile Planning: From Ideas to Story Cards</a> - Launch School</p> <p><a href="https://trunkbaseddevelopment.com/">Trunk-Based Development</a>: Pushing to main/master ain't all as bad as it's cracked up to be. "A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’ *, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge hell, do not break the build, and live happily ever after."</p> <p><a href="https://tuple.app/pair-programming-guide/">Pair Programming Guide</a>, including the <a href="https://tuple.app/pair-programming-guide/antipatterns">Pair Programming Antipatterns</a></p> <p><a href="https://salfreudenberg.wordpress.com/2017/03/26/the-tag-team-tools-tasks-and-roles-in-collaborative-software-development/">"The 'tag team': tools, tasks and roles in collaborative software development"</a> (<a href="https://salfreudenberg.files.wordpress.com/2017/03/finalthesis.pdf">PDF</a>)</p> https://tedneward.github.io/Research/reading/processes/scrum/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/scrum/index.html <p><a href="http://www.infoq.com/minibooks/scrum-xp-from-the-trenches-2">Scrum and XP from the Trenches</a></p> https://tedneward.github.io/Research/security/betterauth/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/betterauth/index.html <p><a href="https://www.better-auth.com/">Website</a> | <a href="https://github.com/better-auth/better-auth">Source</a></p> https://tedneward.github.io/Research/security/oauth/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/oauth/index.html <h2>Reading</h2> <ul> <li><a href="https://pages.apigee.com/oauth-big-picture-ebook.html">OAuth - The Big Picture</a> (email address <em>requested</em>)</li> </ul> https://tedneward.github.io/Research/security/web/index.html Mon, 13 Apr 2026 14:04:56 +0000 security/web/index.html <h3>Organizations</h3> <ul> <li><a href="http://www.owasp.org">OWASP</a> - The Open Web Application Security Project (OWASP) is a 501(c)(3) worldwide not-for-profit charitable organization focused on improving the security of software.</li> <li><a href="https://portswigger.net">Portswigger</a> - PortSwigger offers tools for web application security, testing &amp; scanning. Choose from a wide range of security tools &amp; identify the very latest vulnerabilities.</li> </ul> <h3>Web Application Firewall</h3> <ul> <li><a href="http://www.modsecurity.org/">ModSecurity</a> - ModSecurity is a toolkit for real-time web application monitoring, logging, and access control.</li> <li><a href="https://github.com/bunkerity/bunkerweb">BunkerWeb</a> - BunkerWeb is a full-featured open-source web server with ModeSecurity WAF, HTTPS with transparent Let's Encrypt renewal, automatic ban of strange behaviors based on HTTP codes, bot and bad IPs block, connection limits, state-of-the-art security presets, Web UI and much more.</li> <li><a href="https://github.com/nbs-system/naxsi">NAXSI</a> - NAXSI is an open-source, high performance, low rules maintenance WAF for NGINX, NAXSI means Nginx Anti Xss &amp; Sql Injection.</li> <li><a href="https://github.com/uptimejp/sql_firewall">sql_firewall</a> SQL Firewall Extension for PostgreSQL</li> <li><a href="https://github.com/ironbee/ironbee">ironbee</a> - IronBee is an open source project to build a universal web application security sensor. IronBee as a framework for developing a system for securing web applications - a framework for building a web application firewall (WAF).</li> <li><a href="https://github.com/curiefense/curiefense">Curiefense</a> - Curiefense adds a broad set of automated web security tools, including a WAF to Envoy Proxy.</li> <li><a href="https://github.com/openappsec/openappsec">open-appsec</a> - open-appsec is an open source machine-learning security engine that preemptively and automatically prevents threats against Web Application &amp; APIs.</li> </ul> <h3>Scanning / Pentesting</h3> <ul> <li><a href="https://spyse.com/">Spyse</a> - Spyse is an OSINT search engine that provides fresh data about the entire web. All the data is stored in its own DB for instant access and interconnected with each other for flexible search.<br> Provided data: IPv4 hosts, sub/domains/whois, ports/banners/protocols, technologies, OS, AS, wide SSL/TLS DB and more.</li> <li><a href="http://sqlmap.org/">sqlmap</a> - sqlmap is an open source penetration testing tool that automates the process of detecting and exploiting SQL injection flaws and taking over of database servers. It comes with a powerful detection engine, many niche features for the ultimate penetration tester and a broad range of switches lasting from database fingerprinting, over data fetching from the database, to accessing the underlying file system and executing commands on the operating system via out-of-band connections.</li> <li><a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project">ZAP</a> - The Zed Attack Proxy (ZAP) is an easy to use integrated penetration testing tool for finding vulnerabilities in web applications. It is designed to be used by people with a wide range of security experience and as such is ideal for developers and functional testers who are new to penetration testing. ZAP provides automated scanners as well as a set of tools that allow you to find security vulnerabilities manually.</li> <li><a href="https://www.owasp.org/index.php/Testing_Checklist">OWASP Testing Checklist v4</a> - List of some controls to test during a web vulnerability assessment. Markdown version may be found <a href="https://github.com/amocrenco/owasp-testing-checklist-v4-markdown/blob/master/README.md">here</a>.</li> <li><a href="http://w3af.org/">w3af</a> - w3af is a Web Application Attack and Audit Framework. The project’s goal is to create a framework to help you secure your web applications by finding and exploiting all web application vulnerabilities.</li> <li><a href="https://github.com/lanmaster53/recon-ng">Recon-ng</a> - Recon-ng is a full-featured Web Reconnaissance framework written in Python. Recon-ng has a look and feel similar to the Metasploit Framework.</li> <li><a href="https://github.com/trustedsec/ptf">PTF</a> - The Penetration Testers Framework (PTF) is a way for modular support for up-to-date tools.</li> <li><a href="https://github.com/guardicore/monkey">Infection Monkey</a> - A semi automatic pen testing tool for mapping/pen-testing networks. Simulates a human attacker.</li> <li><a href="https://github.com/tijme/angularjs-csti-scanner">ACSTIS</a> - ACSTIS helps you to scan certain web applications for AngularJS Client-Side Template Injection (sometimes referred to as CSTI, sandbox escape or sandbox bypass). It supports scanning a single request but also crawling the entire web application for the AngularJS CSTI vulnerability.</li> <li><a href="https://github.com/KishanBagaria/padding-oracle-attacker">padding-oracle-attacker</a> - padding-oracle-attacker is a CLI tool and library to execute padding oracle attacks (which decrypts data encrypted in CBC mode) easily, with support for concurrent network requests and an elegant UI.</li> <li><a href="https://github.com/lirantal/is-website-vulnerable">is-website-vulnerable</a> - finds publicly known security vulnerabilities in a website's frontend JavaScript libraries.</li> <li><a href="https://github.com/nil0x42/phpsploit">PhpSploit</a> - Full-featured C2 framework which silently persists on webserver via evil PHP oneliner. Built for stealth persistence, with many privilege-escalation &amp; post-exploitation features.</li> <li><a href="https://github.com/SpectralOps/keyscope">Keyscope</a> - Keyscope is an extensible key and secret validation for checking active secrets against multiple SaaS vendors built in Rust</li> <li><a href="https://github.com/v8blink/Chromium-based-XSS-Taint-Tracking">Cyclops</a> - The Cyclops is a web browser with XSS detection feature, it is chromium-based xss detection that used to find the flows from a source to a sink.</li> <li><a href="https://github.com/marcinguy/scanmycode-ce">Scanmycode CE (Community Edition)</a> - Code Scanning/SAST/Static Analysis/Linting using many tools/Scanners with One Report. Currently supports: PHP, Java, Scala, Python, Ruby, Javascript, GO, Secret Scanning, Dependency Confusion, Trojan Source, Open Source and Proprietary Checks (total ca. 1000 checks)</li> <li><a href="https://github.com/rusty-ferris-club/recon">recon</a> - a fast Rust based CLI that uses SQL to query over files, code, or malware with content classification and processing for security experts</li> <li><a href="https://github.com/Zigrin-Security/CakeFuzzer">CakeFuzzer</a> - The ultimate web application security testing tool for CakePHP-based web applications. CakeFuzzer employs a predefined set of attacks that are randomly modified before execution. Leveraging its deep understanding of the Cake PHP framework, Cake Fuzzer launches attacks on all potential application entry points.</li> <li><a href="https://github.com/CERT-Polska/Artemis/">Artemis</a> - A modular vulnerability scanner with automatic report generation capabilities.</li> </ul> <h3>Runtime Application Self-Protection</h3> <ul> <li><a href="https://www.sqreen.io/">Sqreen</a> - Sqreen is a Runtime Application Self-Protection (RASP) solution for software teams. An in-app agent instruments and monitors the app. Suspicious user activities are reported and attacks are blocked at runtime without code modification or traffic redirection.</li> <li><a href="https://github.com/baidu/openrasp">OpenRASP</a> - An open source RASP solution actively maintained by Baidu Inc. With context-aware detection algorithm the project achieved nearly no false positives. And less than 3% performance reduction is observed under heavy server load.</li> </ul> <h3>Development</h3> <ul> <li><a href="https://www.manning.com/books/api-security-in-action">API Security in Action</a> - Book covering API security including secure development, token-based authentication, JSON Web Tokens, OAuth 2, and Macaroons. (early access, published continuously, final release summer 2020)</li> <li><a href="https://www.manning.com/books/secure-by-design?a_aid=danbjson&amp;a_bid=0b3fac80">Secure by Design</a> - Book that identifies design patterns and coding styles that make lots of security vulnerabilities less likely. (early access, published continuously, final release fall 2017)</li> <li><a href="https://www.manning.com/books/understanding-api-security">Understanding API Security</a> - Free eBook sampler that gives some context for how API security works in the real world by showing how APIs are put together and how the OAuth protocol can be used to protect them.</li> <li><a href="https://www.manning.com/books/oauth-2-in-action">OAuth 2 in Action</a> - Book that teaches you practical use and deployment of OAuth 2 from the perspectives of a client, an authorization server, and a resource server.</li> <li><a href="https://github.com/zaproxy/zap-api-nodejs">OWASP ZAP Node API</a> - Leverage the OWASP Zed Attack Proxy (ZAP) within your NodeJS applications with this official API.</li> <li><a href="https://github.com/apps/guardrails">GuardRails</a> - A GitHub App that provides security feedback in Pull Requests.</li> <li><a href="https://github.com/Bearer/bearer">Bearer</a> - Scan code for security risks and vulnerabilities leading to sensitive data exposures.</li> <li><a href="https://github.com/bridgecrewio/checkov/">Checkov</a> - A static analysis tool for infrastucture as code (Terraform).</li> <li><a href="https://github.com/tfsec/tfsec/">TFSec</a> - A static analysis tool for infrastucture as code (Terraform).</li> <li><a href="https://github.com/Checkmarx/kics">KICS</a> - Scans IaC projects for security vulnerabilities, compliance issues, and infrastructure misconfiguration. Currently working with Terraform projects, Kubernetes manifests, Dockerfiles, AWS CloudFormation Templates, and Ansible playbooks.</li> <li><a href="https://github.com/insidersec/insider">Insider CLI</a> - A open source Static Application Security Testing tool (SAST) written in GoLang for Java (Maven and Android), Kotlin (Android), Swift (iOS), .NET Full Framework, C# and Javascript (Node.js).</li> <li><a href="https://www.manning.com/books/full-stack-python-security">Full Stack Python Security</a> - A comprehensive look at cybersecurity for Python developers</li> <li><a href="https://www.manning.com/books/making-sense-of-cyber-security">Making Sense of Cyber Security</a> - A jargon-free, practical guide to the key concepts, terminology, and technologies of cybersecurity perfect for anyone planning or implementing a security strategy. (early access, published continuously, final release early 2022)</li> <li><a href="https://owasp.org/www-project-application-security-verification-standard/">Security Checklist by OWASP</a> - A checklist by OWASP for testing web applications based on assurance level. Covers multiple topics like Architecture, IAM, Sanitization, Cryptography and Secure Configuration.</li> </ul> https://tedneward.github.io/Research/reading/levels-of-war/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/levels-of-war/index.html <p><a href="https://www.clausewitz.com/readings/Dunn.htm">Levels of War: Just a Set of Labels?</a></p> <p><a href="https://thestrategybridge.org/the-bridge/2016/5/5/the-institutional-level-of-war">The Institutional Level of War</a></p> https://tedneward.github.io/Research/reading/processes/devops/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/processes/devops/index.html <h2>Articles</h2> <p><a href="https://github.com/MichaelCade/90DaysOfDevOps">90 Days of DevOps</a>: "This repository is my documenting repository for learning the world of DevOps. I started this journey on the 1st January 2022 and I plan to run to March 31st for a complete 90-day romp on spending an hour a day including weekends to get a foundational knowledge across a lot of different areas that make up DevOps." Including:</p> <ul> <li>What is and why do we use DevOps</li> <li>Learning a Programming Language</li> <li>Knowing Linux Basics</li> <li>Understand Networking</li> <li>Stick to one Cloud Provider (he uses Azure)</li> <li>Use Git Effectively</li> <li>Containers (Docker)</li> <li>Kubernetes</li> <li>Learn Infrastructure as Code (Terraform)</li> <li>Automate Configuration Management (Ansible)</li> <li>Create CI/CD Pipelines (Jenkins)</li> <li>Monitoring, Log Management, and Data Visualization</li> <li>Store &amp; Protect Your Data</li> </ul> https://tedneward.github.io/Research/reading/development/static-program-analysis/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/static-program-analysis/index.html <ul> <li>Static Analysis Roadmap <ul> <li><a href="https://tpiazza.me/posts/2017-11-01-static-analysis-roadmap.html">https://tpiazza.me/posts/2017-11-01-static-analysis-roadmap.html</a></li> </ul> </li> <li>Best Practices for the use of Static Code Analysis within a Real-World Secure Development Lifecycle <ul> <li>2015; Jeremy Boone</li> <li><a href="https://www.nccgroup.trust/us/our-research/best-practices-for-the-use-of-static-code-analysis-within-a-real-world-secure-development-lifecycle/">https://www.nccgroup.trust/us/our-research/best-practices-for-the-use-of-static-code-analysis-within-a-real-world-secure-development-lifecycle/</a></li> </ul> </li> <li>Bill Torpey - <a href="http://btorpey.github.io/blog/categories/static-analysis/">static analysis posts</a> <ul> <li>Static Analysis with Clang - <a href="http://btorpey.github.io/blog/2015/04/27/static-analysis-with-clang/">http://btorpey.github.io/blog/2015/04/27/static-analysis-with-clang/</a></li> <li>Mo' Static - <a href="http://btorpey.github.io/blog/2016/04/07/mo-static/">http://btorpey.github.io/blog/2016/04/07/mo-static/</a></li> <li>Even Mo' Static - <a href="http://btorpey.github.io/blog/2016/11/12/even-mo-static/">http://btorpey.github.io/blog/2016/11/12/even-mo-static/</a></li> <li>Lots o' static - <a href="http://btorpey.github.io/blog/2017/09/17/lotso-static/">http://btorpey.github.io/blog/2017/09/17/lotso-static/</a></li> </ul> </li> <li>OASIS Static Analysis Results Interchange Format (SARIF) TC <ul> <li>Defining a standard output format for static analysis tools</li> <li><a href="https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=sarif">https://www.oasis-open.org/committees/tc_home.php?wg_abbrev=sarif</a></li> <li><a href="https://github.com/oasis-tcs/sarif-spec">https://github.com/oasis-tcs/sarif-spec</a></li> <li>Static Analysis Results: A Format and a Protocol: SARIF &amp; SASP <ul> <li><a href="https://blogs.grammatech.com/static-analysis-results-a-format-and-a-protocol-sarif-sasp">https://blogs.grammatech.com/static-analysis-results-a-format-and-a-protocol-sarif-sasp</a></li> </ul> </li> <li>Integration of the Static Analysis Results Interchange Format in CogniCrypt <ul> <li>2019 arXiv</li> <li>Sriteja Kummita, Goran Piskachev</li> <li><a href="https://arxiv.org/abs/1907.02558">https://arxiv.org/abs/1907.02558</a></li> </ul> </li> </ul> </li> <li><a href="https://semantic-domain.blogspot.com/2019/08/on-relationship-between-static-analysis.html">On the Relationship Between Static Analysis and Type Theory</a></li> </ul> <h2>Books</h2> <ul> <li><a href="https://cs.au.dk/~amoeller/spa/">Static Program Analysis</a> by Anders Møller, Michael I. Schwartzbach</li> </ul> <h2>Research</h2> <ul> <li>Static Analysis Symposia Central Site <ul> <li><a href="http://staticanalysis.org/">http://staticanalysis.org/</a></li> </ul> </li> <li>A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World <ul> <li>Communications of the ACM, Vol. 53 No. 2, 2010</li> <li>Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, Dawson Engler</li> <li><a href="https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext">https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext</a></li> </ul> </li> <li>Automated Program Transformation for Improving Software Quality <ul> <li>2019 PhD Dissertation; Rijnard van Tonder</li> <li><a href="https://www.cs.cmu.edu/~rvantond/pdfs/rijnard-dissertation.pdf">https://www.cs.cmu.edu/~rvantond/pdfs/rijnard-dissertation.pdf</a></li> </ul> </li> <li>How to Build Static Checking Systems Using Orders of Magnitude Less Code <ul> <li>ASPLOS 2016</li> <li>Fraser Brown, Andres Nötzli, Dawson Engler</li> <li><a href="https://web.stanford.edu/~mlfbrown/paper.pdf">https://web.stanford.edu/~mlfbrown/paper.pdf</a></li> <li><a href="http://lambda-the-ultimate.org/node/5348">http://lambda-the-ultimate.org/node/5348</a></li> <li><a href="https://blog.acolyer.org/2016/05/31/how-to-build-static-checking-systems-using-orders-of-magnitude-less-code/">https://blog.acolyer.org/2016/05/31/how-to-build-static-checking-systems-using-orders-of-magnitude-less-code/</a></li> </ul> </li> <li>Lessons from Building Static Analysis Tools at Google <ul> <li>Communications of the ACM, Vol. 61 No. 4, 2018</li> <li>Caitlin Sadowski, Edward Aftandilian, Alex Eagle, Liam Miller-Cushon, Ciera Jaspan</li> <li><a href="https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/abstract">https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/abstract</a></li> <li><a href="https://research.google.com/pubs/pub46576.html">https://research.google.com/pubs/pub46576.html</a></li> </ul> </li> <li>Object Model Construction for Inheritance in C++ and Its Applications to Program Analysis <ul> <li>Compiler Construction (CC) 2012</li> <li>Yang, Jing, Gogul Balakrishnan, Naoto Maeda, Franjo Ivancic, Aarti Gupta, Nishant Sinha, Sriram Sankaranarayanan, Naveen Sharma</li> <li><a href="https://dl.acm.org/citation.cfm?id=2259241">https://dl.acm.org/citation.cfm?id=2259241</a></li> <li><a href="http://pages.cs.wisc.edu/~bgogul/Research/Papers/cc12.html">http://pages.cs.wisc.edu/~bgogul/Research/Papers/cc12.html</a></li> <li><a href="https://www.semanticscholar.org/paper/Object-Model-Construction-for-Inheritance-in-C%2B%2B-a-Yang-Balakrishnan/510501c7051c03d5e2a70089deeda8dfc3a7304f">https://www.semanticscholar.org/paper/Object-Model-Construction-for-Inheritance-in-C%2B%2B-a-Yang-Balakrishnan/510501c7051c03d5e2a70089deeda8dfc3a7304f</a></li> <li><a href="https://www.cs.colorado.edu/~srirams/papers/cc12-final.pdf">https://www.cs.colorado.edu/~srirams/papers/cc12-final.pdf</a></li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.225.2429">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.225.2429</a></li> </ul> </li> <li>Scaling Static Analyses at Facebook <ul> <li>Dino Distefano, Manuel Fähndrich, Francesco Logozzo, Peter W. O'Hearn</li> <li>Communications of the ACM, Vol. 62 No. 8, 2019</li> <li><a href="https://cacm.acm.org/magazines/2019/8/238344-scaling-static-analyses-at-facebook/fulltext">https://cacm.acm.org/magazines/2019/8/238344-scaling-static-analyses-at-facebook/fulltext</a></li> </ul> </li> <li>Source Language Representation of Function Summaries in Static Analysis <ul> <li>ICOOOLPS / ECOOP 2016</li> <li>Gábor Horváth, Norbert Pataki</li> <li><a href="https://2016.ecoop.org/event/icooolps-2016-source-language-representation-of-function-summaries-in-static-analysis">https://2016.ecoop.org/event/icooolps-2016-source-language-representation-of-function-summaries-in-static-analysis</a></li> </ul> </li> <li>Tailoring Programs for Static Analysis via Program Transformation <ul> <li>International Conference on Software Engineering (ICSE) 2020</li> <li>Rijnard van Tonder and Claire Le Goues</li> <li><a href="https://www.cs.cmu.edu/~rvantond/pdfs/tailoring-analysis-icse-2020.pdf">https://www.cs.cmu.edu/~rvantond/pdfs/tailoring-analysis-icse-2020.pdf</a></li> </ul> </li> <li>Toward Full Elasticity in Distributed Static Analysis <ul> <li>Diego Garbervetsky, Edgardo Zoppi, Thomas Ball, Ben Livshits</li> <li>2016 Microsoft Research Technical Report: MSR-TR-2015-88</li> <li><a href="https://www.microsoft.com/en-us/research/publication/toward-full-elasticity-distributed-static-analysis/">https://www.microsoft.com/en-us/research/publication/toward-full-elasticity-distributed-static-analysis/</a></li> </ul> </li> <li>Undecidability of static analysis <ul> <li>ACM Letters on Programming Languages and Systems (LOPLAS) Volume 1 Issue 4, Dec. 1992</li> <li>William Landi</li> <li><a href="https://dl.acm.org/citation.cfm?id=161501">https://dl.acm.org/citation.cfm?id=161501</a></li> <li>"Two fundamental static-analysis problems are may alias and must alias. The former is not recursive (is undecidable), and the latter is not recursively enumerable (is uncomputable), even when all paths are executable in the program being analyzed for languages with if statements, loops, dynamic storage, and recursive data structures."</li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/testing/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/testing/index.html <p><a href="http://principlesofchaos.org/">Principles of Chaos Engineering</a></p> <p><a href="https://www.jamesshore.com/v2/projects/nullables/testing-without-mocks">"Testing Without Mocks: A Pattern Language"</a></p> <p>See also: <a href="/reading/development/fuzzing">Fuzzing</a> || <a href="/languages/compilers.correctness">Compilers Correctness</a></p> <ul> <li><a href="https://www.sigsoft.org/">ACM Special Interest Group on Software Engineering (SIGSOFT)</a> <ul> <li>Open Table of Contents (TOC) Resources - <a href="https://www.sigsoft.org/resources/opentoc.html">https://www.sigsoft.org/resources/opentoc.html</a></li> </ul> </li> <li>Testing Standards Working Party <ul> <li><a href="http://www.testingstandards.co.uk/">http://www.testingstandards.co.uk/</a></li> <li>Living Glossary of Testing Terms - <a href="http://www.testingstandards.co.uk/living_glossary.htm">http://www.testingstandards.co.uk/living_glossary.htm</a></li> </ul> </li> <li>cpp-testing-no-excuses <ul> <li><a href="https://github.com/d-led/cpp-testing-no-excuses">https://github.com/d-led/cpp-testing-no-excuses</a></li> </ul> </li> <li>What Is a Good Test Case? <ul> <li>2003; Cem Kaner</li> <li><a href="http://www.kaner.com/pdfs/GoodTest.pdf">http://www.kaner.com/pdfs/GoodTest.pdf</a></li> </ul> </li> </ul> <hr> <h1>Blogs</h1> <ul> <li>Google Testing Blog <ul> <li><a href="https://testing.googleblog.com/">https://testing.googleblog.com/</a></li> </ul> </li> <li>Evolving Understanding About Exploratory Testing <ul> <li><a href="https://www.developsense.com/blog/2008/09/evolving-understanding-about/">https://www.developsense.com/blog/2008/09/evolving-understanding-about/</a></li> </ul> </li> <li>Exploratory Testing <ul> <li><a href="https://www.satisfice.com/exploratory-testing">https://www.satisfice.com/exploratory-testing</a></li> </ul> </li> <li>How SQLite Is Tested <ul> <li><a href="https://sqlite.org/testing.html">https://sqlite.org/testing.html</a></li> </ul> </li> <li>How the GNU coreutils are tested <ul> <li><a href="http://www.pixelbeat.org/docs/coreutils-testing.html">http://www.pixelbeat.org/docs/coreutils-testing.html</a></li> </ul> </li> <li>Principles of Automated Testing <ul> <li><a href="http://www.lihaoyi.com/post/PrinciplesofAutomatedTesting.html">http://www.lihaoyi.com/post/PrinciplesofAutomatedTesting.html</a></li> </ul> </li> <li>Refactoring for testability in C++ <ul> <li>Hard-to-test patterns in C++ and how to refactor them.</li> <li><a href="https://github.com/platisd/refactoring-for-testability-cpp">https://github.com/platisd/refactoring-for-testability-cpp</a></li> </ul> </li> <li>Testing in Production, the safe way <ul> <li><a href="https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1">https://medium.com/@copyconstruct/testing-in-production-the-safe-way-18ca102d0ef1</a></li> </ul> </li> </ul> <h2>Blogs: Series</h2> <ul> <li><a href="https://abseil.io/tips/">Abseil C++ Tips of the Week (TotW)</a> <ul> <li> <h1>122: Test Fixtures, Clarity, and Dataflow - <a href="https://abseil.io/tips/122">https://abseil.io/tips/122</a></h1> </li> <li> <h1>135: Test the Contract, not the Implementation - <a href="https://abseil.io/tips/135">https://abseil.io/tips/135</a></h1> </li> </ul> </li> <li>James Grenning <ul> <li>TDD How-to: Get your Legacy C Into a Test Harness <ul> <li><a href="https://wingman-sw.com/articles/tdd-legacy-c">https://wingman-sw.com/articles/tdd-legacy-c</a></li> <li><a href="https://github.com/jwgrenning/cpputest-starter-project">https://github.com/jwgrenning/cpputest-starter-project</a></li> </ul> </li> <li>TDD Guided by ZOMBIES <ul> <li><a href="http://blog.wingman-sw.com/tdd-guided-by-zombies">http://blog.wingman-sw.com/tdd-guided-by-zombies</a></li> </ul> </li> <li>Test-Driven Development For Embedded C++ Programmers <ul> <li><a href="https://www.wingman-sw.com/articles/test-driven-development-for-embedded-c-plus-plus">https://www.wingman-sw.com/articles/test-driven-development-for-embedded-c-plus-plus</a></li> </ul> </li> </ul> </li> <li>John Regehr <ul> <li>Software Testing Using Function/Inverse Pairs - <a href="https://blog.regehr.org/archives/708">https://blog.regehr.org/archives/708</a></li> </ul> </li> <li>Kent Beck <ul> <li>Programmer Test Principles <ul> <li><a href="https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934">https://medium.com/@kentbeck_7670/programmer-test-principles-d01c064d7934</a></li> </ul> </li> <li>“Unit” Tests? <ul> <li><a href="https://www.facebook.com/notes/kent-beck/unit-tests/1726369154062608/">https://www.facebook.com/notes/kent-beck/unit-tests/1726369154062608/</a></li> </ul> </li> </ul> </li> <li>Nelson Elhage <ul> <li>Two kinds of testing - <a href="https://blog.nelhage.com/post/two-kinds-of-testing/">https://blog.nelhage.com/post/two-kinds-of-testing/</a></li> <li>How I Write Tests - <a href="https://blog.nelhage.com/2016/12/how-i-test/">https://blog.nelhage.com/2016/12/how-i-test/</a></li> <li>Design for Testability - <a href="https://blog.nelhage.com/2016/03/design-for-testability/">https://blog.nelhage.com/2016/03/design-for-testability/</a></li> <li>Test suites as classifiers - <a href="https://blog.nelhage.com/post/test-suites-as-classifiers/">https://blog.nelhage.com/post/test-suites-as-classifiers/</a></li> </ul> </li> <li>William Caputo <ul> <li>TDD: "Failing to Falsify" - <a href="http://logosity.net/notes.html#2017.02">http://logosity.net/notes.html#2017.02</a></li> <li>The goal of "unit testing" in TDD - <a href="http://logosity.net/notes.html#2017.09">http://logosity.net/notes.html#2017.09</a></li> </ul> </li> </ul> <hr> <h1>Books</h1> <ul> <li>Software Testing: From Theory to Practice <ul> <li>Maurício Aniche &amp; Arie van Deursen</li> <li><a href="https://sttp.site/">https://sttp.site/</a></li> </ul> </li> <li>The Fuzzing Book: Tools and Techniques for Generating Software Tests <ul> <li>Andreas Zeller, Rahul Gopinath, Marcel Böhme, Gordon Fraser, Christian Holler</li> <li><a href="https://www.fuzzingbook.org/">https://www.fuzzingbook.org/</a></li> </ul> </li> </ul> <hr> <h1>Courses</h1> <ul> <li>CMPT 473: Software Testing, Reliability and Security <ul> <li><a href="https://www.cs.sfu.ca/~wsumner/">Nick Sumner</a></li> <li><a href="http://www2.cs.sfu.ca/~wsumner/teaching/473/">http://www2.cs.sfu.ca/~wsumner/teaching/473/</a></li> </ul> </li> <li>Software Analysis and Testing <ul> <li><a href="http://www.cis.upenn.edu/~mhnaik/">Mayur Naik</a></li> <li><a href="https://RightingCode.org/">https://RightingCode.org/</a></li> <li><a href="http://www.cis.upenn.edu/~mhnaik/edu/cis700/">http://www.cis.upenn.edu/~mhnaik/edu/cis700/</a></li> <li><a href="https://www.youtube.com/channel/UCvwqRhlkE_Wm2FF9qzvHfJw">https://www.youtube.com/channel/UCvwqRhlkE_Wm2FF9qzvHfJw</a></li> </ul> </li> <li>Software Testing: How to Make Software Fail <ul> <li>John Regehr &amp; Sean Bennett</li> <li><a href="https://www.udacity.com/course/software-testing--cs258">https://www.udacity.com/course/software-testing--cs258</a></li> </ul> </li> </ul> <hr> <h1>Research</h1> <h2>Research: 2010s</h2> <ul> <li>Assertions Are Strongly Correlated with Test Suite Effectiveness <ul> <li>ESEC/FSE 2015</li> <li>Yucheng Zhang, Ali Mesbah</li> <li><a href="http://salt.ece.ubc.ca/publications/fse15.html">http://salt.ece.ubc.ca/publications/fse15.html</a></li> </ul> </li> <li>Assurances in Software Testing: A Roadmap <ul> <li>ICSE-NIER 2019: International Conference on Software Engineering: New Ideas and Emerging Results</li> <li>Marcel Böhme</li> <li><a href="https://arxiv.org/abs/1807.10255">https://arxiv.org/abs/1807.10255</a></li> </ul> </li> <li>AUSTIN (AUgmented Search-based TestINg): An open source tool for search based software testing of C programs <ul> <li>Search Based Software Engineering (SSBSE) 2010; Information and Software Technology, 55(1), 2013</li> <li>Kiran Lakhotia, Mark Harman, Hamilton Gross</li> <li><a href="http://www0.cs.ucl.ac.uk/staff/K.Lakhotia/docs/ist12.pdf">http://www0.cs.ucl.ac.uk/staff/K.Lakhotia/docs/ist12.pdf</a></li> <li><a href="http://www0.cs.ucl.ac.uk/staff/mharman/ist-kiran.pdf">http://www0.cs.ucl.ac.uk/staff/mharman/ist-kiran.pdf</a></li> <li><a href="http://www0.cs.ucl.ac.uk/staff/Yuanyuan.Zhang/StuConOS/StuConOS-Kiran.pdf">http://www0.cs.ucl.ac.uk/staff/Yuanyuan.Zhang/StuConOS/StuConOS-Kiran.pdf</a></li> <li><a href="https://code.google.com/archive/p/austin-sbst/">https://code.google.com/archive/p/austin-sbst/</a></li> <li><a href="https://github.com/kiranlak/austin-sbst">https://github.com/kiranlak/austin-sbst</a></li> </ul> </li> <li>Compiler Instrumentation for Testing <ul> <li><a href="https://github.com/martong/finstrument_mock">https://github.com/martong/finstrument_mock</a></li> <li>Unit Testing in C++ with Compiler Instrumentation and Friends <ul> <li>Acta Cybernetica 23 (2017)</li> <li>Gábor Márton and Zoltán Porkoláb</li> <li><a href="http://www.inf.u-szeged.hu/actacybernetica/edb/vol23n2/actacyb_23_2_2017_14.xml">http://www.inf.u-szeged.hu/actacybernetica/edb/vol23n2/actacyb_23_2_2017_14.xml</a></li> <li><a href="http://www.inf.u-szeged.hu/actacybernetica/edb/vol23n2/pdf/actacyb_23_2_2017_14.pdf">http://www.inf.u-szeged.hu/actacybernetica/edb/vol23n2/pdf/actacyb_23_2_2017_14.pdf</a></li> </ul> </li> <li>Compile-Time Function Call Interception for Testing in C/C++ <ul> <li>Studia Universitatis Babeș-Bolyai Informatica, v. 63, n. 1, 2018</li> <li>Gábor Márton, Zoltán Porkoláb</li> <li><a href="http://www.cs.ubbcluj.ro/~studia-i/journal/journal/article/view/19">http://www.cs.ubbcluj.ro/~studia-i/journal/journal/article/view/19</a></li> <li><a href="https://www.researchgate.net/publication/325900400_Compile-Time_Function_Call_Interception_for_Testing_in_CC">https://www.researchgate.net/publication/325900400_Compile-Time_Function_Call_Interception_for_Testing_in_CC</a></li> </ul> </li> </ul> </li> <li>Does Refactoring of Test Smells Induce Fixing Flaky Tests? <ul> <li>International Conference on Software Maintenance and Evolution (ICSME) 2017</li> <li>Fabio Palomba and Andy Zaidman</li> <li><a href="https://azaidman.github.io/publications/palombaICSME2017.pdf">https://azaidman.github.io/publications/palombaICSME2017.pdf</a></li> <li><a href="https://www.slideshare.net/fabiopalomba/does-refactoring-of-test-smells-induce-fixing-flaky-tests">https://www.slideshare.net/fabiopalomba/does-refactoring-of-test-smells-induce-fixing-flaky-tests</a></li> </ul> </li> <li>Regression Testing Minimisation, Selection and Prioritisation: A Survey <ul> <li>Software Testing, Verification and Reliability 22, 2 (2012)</li> <li>Shin Yoo and Mark Harman</li> <li><a href="http://www0.cs.ucl.ac.uk/staff/M.Harman/stvr-shin-survey.pdf">http://www0.cs.ucl.ac.uk/staff/M.Harman/stvr-shin-survey.pdf</a></li> </ul> </li> <li>STADS: Software Testing as Species Discovery <ul> <li>ACM Trans. Softw. Eng. Methodol. 27(2) 2018</li> <li>Marcel Böhme</li> <li><a href="https://dl.acm.org/citation.cfm?doid=3210309">https://dl.acm.org/citation.cfm?doid=3210309</a></li> <li><a href="https://arxiv.org/abs/1803.02130">https://arxiv.org/abs/1803.02130</a></li> </ul> </li> <li>Testing C++ generic libraries <ul> <li>International Conference on Software Maintenance (ICSM) 2012</li> <li>Andrew Sutton, Marcin Zalewski</li> <li><a href="https://ieeexplore.ieee.org/document/6405251">https://ieeexplore.ieee.org/document/6405251</a></li> <li><a href="https://www.ii.uib.no/~magne/inf328s17-info/inf328-2017s-bldl-presentation-08-SuttonZalewski2012icsm-06405251.pdf">https://www.ii.uib.no/~magne/inf328s17-info/inf328-2017s-bldl-presentation-08-SuttonZalewski2012icsm-06405251.pdf</a></li> </ul> </li> <li>The Art of Testing Less Without Sacrificing Quality <ul> <li>ICSE 2015</li> <li>Kim Herzig, Michaela Greiler, Jacek Czerwonka, Brendan Murphy</li> <li><a href="https://www.microsoft.com/en-us/research/publication/the-art-of-testing-less-without-sacrificing-quality/">https://www.microsoft.com/en-us/research/publication/the-art-of-testing-less-without-sacrificing-quality/</a></li> <li><a href="https://blog.acolyer.org/2015/06/25/the-art-of-testing-less-without-sacrificing-quality/">https://blog.acolyer.org/2015/06/25/the-art-of-testing-less-without-sacrificing-quality/</a></li> </ul> </li> <li>The Oracle Problem in Software Testing: A Survey <ul> <li>IEEE Transactions on Software Engineering 41(5) 2015</li> <li>Earl Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, Shin Yoo</li> <li><a href="https://ieeexplore.ieee.org/document/6963470">https://ieeexplore.ieee.org/document/6963470</a></li> <li><a href="http://earlbarr.com/publications/testoracles.pdf">http://earlbarr.com/publications/testoracles.pdf</a></li> <li>Repository of Publications on the Test Oracle Problem <ul> <li><a href="http://crestweb.cs.ucl.ac.uk/resources/oracle_repository/">http://crestweb.cs.ucl.ac.uk/resources/oracle_repository/</a></li> </ul> </li> </ul> </li> <li>Tools and Language Elements for Testing, Encapsulation and Controlling Abstraction in Large Scale C++ Projects <ul> <li>2019 Doctoral dissertation; Gábor Márton</li> <li><a href="http://www.tnkcs.inf.elte.hu/vedes/Marton_Gabor_Ertekezes1.pdf">http://www.tnkcs.inf.elte.hu/vedes/Marton_Gabor_Ertekezes1.pdf</a></li> <li><a href="https://www.researchgate.net/publication/337717375_Tools_and_Language_Elements_for_Testing_Encapsulation_and_Controlling_Abstraction_in_Large_Scale_C_Projects">https://www.researchgate.net/publication/337717375_Tools_and_Language_Elements_for_Testing_Encapsulation_and_Controlling_Abstraction_in_Large_Scale_C_Projects</a></li> </ul> </li> <li>When Testing Meets Code Review: Why and How Developers Review Tests <ul> <li>ICSE 2018</li> <li>Davide Spadini, Mauricio Aniche, Margaret-Anne Storey, Magiel Bruntink, Alberto Bacchelli</li> <li><a href="https://pure.tudelft.nl/portal/en/publications/when-testing-meets-code-review-why-and-how-developers-review-tests(256e7d56-352f-44ae-919b-97fad0eafe69).html">https://pure.tudelft.nl/portal/en/publications/when-testing-meets-code-review-why-and-how-developers-review-tests(256e7d56-352f-44ae-919b-97fad0eafe69).html</a></li> <li><a href="https://pure.tudelft.nl/portal/files/38853938/PID5219697.pdf">https://pure.tudelft.nl/portal/files/38853938/PID5219697.pdf</a></li> </ul> </li> </ul> <h2>Research: 2000s</h2> <ul> <li>Software Testing and Industry Needs <ul> <li>IEEE Software, vol. 23, no. 4, July-Aug. 2006</li> <li>R. L. Glass, R. Collard, A. Bertolino, J. Bach, C. Kaner</li> <li><a href="https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1657939">https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=1657939</a></li> <li><a href="https://ieeexplore.ieee.org/document/1657939/">https://ieeexplore.ieee.org/document/1657939/</a></li> </ul> </li> <li>Software Testing Research: Achievements, Challenges, Dreams <ul> <li>Future of Software Engineering 2007; Antonia Bertolino</li> <li><a href="http://selab.netlab.uky.edu/homepage/sw-test-roadmap-bertolino.pdf">http://selab.netlab.uky.edu/homepage/sw-test-roadmap-bertolino.pdf</a></li> </ul> </li> </ul> <h2>Research: 1990s</h2> <ul> <li>Bringing Testing Into the Fold <ul> <li>IEEE Software 13(3): 91-92 (1996)</li> <li>Edward V. Berard</li> <li><a href="https://doi.org/10.1109/52.493025">https://doi.org/10.1109/52.493025</a></li> </ul> </li> <li>PIE: A Dynamic Failure-Based Technique <ul> <li>Jeffrey M. Voas</li> <li>IEEE Transactions on Software Engineering, Volume 18 Issue 8, 1992</li> <li><a href="https://dl.acm.org/citation.cfm?id=136558">https://dl.acm.org/citation.cfm?id=136558</a></li> <li><a href="http://www.cs.odu.edu/~mln/ltrs-pdfs/NASA-92-ieeeswe.jmv.pdf">http://www.cs.odu.edu/~mln/ltrs-pdfs/NASA-92-ieeeswe.jmv.pdf</a></li> <li>PIE: propagation, infection, and execution</li> </ul> </li> </ul> <h2>Research: 1980s</h2> <ul> <li>On Testing Non-Testable Programs <ul> <li>The Computer Journal, vol. 25, no. 4, 1982</li> <li>E. J. Weyuker</li> <li><a href="https://academic.oup.com/comjnl/article/25/4/465/366384">https://academic.oup.com/comjnl/article/25/4/465/366384</a></li> </ul> </li> </ul> <hr> <h1>Concurrency</h1> <ul> <li>A Survey of Recent Trends in Testing Concurrent Software Systems <ul> <li>IEEE Transactions on Software Engineering (TSE), vol. 44, no. 8, 2018</li> <li><a href="https://www.computer.org/csdl/trans/ts/2018/08/07932530-abs.html">https://www.computer.org/csdl/trans/ts/2018/08/07932530-abs.html</a></li> <li><a href="https://home.deib.polimi.it/margara/papers/2017_tse_testing_concurrent_software.pdf">https://home.deib.polimi.it/margara/papers/2017_tse_testing_concurrent_software.pdf</a></li> </ul> </li> <li>Applications of Synchronization Coverage <ul> <li>Principles and Practice of Parallel Programming (PPoPP) 2005</li> <li>Arkady Bron, Eitan Farchi, Yonit Magid, Yarden Nir, Shmuel Ur</li> <li><a href="https://doi.org/10.1145/1065944.1065972">https://doi.org/10.1145/1065944.1065972</a></li> </ul> </li> <li>Are Concurrency Coverage Metrics Effective for Testing: A Comprehensive Empirical Investigation <ul> <li>Journal of Software Testing, Verification and Reliability (STVR) 25(4) 2015</li> <li>S. Hong, M. Staats, J. Ahn, M. Kim, G. Rothermel</li> <li><a href="http://onlinelibrary.wiley.com/doi/10.1002/stvr.1539/abstract">http://onlinelibrary.wiley.com/doi/10.1002/stvr.1539/abstract</a></li> <li><a href="https://orbilu.uni.lu/bitstream/10993/16540/1/main-journal-concurrent-cov-effectiveness.pdf">https://orbilu.uni.lu/bitstream/10993/16540/1/main-journal-concurrent-cov-effectiveness.pdf</a></li> </ul> </li> <li>Cuzz – Concurrency Fuzzing <ul> <li><a href="https://www.microsoft.com/en-us/research/project/cuzz-concurrency-fuzzing/">https://www.microsoft.com/en-us/research/project/cuzz-concurrency-fuzzing/</a></li> </ul> </li> </ul> <hr> <h1>Continuous Integration</h1> <ul> <li>ci_helloworld: A simple example of how to setup a complete CI environment for C and C++ <ul> <li><a href="https://github.com/ainfosec/ci_helloworld">https://github.com/ainfosec/ci_helloworld</a></li> <li>A Test a Day Keeps Your Manager Away! <ul> <li>CppCon 2017; Rian Quinn</li> <li><a href="https://www.youtube.com/watch?v=KdJhQuycD78">https://www.youtube.com/watch?v=KdJhQuycD78</a></li> <li>"During this session we will step through an open source project designed to demonstrate how to integrate different C++ analysis tools into your CI services. These tools include static analysis (Clang Tidy, Coverity Scan, Codeacy and CppCheck), dynamic analysis (Valgrind and Google's Sanitizers), source formatting (Astyle and Clang Format), documentation (Doxygen), code coverage (Codecov, Coveralls, and LLVM's Software-based Code Coverage), cross platform tests (Windows, Cygwin, Linux, and macOS), compiler tests (GCC, Clang, and Visual Studio) and finally C++ libraries designed to assist in reliability and automated testing (Catch, Hippomocks and the Guideline Support Library). In addition we will openly discuss the advantages and disadvantages of using various analysis tools, how to integrate these tools into existing projects (both large and small) as well as common problems encountered while using these tools autonomously in a CI environment."</li> </ul> </li> </ul> </li> <li>Code Coverage &amp; Continuous Integration <ul> <li>2019 ATPESC Tutorial: Better Scientific Software</li> <li>Jared O'Neal</li> <li><a href="https://www.youtube.com/watch?v=8JZi7OvGGCc">https://www.youtube.com/watch?v=8JZi7OvGGCc</a></li> <li><a href="https://doi.org/10.6084/m9.figshare.9272813">https://doi.org/10.6084/m9.figshare.9272813</a></li> <li>Examples <ul> <li>Travis CI Hello World: Simplest CI example <ul> <li><a href="https://github.com/jrdoneal/CI_HelloWorld">https://github.com/jrdoneal/CI_HelloWorld</a></li> <li><a href="https://travis-ci.org/jrdoneal/CI_HelloWorld">https://travis-ci.org/jrdoneal/CI_HelloWorld</a></li> </ul> </li> <li>CI_Multiplatform: Multiplatform/Multicompiler Travis CI Example <ul> <li>CI example w/ multiple platforms and specific compiler versions</li> <li><a href="https://github.com/jrdoneal/CI_Multiplatform">https://github.com/jrdoneal/CI_Multiplatform</a></li> </ul> </li> <li>Code coverage, testing, and CI example (Fortran, C++) <ul> <li><a href="https://github.com/jrdoneal/infrastructure">https://github.com/jrdoneal/infrastructure</a></li> </ul> </li> </ul> </li> </ul> </li> </ul> <h2>Continuous Integration: Readings</h2> <ul> <li>Automated Reporting of Anti-Patterns and Decay in Continuous Integration <ul> <li>ICSE 2019</li> <li>Carmine Vassallo, Sebastian Proksch, Harald Gall, Massimiliano Di Penta</li> <li><a href="http://doi.org/10.5281/zenodo.2578271">http://doi.org/10.5281/zenodo.2578271</a></li> </ul> </li> <li>Continuous delivery: Patterns and antipatterns in the software life cycle <ul> <li>DZone refcard #145, 2011; P. M. Duvall</li> <li><a href="https://dzone.com/refcardz/continuous-delivery-patterns">https://dzone.com/refcardz/continuous-delivery-patterns</a></li> </ul> </li> <li>Continuous Integration: Improving Software Quality and Reducing Risk <ul> <li>2007; P. Duvall, S. M. Matyas, A. Glover</li> <li><a href="http://www.integratebutton.com/">http://www.integratebutton.com/</a></li> </ul> </li> <li>Continuous Integration Theater <ul> <li>Empirical Software Engineering and Measurement (ESEM) 2019</li> <li>Wagner Felidré, Leonardo Furtado, Daniel da Costa, Bruno Cartaxo, Gustavo Pinto</li> <li><a href="https://arxiv.org/abs/1907.01602">https://arxiv.org/abs/1907.01602</a></li> <li><a href="http://gustavopinto.org/blog/continuous-integration-theater/">http://gustavopinto.org/blog/continuous-integration-theater/</a></li> </ul> </li> <li>Game Changers - Trunk Based Development <ul> <li><a href="https://trunkbaseddevelopment.com/game-changers/">https://trunkbaseddevelopment.com/game-changers/</a></li> </ul> </li> <li>How to get code coverage from CI <ul> <li><a href="https://codingnest.com/how-to-get-code-coverage-from-ci/">https://codingnest.com/how-to-get-code-coverage-from-ci/</a></li> </ul> </li> <li>Modern C++ CI <ul> <li>CMake, Catch, spdlog, Travis, AppVeyor</li> <li><a href="https://juan-medina.com/2017/07/01/moderncppci/">https://juan-medina.com/2017/07/01/moderncppci/</a></li> <li><a href="https://github.com/LearningByExample/ModernCppCI">https://github.com/LearningByExample/ModernCppCI</a></li> </ul> </li> <li>Taming Google-Scale Continuous Testing <ul> <li>ICSE 2017</li> <li>Atif Memon, Bao Nguyen, Eric Nickell, John Micco, Sanjeev Dhanda, Rob Siemborski, Zebao Gao</li> <li><a href="https://research.google/pubs/pub45861/">https://research.google/pubs/pub45861/</a></li> </ul> </li> </ul> <h2>Continuous Integration: Software</h2> <ul> <li>Azure DevOps C++ build tasks for CMake and vcpkg <ul> <li><a href="https://github.com/lukka/CppBuildTasks">https://github.com/lukka/CppBuildTasks</a></li> </ul> </li> <li>CMake Modern: C++, CMake and Travis-CI Hello World <ul> <li><a href="https://github.com/nboutin/cmake_modern">https://github.com/nboutin/cmake_modern</a></li> </ul> </li> <li>Continuous Integration Demo <ul> <li>A demo project for C++14 using CI systems like Travis CI, Circle CI, Google Test, Catch, etc.</li> <li><a href="https://github.com/daixtrose/continuous-integration-demo">https://github.com/daixtrose/continuous-integration-demo</a></li> </ul> </li> <li>GitLab CI for C++ projects <ul> <li><a href="https://mklimenko.github.io/english/2020/02/02/gitlab-ci-cpp/">https://mklimenko.github.io/english/2020/02/02/gitlab-ci-cpp/</a></li> </ul> </li> </ul> <h3>Continuous Integration: Software: AppVeyor</h3> <ul> <li>Intro to AppVeyor - C++ Weekly - Ep 80 <ul> <li><a href="https://www.youtube.com/watch?v=R8OrWVVf5CM">https://www.youtube.com/watch?v=R8OrWVVf5CM</a></li> </ul> </li> <li>Boost.Hana .appveyor.yml <ul> <li><a href="https://github.com/boostorg/hana/blob/master/.appveyor.yml">https://github.com/boostorg/hana/blob/master/.appveyor.yml</a></li> </ul> </li> </ul> <h3>Continuous Integration: Software: GitHub Actions</h3> <ul> <li>Using GitHub Actions with C++ and CMake <ul> <li><a href="https://cristianadam.eu/20191222/using-github-actions-with-c-plus-plus-and-cmake/">https://cristianadam.eu/20191222/using-github-actions-with-c-plus-plus-and-cmake/</a></li> </ul> </li> <li>Speeding up C++ GitHub Actions using ccache <ul> <li><a href="https://cristianadam.eu/20200113/speeding-up-c-plus-plus-github-actions-using-ccache/">https://cristianadam.eu/20200113/speeding-up-c-plus-plus-github-actions-using-ccache/</a></li> </ul> </li> </ul> <h3>Continuous Integration: Software: Travis</h3> <ul> <li>.travis.yml Examples <ul> <li>Boost.Hana: clang++, g++, Boost, Valgrind - <a href="https://github.com/boostorg/hana/blob/master/.travis.yml">https://github.com/boostorg/hana/blob/master/.travis.yml</a></li> <li>Catch2: <a href="https://github.com/catchorg/Catch2/blob/master/.travis.yml">https://github.com/catchorg/Catch2/blob/master/.travis.yml</a></li> <li>meta: <a href="https://github.com/Leandros/meta/blob/master/.travis.yml">https://github.com/Leandros/meta/blob/master/.travis.yml</a></li> <li>MPark.Variant: <a href="https://github.com/mpark/variant/blob/master/.travis.yml">https://github.com/mpark/variant/blob/master/.travis.yml</a></li> <li>xstd: clang++, g++, binutils, Boost, libc++, lcov, lld, lldb - <a href="https://github.com/rhalbersma/xstd/blob/master/.travis.yml">https://github.com/rhalbersma/xstd/blob/master/.travis.yml</a></li> </ul> </li> <li>C++ CI: Clang, Travis, CMake, GTest, Coveralls &amp; Appveyor <ul> <li><a href="http://david-grs.github.io/cpp-clang-travis-cmake-gtest-coveralls-appveyor/">http://david-grs.github.io/cpp-clang-travis-cmake-gtest-coveralls-appveyor/</a></li> <li><a href="https://github.com/david-grs/clang_travis_cmake_gtest_coveralls_example">https://github.com/david-grs/clang_travis_cmake_gtest_coveralls_example</a></li> </ul> </li> <li>Intro To Travis CI - C++ Weekly - Ep 79 <ul> <li><a href="https://www.youtube.com/watch?v=3ulKzD2cmSw">https://www.youtube.com/watch?v=3ulKzD2cmSw</a></li> </ul> </li> <li>Continuous integration with Travis CI <ul> <li><a href="https://arne-mertz.de/2017/04/continuous-integration-travis-ci/">https://arne-mertz.de/2017/04/continuous-integration-travis-ci/</a></li> </ul> </li> <li>icc-travis: Script to help install Intel C/C++ Compiler on Travis CI. <ul> <li><a href="https://github.com/nemequ/icc-travis">https://github.com/nemequ/icc-travis</a></li> </ul> </li> <li>Minimal Travis CI example for modern C++ <ul> <li><a href="https://github.com/mpoullet/travis-cpp">https://github.com/mpoullet/travis-cpp</a></li> </ul> </li> <li>Richel Bilderbeek <ul> <li>C++ Travis CI tutorial <ul> <li><a href="http://richelbilderbeek.nl/CppTravisCiTutorial.htm">http://richelbilderbeek.nl/CppTravisCiTutorial.htm</a></li> <li><a href="https://github.com/richelbilderbeek/travis_cpp_tutorial">https://github.com/richelbilderbeek/travis_cpp_tutorial</a></li> </ul> </li> <li>Favorite C++ setup <ul> <li><a href="https://github.com/richelbilderbeek/the_richel_setup">https://github.com/richelbilderbeek/the_richel_setup</a></li> </ul> </li> </ul> </li> <li>TravisTorrent: Synthesizing Travis CI and GitHub for Full-Stack Research on Continuous Integration <ul> <li><a href="https://travistorrent.testroots.org/">https://travistorrent.testroots.org/</a></li> </ul> </li> </ul> <hr> <h1>Coverage</h1> <h2>Coverage: Readings</h2> <ul> <li>A Large-Scale, Longitudinal Study of Test Coverage Evolution <ul> <li>Automated Software Engineering (ASE) 2018</li> <li>Michael Hilton, Jonathan Bell, Darko Marinov</li> <li><a href="http://jonbell.net/publications/coverage">http://jonbell.net/publications/coverage</a></li> <li><a href="https://www.code-coverage.org/publications/">https://www.code-coverage.org/publications/</a></li> <li>Code Coverage Analytics: Understanding how lines are (un)covered <ul> <li><a href="https://www.code-coverage.org/">https://www.code-coverage.org/</a></li> </ul> </li> </ul> </li> <li>Applications of Synchronization Coverage <ul> <li>Principles and Practice of Parallel Programming (PPoPP) 2005</li> <li>Arkady Bron, Eitan Farchi, Yonit Magid, Yarden Nir, Shmuel Ur</li> <li><a href="https://doi.org/10.1145/1065944.1065972">https://doi.org/10.1145/1065944.1065972</a></li> </ul> </li> <li>Assessing Oracle Quality with Checked Coverage <ul> <li>International Conference on Software Testing, Verification and Validation ICST 2011</li> <li>David Schuler and Andreas Zeller</li> <li><a href="https://dl.acm.org/citation.cfm?id=1990075">https://dl.acm.org/citation.cfm?id=1990075</a></li> </ul> </li> <li>Automatic Self-Validation for Code Coverage Profilers <ul> <li>ASE 2019</li> <li>Yibiao Yang, Yanyan Jiang, Zhiqiang Zuo, Yang Wang, Hao Sun, Hongmin Lu, Yuming Zhou, Baowen Xu</li> <li><a href="https://2019.ase-conferences.org/details/ase-2019-papers/2/Automatic-Self-Validation-for-Code-Coverage-Profilers">https://2019.ase-conferences.org/details/ase-2019-papers/2/Automatic-Self-Validation-for-Code-Coverage-Profilers</a></li> </ul> </li> <li>Can Testedness be Effectively Measured <ul> <li>FSE 2016</li> <li>Ahmed, Gopinath, Brindescu, Groce, Jensen</li> <li><a href="https://rahul.gopinath.org/resources/fse2016/ahmed2016can.pdf">https://rahul.gopinath.org/resources/fse2016/ahmed2016can.pdf</a></li> <li><a href="https://speakerdeck.com/ahmedi/can-testedness-be-effectively-measured">https://speakerdeck.com/ahmedi/can-testedness-be-effectively-measured</a></li> <li><a href="https://rahul.gopinath.org/publications/#ahmed2016can">https://rahul.gopinath.org/publications/#ahmed2016can</a></li> </ul> </li> <li>Code Coverage at Google <ul> <li>ESEC/FSE 2019</li> <li>Marko Ivanković, Goran Petrović, René Just, Gordon Fraser</li> <li><a href="https://homes.cs.washington.edu/~rjust/publ/google_coverage_fse_2019.pdf">https://homes.cs.washington.edu/~rjust/publ/google_coverage_fse_2019.pdf</a></li> <li><a href="https://ai.google/research/pubs/pub48413/">https://ai.google/research/pubs/pub48413/</a></li> </ul> </li> <li>Code Coverage for Suite Evaluation by Developers <ul> <li>ICSE 2014</li> <li>Rahul Gopinath, Carlos Jensen, Alex Groce</li> <li><a href="http://research.engr.oregonstate.edu/hci/sites/research.engr.oregonstate.edu.hci/files/papers/gopinath2014code_0.pdf">http://research.engr.oregonstate.edu/hci/sites/research.engr.oregonstate.edu.hci/files/papers/gopinath2014code_0.pdf</a></li> </ul> </li> <li>Code Coverage is a Strong Predictor of Test Suite Effectiveness <ul> <li>GTAC 2016</li> <li>Rahul Gopinath</li> <li><a href="https://www.youtube.com/watch?v=NKEptA3KP08">https://www.youtube.com/watch?v=NKEptA3KP08</a></li> <li><a href="https://speakerdeck.com/rahulgopinath/code-coverage-is-a-strong-predictor-of-test-suite-effectiveness-in-the-real-world">https://speakerdeck.com/rahulgopinath/code-coverage-is-a-strong-predictor-of-test-suite-effectiveness-in-the-real-world</a></li> <li><a href="https://docs.google.com/presentation/d/1_bNn_HTI1Vst6WAB62KHTtgbvxocjpnt2Y-8ugGyP8U">https://docs.google.com/presentation/d/1_bNn_HTI1Vst6WAB62KHTtgbvxocjpnt2Y-8ugGyP8U</a></li> </ul> </li> <li>Comparing non-adequate test suites using coverage criteria <ul> <li>International Symposium on Software Testing and Analysis (2013)</li> <li>Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, A., Marinov, D.</li> <li><a href="https://agroce.github.io/issta13.pdf">https://agroce.github.io/issta13.pdf</a></li> </ul> </li> <li>Coverage and Its Discontents <ul> <li>Onward! Essays, SPLASH 2014</li> <li>Alex Groce, Mohammad Amin Alipour, Rahul Gopinath</li> <li><a href="https://agroce.github.io/onwardessays14.pdf">https://agroce.github.io/onwardessays14.pdf</a></li> </ul> </li> <li>Coverage is Not Strongly Correlated with Test Suite Effectiveness <ul> <li>International Conference on Software Engineering 2014</li> <li>Laura Inozemtseva and Reid Holmes</li> <li><a href="http://www.linozemtseva.com/research/2014/icse/coverage/">http://www.linozemtseva.com/research/2014/icse/coverage/</a></li> <li>ICSE 2014: <a href="https://vimeo.com/92273013">https://vimeo.com/92273013</a></li> <li>GTAC 2015: <a href="https://www.youtube.com/watch?v=sAfROROGujU">https://www.youtube.com/watch?v=sAfROROGujU</a></li> </ul> </li> <li>Covrig: A Framework for the Analysis of Code, Test, and Coverage Evolution in Real Software <ul> <li>ISSTA 2014</li> <li>Paul Marinescu, Petr Hosek, Cristian Cadar</li> <li><a href="https://srg.doc.ic.ac.uk/publications/covrig-issta-14.html">https://srg.doc.ic.ac.uk/publications/covrig-issta-14.html</a></li> </ul> </li> <li>Guidelines for coverage-based comparisons of non-adequate test suites <ul> <li>ACM Transactions on Software Engineering and Methodology (2015)</li> <li>Gligoric, M., Groce, A., Zhang, C., Sharma, R., Alipour, A., Marinov, D.</li> <li><a href="https://dl.acm.org/citation.cfm?id=2660767">https://dl.acm.org/citation.cfm?id=2660767</a></li> <li><a href="http://mir.cs.illinois.edu/coco/">http://mir.cs.illinois.edu/coco/</a></li> <li><a href="http://users.ece.utexas.edu/~gligoric/papers/GligoricETAL15CoCoJournal.pdf">http://users.ece.utexas.edu/~gligoric/papers/GligoricETAL15CoCoJournal.pdf</a></li> </ul> </li> <li>How to Misuse Code Coverage <ul> <li>1999; Brian Marick</li> <li><a href="http://www.exampler.com/testing-com/writings/coverage.pdf">http://www.exampler.com/testing-com/writings/coverage.pdf</a></li> </ul> </li> <li>Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing <ul> <li>ICSE 2019</li> <li>Yibiao Yang, Yuming Zhou, Hao Sun, Zhendong Su, Zhiqiang Zuo, Lei Xu, Baowen Xu</li> <li><a href="https://2019.icse-conferences.org/event/icse-2019-technical-papers-how-reliable-are-code-coverage-tools-">https://2019.icse-conferences.org/event/icse-2019-technical-papers-how-reliable-are-code-coverage-tools-</a></li> </ul> </li> <li>Invasive Software Testing: Mutating Target Programs to Diversify Test Exploration for High Test Coverage <ul> <li>IEEE International Conference on Software Testing, Verification and Validation (ICST) 2018</li> <li>Y. Kim, S. Hong, B. Ko, L. Phan, M. Kim</li> <li><a href="http://swtv.kaist.ac.kr/publications/icst18-deminer.pdf">http://swtv.kaist.ac.kr/publications/icst18-deminer.pdf</a></li> </ul> </li> <li>Mythical Unit Test Coverage <ul> <li>IEEE Software, Volume 35, Issue 3, May/June 2018</li> <li>V. Antinyan, J. Derehag, A. Sandberg, M. Staron</li> <li><a href="https://ieeexplore.ieee.org/document/8354427/">https://ieeexplore.ieee.org/document/8354427/</a></li> <li><a href="https://www.researchgate.net/publication/324959836_Mythical_Unit_Test_Coverage">https://www.researchgate.net/publication/324959836_Mythical_Unit_Test_Coverage</a></li> </ul> </li> <li>Not All Coverage Measurements Are Equal: Fuzzing by Coverage Accounting for Input Prioritization <ul> <li>Network and Distributed System Security Symposium (NDSS) 2020</li> <li>Yanhao Wang, Xiangkun Jia, Yuwei Liu, Tiffany Bao, Dinghao Wu, and Purui Su</li> <li><a href="https://ajax4sec.github.io/papers/ndss20-fall-paper422.pdf">https://ajax4sec.github.io/papers/ndss20-fall-paper422.pdf</a></li> </ul> </li> <li>Selecting Software Test Data Using Data Flow Information <ul> <li>IEEE Transactions on Software Engineering 11(4) 1985</li> <li>Sandra Rapps and Elaine J. Weyuker</li> <li><a href="http://ieeexplore.ieee.org/document/1702019/">http://ieeexplore.ieee.org/document/1702019/</a></li> </ul> </li> <li>TestCov: Robust Test-Suite Execution and Coverage Measurement <ul> <li>ASE 2019</li> <li>Dirk Beyer, Thomas Lemberger</li> <li><a href="https://2019.ase-conferences.org/details/ase-2019-Demonstrations/34/TestCov-Robust-Test-Suite-Execution-and-Coverage-Measurement">https://2019.ase-conferences.org/details/ase-2019-Demonstrations/34/TestCov-Robust-Test-Suite-Execution-and-Coverage-Measurement</a></li> </ul> </li> <li>Turning Programs against Each Other: High Coverage Fuzz-Testing using Binary-Code Mutation and Dynamic Slicing <ul> <li>ESEC/FSE 2015</li> <li>Ulf Kargén and Nahid Shahmehri</li> <li><a href="https://www.ida.liu.se/~ulfka17/papers/FSE2015.pdf">https://www.ida.liu.se/~ulfka17/papers/FSE2015.pdf</a></li> </ul> </li> <li>Verification, coverage and maximization: The big picture <ul> <li><a href="https://blog.foretellix.com/2016/12/23/verification-coverage-and-maximization-the-big-picture/">https://blog.foretellix.com/2016/12/23/verification-coverage-and-maximization-the-big-picture/</a></li> </ul> </li> </ul> <h2>Coverage: Software</h2> <ul> <li>afl-cov - AFL Fuzzing Code Coverage <ul> <li><a href="https://github.com/mrash/afl-cov">https://github.com/mrash/afl-cov</a></li> </ul> </li> <li>cmake_catch_coverage <ul> <li>Integration of CMake, Catch and CMake CodeCoverage module for C++ code.</li> <li>(CMake CodeCoverage depends on: gcov, gcovr, genhtml, lcov.)</li> <li><a href="https://github.com/fkromer/catch_cmake_coverage">https://github.com/fkromer/catch_cmake_coverage</a></li> </ul> </li> <li>Code Coverage in Chromium <ul> <li><a href="https://chromium.googlesource.com/chromium/src/+/master/docs/testing/code_coverage.md">https://chromium.googlesource.com/chromium/src/+/master/docs/testing/code_coverage.md</a></li> </ul> </li> <li>CompareCoverage <ul> <li>CompareCoverage (CmpCov in short) is a simple instrumentation module for C/C++ programs and libraries, which extracts information about data comparisons taking place in the code at run time, and saves it to disk in the form of standard .sancov files. It is based on the SanitizerCoverage instrumentation available in the clang compiler, which itself is tightly related to AddressSanitizer. Specifically, the library implements the instrumentation callbacks defined by the Tracing data flow feature of SanitizerCoverage.</li> <li><a href="https://github.com/googleprojectzero/CompareCoverage">https://github.com/googleprojectzero/CompareCoverage</a></li> </ul> </li> <li>CovNavi: Code coverage navigation and analysis <ul> <li><a href="https://github.com/Cisco-Talos/covnavi">https://github.com/Cisco-Talos/covnavi</a></li> <li>Fuzzing driven code auditing and vice versa: Coverage analysis for better fuzzing <ul> <li><a href="https://talosintelligence.com/resources/61">https://talosintelligence.com/resources/61</a></li> </ul> </li> </ul> </li> <li>cpp-llvm-coverage: How to use LLVM coverage for for C++ <ul> <li><a href="https://github.com/Longhanks/cpp-llvm-coverage">https://github.com/Longhanks/cpp-llvm-coverage</a></li> </ul> </li> <li>drcov: a DynamoRIO client tool that collects code coverage information <ul> <li>Dr. Memory provides optional code coverage information to aid in fuzz testing (Fuzz Testing Mode) or in general testing.</li> <li><a href="http://dynamorio.org/docs/page_drcov.html">http://dynamorio.org/docs/page_drcov.html</a></li> <li><a href="http://drmemory.org/docs/page_coverage.html">http://drmemory.org/docs/page_coverage.html</a></li> <li><a href="https://github.com/DynamoRIO/dynamorio/tree/master/clients/drcov">https://github.com/DynamoRIO/dynamorio/tree/master/clients/drcov</a></li> </ul> </li> <li>GNU gcov <ul> <li><a href="https://gcc.gnu.org/onlinedocs/gcc/Gcov.html">https://gcc.gnu.org/onlinedocs/gcc/Gcov.html</a></li> <li>Writing Better Function Tests with GCOV <ul> <li>Linaro Connect 2018; Masami Hiramatsu</li> <li><a href="https://www.youtube.com/watch?v=U_qWLa9KnW8">https://www.youtube.com/watch?v=U_qWLa9KnW8</a></li> </ul> </li> <li>Gcovr <ul> <li>Gcovr provides a utility for managing the use of the GNU gcov utility and generating summarized code coverage results.</li> <li><a href="http://gcovr.com/">http://gcovr.com/</a></li> <li><a href="https://github.com/gcovr/gcovr">https://github.com/gcovr/gcovr</a></li> </ul> </li> <li>GCovHTML - Generate reports on code coverage <ul> <li><a href="https://gitlab.com/stone.code/gcovhtml">https://gitlab.com/stone.code/gcovhtml</a></li> </ul> </li> <li>Dev Santa Claus Part 2: C++ code coverage metrics with gcov <ul> <li>setting up code coverage metrics for a C++ codebase built using Bamboo, CMake and GCC.</li> <li><a href="https://genbattle.bitbucket.io/blog/2018/01/19/Dev-Santa-Claus-Part-2/">https://genbattle.bitbucket.io/blog/2018/01/19/Dev-Santa-Claus-Part-2/</a></li> </ul> </li> <li>Using Gcov and Lcov to generate beautiful C++ code coverage statistics <ul> <li><a href="https://codeflu.blog/2014/12/26/using-gcov-and-lcov-to-generate-beautiful-c-code-coverage-statistics/">https://codeflu.blog/2014/12/26/using-gcov-and-lcov-to-generate-beautiful-c-code-coverage-statistics/</a></li> </ul> </li> </ul> </li> <li>Kcov <ul> <li>Code coverage tool for compiled programs, Python and Bash which uses debugging information to collect and report data without special compilation options</li> <li><a href="http://simonkagstrom.github.io/kcov/">http://simonkagstrom.github.io/kcov/</a></li> <li><a href="https://github.com/simonkagstrom/kcov">https://github.com/simonkagstrom/kcov</a></li> <li>C++ Weekly - Ep 111 - kcov - <a href="https://www.youtube.com/watch?v=NRCnS5alouI">https://www.youtube.com/watch?v=NRCnS5alouI</a></li> <li>Kcov - gcov, lcov and bcov - <a href="https://simonkagstrom.livejournal.com/50380.html">https://simonkagstrom.livejournal.com/50380.html</a></li> <li>Kcov - a single-step code coverage tool <ul> <li>Simon Kågström; SwedenCpp::Stockholm::0x0F, September 20, 2018</li> <li><a href="https://www.youtube.com/watch?v=1QMHbp5LUKg">https://www.youtube.com/watch?v=1QMHbp5LUKg</a></li> </ul> </li> </ul> </li> <li>OpenCppCoverage: an open source code coverage tool for C++ under Windows <ul> <li><a href="https://github.com/OpenCppCoverage/OpenCppCoverage">https://github.com/OpenCppCoverage/OpenCppCoverage</a></li> </ul> </li> <li>LLVM <ul> <li>Clang Source-based Code Coverage <ul> <li><a href="https://clang.llvm.org/docs/SourceBasedCodeCoverage.html">https://clang.llvm.org/docs/SourceBasedCodeCoverage.html</a></li> </ul> </li> <li>SanitizerCoverage <ul> <li><a href="https://clang.llvm.org/docs/SanitizerCoverage.html">https://clang.llvm.org/docs/SanitizerCoverage.html</a></li> <li>A quick tour of LLVM’s Sanitizer coverage <ul> <li><a href="https://tech.labs.oliverwyman.com/blog/2017/10/04/a-quick-tour-of-llvms-sanitizer-coverage/">https://tech.labs.oliverwyman.com/blog/2017/10/04/a-quick-tour-of-llvms-sanitizer-coverage/</a></li> </ul> </li> </ul> </li> </ul> </li> <li>TestCov: Test execution, coverage measurement, and test-suite reduction <ul> <li><a href="https://gitlab.com/sosy-lab/software/test-suite-validator">https://gitlab.com/sosy-lab/software/test-suite-validator</a></li> </ul> </li> </ul> <h2>Coverage: Mutation</h2> <p>Mutation Analysis, Mutation Coverage, Mutation Testing</p> <ul> <li>Mutation testing resources: how to make better code by introducing bugs <ul> <li><a href="https://github.com/theofidry/mutation-testing">https://github.com/theofidry/mutation-testing</a></li> </ul> </li> <li><a href="https://mutation-testing.slack.com/">https://mutation-testing.slack.com/</a></li> </ul> <h3>Coverage: Mutation: Readings</h3> <ul> <li>Mutation Testing Repository <ul> <li><a href="https://mutationtesting.uni.lu/">https://mutationtesting.uni.lu/</a></li> </ul> </li> <li>Evaluating Test Effectiveness with Mutation Analysis <ul> <li><a href="https://www.fuzzingbook.org/html/MutationAnalysis.html">https://www.fuzzingbook.org/html/MutationAnalysis.html</a></li> </ul> </li> <li>A Systematic Literature Review of How Mutation Testing Supports Quality Assurance Processes <ul> <li>Software Testing, Verification and Reliability 2018</li> <li>Qianqian Zhu, Annibale Panichella, Andy Zaidman</li> <li><a href="https://pure.tudelft.nl/portal/en/publications/a-systematic-literature-review-of-how-mutation-testing-supports-quality-assurance-processes(8767f68d-cea1-40b1-8933-a34825212860).html">https://pure.tudelft.nl/portal/en/publications/a-systematic-literature-review-of-how-mutation-testing-supports-quality-assurance-processes(8767f68d-cea1-40b1-8933-a34825212860).html</a></li> </ul> </li> <li>A Systematic Literature Review of Techniques and Metrics to Reduce the Cost of Mutation Testing <ul> <li>Journal of Systems and Software (2019)</li> <li>Pizzoleto, Alessandro Viola, Fabiano Cutigi Ferrari, Jeff Offutt, Leo Fernandes, Márcio Ribeiro</li> <li><a href="https://cs.gmu.edu/~offutt/rsrch/papers/SLR-CostReductionMutation.pdf">https://cs.gmu.edu/~offutt/rsrch/papers/SLR-CostReductionMutation.pdf</a></li> </ul> </li> <li>An Analysis and Survey of the Development of Mutation Testing <ul> <li>IEEE Trans. Software Eng. 37 (5) (2011)</li> <li>Y. Jia, M. Harman</li> <li><a href="http://dx.doi.org/10.1109/TSE.2010.62">http://dx.doi.org/10.1109/TSE.2010.62</a></li> <li><a href="http://crest.cs.ucl.ac.uk/fileadmin/crest/sebasepaper/JiaH10.pdf">http://crest.cs.ucl.ac.uk/fileadmin/crest/sebasepaper/JiaH10.pdf</a></li> </ul> </li> <li>An Empirical Study on Mutation, Statement and Branch Coverage Fault Revelation that Avoids the Unreliable Clean Program Assumption <ul> <li>ICSE 2017</li> <li>Titcheu Chekam Thierry, Mike Papadakis, Yves Le Traon, Mark Harman</li> <li><a href="https://sites.google.com/site/mikepapadakis/ICSE17.pdf">https://sites.google.com/site/mikepapadakis/ICSE17.pdf</a></li> <li><a href="https://sites.google.com/site/mikepapadakis/faults-mutants">https://sites.google.com/site/mikepapadakis/faults-mutants</a></li> <li><a href="https://sites.google.com/site/mikepapadakis/metallaxis">https://sites.google.com/site/mikepapadakis/metallaxis</a></li> </ul> </li> <li>An Industrial Application of Mutation Testing: Lessons, Challenges, and Research Directions <ul> <li>International Workshop on Mutation Analysis (Mutation 2018)</li> <li>Goran Petrovic, Marko Ivankovic, Robert Kurtz, Paul Ammann, René Just</li> <li><a href="https://research.google/pubs/pub46907/">https://research.google/pubs/pub46907/</a></li> </ul> </li> <li>Are Mutants a Valid Substitute for Real Faults in Software Testing? <ul> <li>Proceedings of the Symposium on the Foundations of Software Engineering 2014</li> <li>René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, Gordon Fraser</li> <li><a href="http://www.linozemtseva.com/research/2014/fse/mutant_validity">http://www.linozemtseva.com/research/2014/fse/mutant_validity</a></li> </ul> </li> <li>C++11/14 Mutation Operators Based on Common Fault Patterns <ul> <li>International Conference on Testing Software and Systems (ICTSS) 2018</li> <li>Parsai A., Demeyer S., De Busser S.</li> <li><a href="https://link.springer.com/chapter/10.1007%2F978-3-319-99927-2_9">https://link.springer.com/chapter/10.1007%2F978-3-319-99927-2_9</a></li> <li><a href="https://www.parsai.net/files/research/C++11-14%20Mutation%20Operators%20Based%20on%20Common%20Fault%20Patterns%20(pre-print).pdf">https://www.parsai.net/files/research/C++11-14%20Mutation%20Operators%20Based%20on%20Common%20Fault%20Patterns%20(pre-print).pdf</a></li> </ul> </li> <li>Detecting Trivial Mutant Equivalences via Compiler Optimisations <ul> <li>IEEE Transactions on Software Engineering 44(4) 2018</li> <li>Marinos Kintis, Mike Papadakis, Yue Jia, Nicos Malevris, Yves Le Traon, Mark Harman</li> <li><a href="http://pages.cs.aueb.gr/~kintism/papers/tce/">http://pages.cs.aueb.gr/~kintism/papers/tce/</a></li> <li><a href="https://sites.google.com/site/mikepapadakis/tce-tse.pdf?attredirects=0&amp;d=1">https://sites.google.com/site/mikepapadakis/tce-tse.pdf?attredirects=0&amp;d=1</a></li> </ul> </li> <li>How Verified (or Tested) is My Code? Falsification-Driven Verification and Testing <ul> <li>Automated Software Engineering (2018)</li> <li>Groce, Alex &amp; Ahmed, Iftekhar &amp; Jensen, Carlos &amp; Mckenney, Paul &amp; Holmes, Josie</li> <li><a href="https://agroce.github.io/asej18.pdf">https://agroce.github.io/asej18.pdf</a></li> </ul> </li> <li>If You Can’t Kill a Supermutant, You Have a Problem <ul> <li>ICSTW Mutation 2018</li> <li>Rahul Gopinath, Björn Mathis, Andreas Zeller</li> <li><a href="https://rahul.gopinath.org/publications/#gopinath2018if">https://rahul.gopinath.org/publications/#gopinath2018if</a></li> <li><a href="https://rahul.gopinath.org/resources/icstw2018/gopinath2018if.pdf">https://rahul.gopinath.org/resources/icstw2018/gopinath2018if.pdf</a></li> </ul> </li> <li>Is Mutation an Appropriate Tool for Testing Experiments? <ul> <li>ICSE 2005</li> <li>J. H. Andrews, L. C. Briand, Y. Labiche</li> <li><a href="https://dl.acm.org/citation.cfm?doid=1062455.1062530">https://dl.acm.org/citation.cfm?doid=1062455.1062530</a></li> <li><a href="https://www.researchgate.net/publication/221554380_Is_Mutation_an_Appropriate_Tool_for_Testing_Experiments">https://www.researchgate.net/publication/221554380_Is_Mutation_an_Appropriate_Tool_for_Testing_Experiments</a></li> </ul> </li> <li>Mutant Quality Indicators <ul> <li>International Workshop on Mutation Analysis (MUTATION) 2018</li> <li>Mike Papadakis, Titcheu Chekam Thierry, Yves Le Traon</li> <li><a href="http://orbilu.uni.lu/bitstream/10993/34352/1/bare_conf1.pdf">http://orbilu.uni.lu/bitstream/10993/34352/1/bare_conf1.pdf</a></li> </ul> </li> <li>Mutation Testing Advances: An Analysis and Survey <ul> <li>Advances in Computers, Volume 112, 2018</li> <li>Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, Mark Harman</li> <li><a href="https://sites.google.com/site/mikepapadakis/survey.pdf?attredirects=0&amp;d=1">https://sites.google.com/site/mikepapadakis/survey.pdf?attredirects=0&amp;d=1</a></li> <li><a href="https://mutationtesting.uni.lu/survey.pdf">https://mutationtesting.uni.lu/survey.pdf</a></li> </ul> </li> <li>Mutation Testing: From Theory to Practice <ul> <li>2019 PhD Dissertation; Ali Parsai</li> <li><a href="https://repository.uantwerpen.be/docman/irua/479e80/161581.pdf">https://repository.uantwerpen.be/docman/irua/479e80/161581.pdf</a></li> <li><a href="https://www.parsai.net/files/research/PhDThesisAliParsai2019.pdf">https://www.parsai.net/files/research/PhDThesisAliParsai2019.pdf</a></li> </ul> </li> <li>On the Limits of Mutation Analysis <ul> <li>2017 PhD Dissertation; Rahul Gopinath</li> <li><a href="https://rahul.gopinath.org/resources/phd2017/gopinath2017on.pdf">https://rahul.gopinath.org/resources/phd2017/gopinath2017on.pdf</a></li> </ul> </li> <li>State of Mutation Testing at Google <ul> <li>International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) 2018</li> <li>Goran Petrović and Marko Ivanković</li> <li><a href="https://ai.google/research/pubs/pub46584">https://ai.google/research/pubs/pub46584</a></li> <li><a href="https://dl.acm.org/citation.cfm?id=3183521">https://dl.acm.org/citation.cfm?id=3183521</a></li> </ul> </li> <li>The Care and Feeding of Wild-Caught Mutants <ul> <li>Foundations of Software Engineering (FSE) 2017</li> <li>David Bingham Brown, Michael Vaughn, Ben Liblit, Thomas W. Reps</li> <li><a href="http://research.cs.wisc.edu/wpis/abstracts/fse17.abs.html">http://research.cs.wisc.edu/wpis/abstracts/fse17.abs.html</a></li> </ul> </li> </ul> <h3>Coverage: Mutation: Software</h3> <ul> <li>MART Framework for Multi-Programming Language Mutation Testing based on LLVM <ul> <li>LLVM Mutation Artisan (MART) -- a configurable mutation testing framework based on LLVM</li> <li><a href="https://github.com/thierry-tct/mart">https://github.com/thierry-tct/mart</a></li> <li>Mart: A Mutant Generation Tool for LLVM <ul> <li>ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) 2019</li> <li>Thierry Titcheu Chekam, Mike Papadakis, Yves Le Traon</li> <li><a href="https://orbilu.uni.lu/retrieve/63878/72722/main.pdf">https://orbilu.uni.lu/retrieve/63878/72722/main.pdf</a></li> </ul> </li> </ul> </li> <li>Mull: Mutation testing system built on top of LLVM <ul> <li><a href="https://github.com/mull-project/mull">https://github.com/mull-project/mull</a></li> <li><a href="https://lowlevelbits.org/llvm-based-mutation-testing-system.-request-for-comments/">https://lowlevelbits.org/llvm-based-mutation-testing-system.-request-for-comments/</a></li> <li>Mutation Testing Leaving the Stone Age - Alex Denisov, FOSDEM 2017 <ul> <li><a href="https://www.youtube.com/watch?v=pVMupHK1BVY">https://www.youtube.com/watch?v=pVMupHK1BVY</a></li> </ul> </li> <li>Mull it over: mutation testing based on LLVM <ul> <li>Mutation 2018; Alex Denisov, Stanislav Pankevich</li> <li><a href="https://lowlevelbits.org/pdfs/Mull_Mutation_2018.pdf">https://lowlevelbits.org/pdfs/Mull_Mutation_2018.pdf</a></li> </ul> </li> </ul> </li> <li>Mutate++ - A C++ Mutation Test Environment <ul> <li><a href="https://github.com/nlohmann/mutate_cpp">https://github.com/nlohmann/mutate_cpp</a></li> </ul> </li> <li>SRCIROR: SRC and IR mutatOR <ul> <li>performs mutations on source code written in C/C++ and on the LLVM IR</li> <li><a href="https://github.com/TestingResearchIllinois/srciror">https://github.com/TestingResearchIllinois/srciror</a></li> <li>SRCIROR: A Toolset for Mutation Testing of C Source Code and LLVM Intermediate Representation <ul> <li>Automated Software Engineering (ASE) 2018; Farah Hariri and August Shi</li> <li><a href="http://mir.cs.illinois.edu/farah/publications/ase18_srciror.pdf">http://mir.cs.illinois.edu/farah/publications/ase18_srciror.pdf</a></li> </ul> </li> </ul> </li> </ul> <h3>Coverage: Mutation: Talks</h3> <ul> <li>Testing The Tests: Mutation Testing for C++ <ul> <li>NDC TechTown 2019; Seph De Busser</li> <li><a href="https://www.youtube.com/watch?v=M-5_M8qZXaE">https://www.youtube.com/watch?v=M-5_M8qZXaE</a></li> </ul> </li> </ul> <hr> <h1>Generation</h1> <ul> <li>An Orchestrated Survey on Automated Software Test Case Generation <ul> <li>Journal of Systems and Software 86 (2013)</li> <li>Saswat Anand, Edmund K. Burke, Tsong Yueh Chen, John Clark, Myra B. Cohen, Wolfgang Grieskamp, Mark Harman, Mary Jean Harrold, Phil Mcminn</li> <li><a href="https://dl.acm.org/citation.cfm?id=2503991">https://dl.acm.org/citation.cfm?id=2503991</a></li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.678.1106">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.678.1106</a></li> <li><a href="http://romisatriawahono.net/lecture/rm/survey/software%20engineering/Software%20Testing/Anand%20-%20Automated%20Software%20Test%20Case%20generation%20-%202013.pdf">http://romisatriawahono.net/lecture/rm/survey/software%20engineering/Software%20Testing/Anand%20-%20Automated%20Software%20Test%20Case%20generation%20-%202013.pdf</a></li> </ul> </li> <li>Automated Software Test Generation: Some Challenges, Solutions, and Recent Advances <ul> <li>Special issue Nr. 10,000 of Lecture Notes in Computer Science, Springer, 2018</li> <li>George Candea and Patrice Godefroid</li> <li><a href="http://dslab.epfl.ch/pubs/automated-test-generation.pdf">http://dslab.epfl.ch/pubs/automated-test-generation.pdf</a></li> <li><a href="https://patricegodefroid.github.io/public_psfiles/lncs10000-2018.pdf">https://patricegodefroid.github.io/public_psfiles/lncs10000-2018.pdf</a></li> </ul> </li> <li>FSX: fine-grained incremental unit test generation for C/C++ programs <ul> <li>ISSTA 2016</li> <li>Hiroaki Yoshida, Susumu Tokumoto, Mukul R. Prasad, Indradeep Ghosh, Tadahiro Uehara</li> <li><a href="https://dl.acm.org/citation.cfm?id=2931055">https://dl.acm.org/citation.cfm?id=2931055</a></li> </ul> </li> <li>KLEE: Unassisted and Automatic Generation of High-Coverage Tests for Complex Systems Programs <ul> <li>OSDI 2008</li> <li>C. Cadar, D. Dunbar, D. Engler</li> <li><a href="http://www.doc.ic.ac.uk/~cristic/papers/klee-osdi-08.pdf">http://www.doc.ic.ac.uk/~cristic/papers/klee-osdi-08.pdf</a></li> <li><a href="https://klee.github.io/publications/">https://klee.github.io/publications/</a></li> </ul> </li> <li>KLOVER (KLee based cOVERage tool, for C/C++) <ul> <li>KLOVER: Automatic Test Generation for C and C++ Programs, Using Symbolic Execution <ul> <li>IEEE Software Volume 34, Issue 5, 2017</li> <li>H. Yoshida, G. Li, T. Kamiya, I. Ghosh, S. Rajan, S. Tokumoto, K. Munakata, T. Uehara</li> <li><a href="http://ieeexplore.ieee.org/document/8048666/">http://ieeexplore.ieee.org/document/8048666/</a></li> </ul> </li> <li>KLOVER: A Symbolic Execution and Automatic Test Generation Tool for C++ Programs <ul> <li>CAV 2011</li> <li>Guodong Li, Indradeep Ghosh, and Sreeranga P. Rajan</li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-642-22110-1_49">https://link.springer.com/chapter/10.1007/978-3-642-22110-1_49</a></li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.205.5180">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.205.5180</a></li> <li><a href="http://www.cs.utah.edu/~ligd/publications/KLOVER-CAV11.pdf">http://www.cs.utah.edu/~ligd/publications/KLOVER-CAV11.pdf</a></li> </ul> </li> </ul> </li> <li>Symbolic Execution Algorithms for Test Generation <ul> <li>2009 Dissertation; Ru-Gang Xu</li> <li><a href="https://people.mpi-sws.org/~rupak/Papers/RuGangXuThesis.pdf">https://people.mpi-sws.org/~rupak/Papers/RuGangXuThesis.pdf</a></li> </ul> </li> <li>Symbolic Execution for Software Testing: Three Decades Later <ul> <li>CACM 2013</li> <li>C. Cadar and K. Sen</li> <li><a href="https://people.eecs.berkeley.edu/~ksen/papers/cacm13.pdf">https://people.eecs.berkeley.edu/~ksen/papers/cacm13.pdf</a></li> </ul> </li> <li>Test-Comp: International Competition on Software Testing <ul> <li>a comparative evaluation of automatic tools for software test generation</li> <li><a href="https://test-comp.sosy-lab.org/">https://test-comp.sosy-lab.org/</a></li> </ul> </li> <li>Test Generation Using Symbolic Execution <ul> <li>Foundations of Software Technology and Theoretical Computer Science (FSTTCS) 2012</li> <li>Patrice Godefroid</li> <li><a href="http://drops.dagstuhl.de/opus/volltexte/2012/3845/">http://drops.dagstuhl.de/opus/volltexte/2012/3845/</a></li> </ul> </li> </ul> <h2>Generation: Random</h2> <p>Random Testing</p> <ul> <li>A Survey on Adaptive Random Testing <ul> <li>IEEE Transactions on Software Engineering (TSE) 14(8) 2015</li> <li>Rubing Huang, Weifeng Sun, Yinyin Xu, Haibo Chen, Dave Towey, Xin Xia</li> <li><a href="https://xin-xia.github.io/publication/tse198.pdf">https://xin-xia.github.io/publication/tse198.pdf</a></li> </ul> </li> <li>Adaptive Random Testing <ul> <li>Asian Computing Science Conference (ASIAN) 2004</li> <li>T.Y. Chen, H. Leung, I.K. Mak</li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.2054&amp;rep=rep1&amp;type=pdf">http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.397.2054&amp;rep=rep1&amp;type=pdf</a></li> </ul> </li> <li>Adaptive Random Testing: the ART of Test Case Diversity <ul> <li>Journal of Systems and Software 83 (1) 2010</li> <li>Chen, T.Y., Kuo, F.-C., Merkel, R.G., Tse, T.H.</li> <li><a href="http://i.cs.hku.hk/~tse/Papers/2010s/rbartJSS.html">http://i.cs.hku.hk/~tse/Papers/2010s/rbartJSS.html</a></li> </ul> </li> <li>John Regehr <ul> <li>The Central Limit Theorem Makes Random Testing Hard <ul> <li><a href="https://blog.regehr.org/archives/660">https://blog.regehr.org/archives/660</a></li> </ul> </li> <li>Randomly Testing a Static Analyzer <ul> <li><a href="https://blog.regehr.org/archives/680">https://blog.regehr.org/archives/680</a></li> </ul> </li> <li>Burning in a Module with Random Unit Testing <ul> <li><a href="https://blog.regehr.org/archives/737">https://blog.regehr.org/archives/737</a></li> </ul> </li> <li>Oracles for Random Testing <ul> <li><a href="https://blog.regehr.org/archives/856">https://blog.regehr.org/archives/856</a></li> </ul> </li> </ul> </li> <li>Random Test Run Length and Effectiveness <ul> <li>Automated Software Engineering (ASE) 2008</li> <li>J. H. Andrews, A. Groce, M. Weston, and Ru-Gang Xu</li> <li><a href="https://agroce.github.io/ase08.pdf">https://agroce.github.io/ase08.pdf</a></li> </ul> </li> <li>The Confidence Sequence Method: a computer-age test for statistical SLOs <ul> <li><a href="https://pvk.ca/Blog/2018/07/06/testing-slo-type-properties-with-the-confidence-sequence-method/">https://pvk.ca/Blog/2018/07/06/testing-slo-type-properties-with-the-confidence-sequence-method/</a></li> <li>Lazy Linear Knapsack <ul> <li><a href="https://pvk.ca/Blog/2020/01/20/lazy-linear-knapsack/">https://pvk.ca/Blog/2020/01/20/lazy-linear-knapsack/</a></li> <li>"TL;DR: the estimation algorithm for individual sampling passes works, and the combination of Hypothesis and Confidence Sequence Method lets us painlessly test for a statistical property."</li> </ul> </li> </ul> </li> </ul> <hr> <h1>Property-Based Testing</h1> <ul> <li>What is Property Based Testing? <ul> <li><a href="https://hypothesis.works/articles/what-is-property-based-testing/">https://hypothesis.works/articles/what-is-property-based-testing/</a></li> </ul> </li> <li>Finding Property Tests <ul> <li><a href="https://www.hillelwayne.com/post/contract-examples/">https://www.hillelwayne.com/post/contract-examples/</a></li> </ul> </li> <li>Generating Good Generators for Inductive Relations <ul> <li>POPL 2018</li> <li>Leonidas Lampropoulos, Zoe Paraskevopoulou, Benjamin C. Pierce</li> <li><a href="https://www.cis.upenn.edu/~llamp/pdf/GeneratingGoodGenerators.pdf">https://www.cis.upenn.edu/~llamp/pdf/GeneratingGoodGenerators.pdf</a></li> <li><a href="https://popl18.sigplan.org/event/popl-2018-papers-generating-good-generators-for-inductive-relations">https://popl18.sigplan.org/event/popl-2018-papers-generating-good-generators-for-inductive-relations</a></li> <li><a href="http://www.cs.cmu.edu/~popl-interviews/paraskevopoulou.html">http://www.cs.cmu.edu/~popl-interviews/paraskevopoulou.html</a></li> </ul> </li> <li>How hard is it to guide test case generators with branch coverage feedback? <ul> <li><a href="https://engineering.backtrace.io/posts/2020-03-11-how-hard-is-it-to-guide-test-case-generators-with-branch-coverage-feedback/">https://engineering.backtrace.io/posts/2020-03-11-how-hard-is-it-to-guide-test-case-generators-with-branch-coverage-feedback/</a></li> </ul> </li> <li>How to Specify it! A Guide to Writing Properties of Pure Functions <ul> <li>Trends in Functional Programming (TFP) 2019</li> <li>John Hughes</li> <li><a href="https://www.tfp2019.org/resources/tfp2019-how-to-specify-it.pdf">https://www.tfp2019.org/resources/tfp2019-how-to-specify-it.pdf</a></li> <li>Lambda Days 2020 <ul> <li><a href="https://www.youtube.com/watch?v=G0NUOst-53U">https://www.youtube.com/watch?v=G0NUOst-53U</a></li> <li><a href="https://www.infoq.com/news/2020/02/property-based-testing-guide/">https://www.infoq.com/news/2020/02/property-based-testing-guide/</a></li> </ul> </li> </ul> </li> </ul> <h2>Property-Based Testing: Metamorphic</h2> <p>Metamorphic Relations, Metamorphic Testing</p> <ul> <li>Metamorphic Testing <ul> <li><a href="https://www.hillelwayne.com/post/metamorphic-testing/">https://www.hillelwayne.com/post/metamorphic-testing/</a></li> </ul> </li> <li><a href="https://ieeexplore.ieee.org/xpl/conhome/1817204/all-proceedings">IEEE/ACM International Workshop on Metamorphic Testing (MET)</a> <ul> <li><a href="http://icse-conferences.org/">ICSE (International Conference on Software Engineering)</a>: <a href="https://dblp.org/db/conf/icse/">https://dblp.org/db/conf/icse/</a></li> </ul> </li> <li>A Survey on Metamorphic Testing <ul> <li>IEEE Transactions on Software Engineering 42(9) 2016</li> <li><a href="http://eprints.whiterose.ac.uk/110335/1/segura16-tse.pdf">http://eprints.whiterose.ac.uk/110335/1/segura16-tse.pdf</a></li> <li><a href="http://www.cs.ecu.edu/reu/reufiles/read/metamorphicTesting-16.pdf">http://www.cs.ecu.edu/reu/reufiles/read/metamorphicTesting-16.pdf</a></li> <li><a href="https://pdfs.semanticscholar.org/bc06/14b761bd5f5bad9e157515d7b428ebe63170.pdf">https://pdfs.semanticscholar.org/bc06/14b761bd5f5bad9e157515d7b428ebe63170.pdf</a></li> </ul> </li> <li>Automated Metamorphic Testing <ul> <li>COMPSAC 2003: International Conference on Computer Software and Applications</li> <li>Arnaud Gotlieb, Bernard Botella</li> <li><a href="http://people.rennes.inria.fr/Arnaud.Gotlieb/publications/GB03.pdf">http://people.rennes.inria.fr/Arnaud.Gotlieb/publications/GB03.pdf</a></li> </ul> </li> <li>Metamorphic Testing <ul> <li>CREST Open Workshop 2012</li> <li>Tsong Yueh Chen</li> <li><a href="http://crest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf">http://crest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf</a></li> <li><a href="http://crest.cs.ucl.ac.uk/cow/20/videos/COW20_Chen.mp4">http://crest.cs.ucl.ac.uk/cow/20/videos/COW20_Chen.mp4</a></li> <li>How to test a non-testable program? <ul> <li>STEM Blitz October 2014</li> <li><a href="https://www.youtube.com/watch?v=s_geSjqYEyo">https://www.youtube.com/watch?v=s_geSjqYEyo</a></li> </ul> </li> </ul> </li> <li>Metamorphic Testing 20 Years Later: A Hands-on Introduction <ul> <li>ICSE 2018</li> <li>Sergio Segura, Zhi Quan (George) Zhou</li> <li><a href="https://dl.acm.org/citation.cfm?id=3183468">https://dl.acm.org/citation.cfm?id=3183468</a></li> <li><a href="https://www.researchgate.net/publication/323025715_Metamorphic_Testing_20_Years_Later_A_Hands-on_Introduction">https://www.researchgate.net/publication/323025715_Metamorphic_Testing_20_Years_Later_A_Hands-on_Introduction</a></li> <li><a href="https://zenodo.org/record/1256230/files/ICSE18-TB-Segura-Zhou.pdf">https://zenodo.org/record/1256230/files/ICSE18-TB-Segura-Zhou.pdf</a></li> </ul> </li> <li>Metamorphic Testing and Its Applications <ul> <li>International Symposium on Future Software Technology (ISFST) 2004</li> <li>Zhou, Zhi Quan, D. H. Huang, T. H. Tse, Zongyuan Yang, Haitao Huang, T. Y. Chen</li> <li><a href="https://www.cs.hku.hk/data/techreps/document/TR-2004-12.pdf">https://www.cs.hku.hk/data/techreps/document/TR-2004-12.pdf</a></li> </ul> </li> <li>Metamorphic Testing: A New Approach for Generating Next Test Cases <ul> <li>Technical Report HKUST-CS98-01 (Hong Kong University of Science and Technology) 1998</li> <li>T. Y. Chen, S. C. Cheung, S. M. Yiu</li> <li><a href="https://arxiv.org/abs/2002.12543">https://arxiv.org/abs/2002.12543</a></li> <li><a href="https://www.cse.ust.hk/~scc/publ/CS98-01-metamorphictesting.pdf">https://www.cse.ust.hk/~scc/publ/CS98-01-metamorphictesting.pdf</a></li> </ul> </li> <li>Metamorphic Testing: A Review of Challenges and Opportunities <ul> <li>ACM Computing Surveys (CSUR) Volume 51, Issue 1, 2018</li> <li>Tsong Yueh Chen, Fei-Ching Kuo, Huai Liu, Pak-Lok Poon, Dave Towey, T. H. Tse, and Zhi Quan Zhou</li> <li><a href="https://dl.acm.org/citation.cfm?id=3143561">https://dl.acm.org/citation.cfm?id=3143561</a></li> <li><a href="http://www.cs.hku.hk/research/techreps/document/TR-2017-04.pdf">http://www.cs.hku.hk/research/techreps/document/TR-2017-04.pdf</a></li> </ul> </li> <li>Metamorphic Testing: A Simple Yet Effective Approach for Testing Scientific Software <ul> <li>Computing in Science &amp; Engineering 21(1) 2019</li> <li>U. Kanewala and T. Yueh Chen</li> <li><a href="https://doi.ieeecomputersociety.org/10.1109/MCSE.2018.2875368">https://doi.ieeecomputersociety.org/10.1109/MCSE.2018.2875368</a></li> </ul> </li> <li>Techniques for Testing Scientific Programs Without an Oracle <ul> <li>International Workshop on Software Engineering for Computational Science and Engineering (SE-CSE) 2013</li> <li>U. Kanewala and J. Bieman</li> <li><a href="https://www.cs.colostate.edu/~bieman/Pubs/kanewalaBieman_icsews13secse_preprint.pdf">https://www.cs.colostate.edu/~bieman/Pubs/kanewalaBieman_icsews13secse_preprint.pdf</a></li> </ul> </li> </ul> <hr> <h1>Reduction</h1> <p>Test-Case Reduction, Minimization</p> <p>See also: <a href="https://github.com/MattPD/cpplinks/blob/master/debugging.md">Debugging</a>: <a href="https://github.com/MattPD/cpplinks/blob/master/debugging.md#readings">Readings</a>: Delta Debugging</p> <h2>Reduction: Readings</h2> <ul> <li>Everything You Ever Wanted To Know About Test-Case Reduction, But Didn’t Know to Ask <ul> <li><a href="https://blog.trailofbits.com/2019/11/11/test-case-reduction/">https://blog.trailofbits.com/2019/11/11/test-case-reduction/</a></li> </ul> </li> <li>Mitigating (and Exploiting) Test Reduction Slippage <ul> <li>A-TEST 2016: International Workshop on Automating Test Case Design, Selection, and Evaluation</li> <li>Josie Holmes, Alex Groce, Mohammad Amin Alipour</li> <li><a href="https://agroce.github.io/atest16.pdf">https://agroce.github.io/atest16.pdf</a></li> </ul> </li> <li>Notes on Test-Case Reduction <ul> <li><a href="https://www.drmaciver.com/2019/01/notes-on-test-case-reduction/">https://www.drmaciver.com/2019/01/notes-on-test-case-reduction/</a></li> </ul> </li> <li>Reduce Before You Localize: Delta-Debugging and Spectrum-Based Fault Localization <ul> <li>2018 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)</li> <li>Arpit Christi, Matthew Lyle Olson, Mohammad Amin Alipour, Alex Groce</li> <li><a href="https://doi.org/10.1109/ISSREW.2018.00005">https://doi.org/10.1109/ISSREW.2018.00005</a></li> <li><a href="https://agroce.github.io/idear18.pdf">https://agroce.github.io/idear18.pdf</a></li> </ul> </li> <li>Rust Bug Minimization Patterns <ul> <li><a href="http://blog.pnkfx.org/blog/2019/11/18/rust-bug-minimization-patterns/">http://blog.pnkfx.org/blog/2019/11/18/rust-bug-minimization-patterns/</a></li> </ul> </li> </ul> <h2>Reduction: Software</h2> <ul> <li>C-Reduce <ul> <li>"C-Reduce is a tool that takes a large C, C++, or OpenCL file that has a property of interest (such as triggering a compiler bug) and automatically produces a much smaller C/C++ file that has the same property. It is intended for use by people who discover and report bugs in compilers and other tools that process source code."</li> <li><a href="https://embed.cs.utah.edu/creduce/">https://embed.cs.utah.edu/creduce/</a></li> </ul> </li> <li>Delta Debugging <ul> <li><a href="https://www.st.cs.uni-saarland.de/dd/">https://www.st.cs.uni-saarland.de/dd/</a></li> <li>Delta: Heuristically minimizes interesting files <ul> <li><a href="http://delta.stage.tigris.org/">http://delta.stage.tigris.org/</a></li> <li>Minimizing Interesting Files with Delta <ul> <li><a href="http://delta.stage.tigris.org/using_delta.html">http://delta.stage.tigris.org/using_delta.html</a></li> </ul> </li> </ul> </li> </ul> </li> <li>halfempty: Fast, Parallel Testcase Minimization <ul> <li><a href="https://github.com/googleprojectzero/halfempty">https://github.com/googleprojectzero/halfempty</a></li> </ul> </li> <li>Shrink Ray: a test-case reducer designed to be effective on a wide range of formats <ul> <li><a href="https://github.com/DRMacIver/shrinkray">https://github.com/DRMacIver/shrinkray</a></li> </ul> </li> <li>Structured Shrinking: Structured shrinking of unknown file formats <ul> <li>a program and library that attempts to find structure in a file and uses it to produce smaller examples of the same sort of file.</li> <li><a href="https://github.com/DRMacIver/structureshrink">https://github.com/DRMacIver/structureshrink</a></li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>Approval Tests for C++ <ul> <li><a href="https://github.com/approvals/ApprovalTests.cpp">https://github.com/approvals/ApprovalTests.cpp</a></li> <li><a href="https://claremacrae.co.uk/conferences/presentations.html">https://claremacrae.co.uk/conferences/presentations.html</a></li> </ul> </li> <li>Boost.Test <ul> <li><a href="http://www.boost.org/libs/test/">http://www.boost.org/libs/test/</a></li> <li>Production use of Boost.Test <ul> <li>BoostCon 2010; Gennadiy Rozental</li> <li><a href="https://www.youtube.com/watch?v=WUQkn4CnzF4">https://www.youtube.com/watch?v=WUQkn4CnzF4</a></li> </ul> </li> </ul> </li> <li>[Boost].UT: C++20 μ(micro)/Unit Testing Framework <ul> <li><a href="https://github.com/boost-experimental/ut">https://github.com/boost-experimental/ut</a></li> </ul> </li> <li>Catch2 <ul> <li>A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)</li> <li><a href="http://catch-lib.net">http://catch-lib.net</a></li> <li><a href="https://github.com/catchorg/Catch2">https://github.com/catchorg/Catch2</a></li> <li>Modern C++ Testing with Catch2 - Phil Nash <ul> <li>Meeting C++ 2017: <a href="https://www.youtube.com/watch?v=3tIE6X5FjDE">https://www.youtube.com/watch?v=3tIE6X5FjDE</a></li> <li>CppCon 2018: <a href="https://www.youtube.com/watch?v=Ob5_XZrFQH0">https://www.youtube.com/watch?v=Ob5_XZrFQH0</a></li> </ul> </li> </ul> </li> <li>Cgreen - The Modern Unit Test and Mocking Framework for C and C++ <ul> <li><a href="https://github.com/cgreen-devs/cgreen">https://github.com/cgreen-devs/cgreen</a></li> </ul> </li> <li>CppUTest unit testing and mocking framework for C/C++ <ul> <li><a href="http://cpputest.github.com">http://cpputest.github.com</a></li> <li><a href="https://github.com/cpputest/cpputest">https://github.com/cpputest/cpputest</a></li> </ul> </li> <li>CUTE: C++ Unit Testing Easier <ul> <li><a href="https://cute-test.com/">https://cute-test.com/</a></li> <li><a href="https://github.com/PeterSommerlad/CUTE">https://github.com/PeterSommerlad/CUTE</a></li> </ul> </li> <li>cwrap - A toolset for client server testing <ul> <li><a href="https://cwrap.org/">https://cwrap.org/</a></li> <li>cwrap's mission is to enable developers to test complex network-based and privileged software stacks on UNIX machines with limited network access and without root privileges by providing preloadable libraries to wrap standard libc functions.</li> <li>Testing your full software stack with cwrap <ul> <li><a href="https://lwn.net/Articles/594863/">https://lwn.net/Articles/594863/</a></li> </ul> </li> <li>Testing your software stack without root privileges using cwrap <ul> <li><a href="https://developers.redhat.com/blog/2015/05/05/testing-your-software-stack-without-root-privileges-using-cwrap/">https://developers.redhat.com/blog/2015/05/05/testing-your-software-stack-without-root-privileges-using-cwrap/</a></li> </ul> </li> </ul> </li> <li>DeepState: A unit test-like interface for fuzzing and symbolic execution <ul> <li><a href="https://github.com/trailofbits/deepstate">https://github.com/trailofbits/deepstate</a></li> <li>DeepState: Symbolic Unit Testing for C and C++ <ul> <li>NDSS 18 Workshop on Binary Analysis Research</li> <li>Peter Goodman and Alex Groce</li> <li><a href="https://www.cefns.nau.edu/~adg326/bar18.pdf">https://www.cefns.nau.edu/~adg326/bar18.pdf</a></li> </ul> </li> <li>Fuzzing an API with DeepState <ul> <li>Part 1: <a href="https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/">https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/</a></li> <li>Part 2: <a href="https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/">https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/</a></li> </ul> </li> </ul> </li> <li>doctest <ul> <li>The fastest feature-rich C++11/14/17/20 single-header testing framework for unit tests and TDD</li> <li><a href="https://github.com/onqtam/doctest">https://github.com/onqtam/doctest</a></li> </ul> </li> <li>Google Test <ul> <li><a href="https://github.com/google/googletest">https://github.com/google/googletest</a></li> </ul> </li> <li>GUnit - Google.Test/Google.Mock/Cucumber on steroids <ul> <li>extends/simplifies Google.Test/Google.Mock and adds support for Gherkin (Behaviour Driven Development) to it</li> <li><a href="https://github.com/cpp-testing/GUnit">https://github.com/cpp-testing/GUnit</a></li> </ul> </li> <li>OutputCheck: A tool for checking tool output inspired by LLVM's FileCheck <ul> <li><a href="https://github.com/stp/OutputCheck">https://github.com/stp/OutputCheck</a></li> </ul> </li> <li>RapidCheck: QuickCheck clone for C++ with the goal of being simple to use with as little boilerplate as possible. <ul> <li><a href="https://github.com/emil-e/rapidcheck">https://github.com/emil-e/rapidcheck</a></li> <li>Generating test cases so you don’t have to <ul> <li><a href="https://labs.spotify.com/2015/06/25/rapid-check/">https://labs.spotify.com/2015/06/25/rapid-check/</a></li> </ul> </li> <li>Property based testing in C++ - How to write 1000s of tests in one sitting? <ul> <li>code::dive 2016; Patryk Małek</li> <li><a href="https://codedive.pl/index/speaker/name/patryk-malek/year/2016/">https://codedive.pl/index/speaker/name/patryk-malek/year/2016/</a></li> <li><a href="https://www.youtube.com/watch?v=aiapg-3vDcQ">https://www.youtube.com/watch?v=aiapg-3vDcQ</a></li> <li><a href="https://cdn2-ecros.pl/event/codedive/files/presentations/2016/Patryk_Malek_Property_based_testing_in_cpp.pdf">https://cdn2-ecros.pl/event/codedive/files/presentations/2016/Patryk_Malek_Property_based_testing_in_cpp.pdf</a></li> </ul> </li> </ul> </li> </ul> <h2>Software: Test Doubles</h2> <p>Test Doubles: Faking, Mocking</p> <ul> <li>Fake Function Framework (fff) <ul> <li>A testing micro framework for creating function test doubles.</li> <li><a href="https://github.com/meekrosoft/fff">https://github.com/meekrosoft/fff</a></li> </ul> </li> <li>gen-xfakes <ul> <li>Generate exploding fakes from C/C++ linker error output (used for unit testing)</li> <li><a href="https://github.com/jwgrenning/gen-xfakes">https://github.com/jwgrenning/gen-xfakes</a></li> <li>Wrestle Legacy C and C++ into a Test Harness – Linker Errors <ul> <li><a href="http://blog.wingman-sw.com/wrestle-legacy-c-cpp-into-tests-linker-errors">http://blog.wingman-sw.com/wrestle-legacy-c-cpp-into-tests-linker-errors</a></li> </ul> </li> </ul> </li> <li>PowerFake <ul> <li>C++ Faking library, which allows faking regular functions, static member functions and non-virtual member functions for testing purposes.</li> <li><a href="https://gitlab.com/hedayat/powerfake">https://gitlab.com/hedayat/powerfake</a></li> <li><a href="https://hedayatvk.wordpress.com/2018/01/22/introducing-powerfake-for-c/">https://hedayatvk.wordpress.com/2018/01/22/introducing-powerfake-for-c/</a></li> </ul> </li> </ul> <h2>Software: Test Doubles: Mocking</h2> <ul> <li>FakeIt: a simple mocking framework for C++ <ul> <li><a href="https://github.com/eranpeer/FakeIt">https://github.com/eranpeer/FakeIt</a></li> </ul> </li> <li>Google Mock: The Google C++ mocking framework <ul> <li><a href="https://github.com/google/googletest/tree/master/googlemock">https://github.com/google/googletest/tree/master/googlemock</a></li> <li><a href="https://github.com/google/googletest/tree/master/googlemock/docs">https://github.com/google/googletest/tree/master/googlemock/docs</a></li> </ul> </li> <li>GUnit.GMock <ul> <li>GoogleMock without writing and maintaining mocks by hand</li> <li><a href="https://github.com/cpp-testing/GUnit/blob/master/docs/GMock.md">https://github.com/cpp-testing/GUnit/blob/master/docs/GMock.md</a></li> </ul> </li> <li>Hippomocks: Single-header mocking framework <ul> <li><a href="https://github.com/dascandy/hippomocks">https://github.com/dascandy/hippomocks</a></li> </ul> </li> <li>Mockitopp: Simple mocking for C++ <ul> <li><a href="https://github.com/tpounds/mockitopp">https://github.com/tpounds/mockitopp</a></li> </ul> </li> <li>Trompeloeil: Header-only C++14 mocking framework <ul> <li><a href="https://github.com/rollbear/trompeloeil">https://github.com/rollbear/trompeloeil</a></li> </ul> </li> </ul> <hr> <h1>Talks</h1> <ul> <li>Google Test Automation Conference (GTAC) <ul> <li><a href="https://developers.google.com/google-test-automation-conference/">https://developers.google.com/google-test-automation-conference/</a></li> </ul> </li> </ul> <h2>Talks: 2019</h2> <ul> <li>Back to Basics: Test-driven Development <ul> <li>CppCon 2019; Fedor Pikus</li> <li><a href="https://www.youtube.com/watch?v=RoYljVOj2H8">https://www.youtube.com/watch?v=RoYljVOj2H8</a></li> </ul> </li> <li>Making Testing C++ Binaries Practical @ Facebook Scale: A CI Story <ul> <li>CppCon 2019; Mark Isaacson</li> <li><a href="https://www.youtube.com/watch?v=p7xdNjdtojU">https://www.youtube.com/watch?v=p7xdNjdtojU</a></li> </ul> </li> </ul> <h2>Talks: 2018</h2> <ul> <li>But is it fun? Software Testing in the Video Game Industry <ul> <li>ICST 2018</li> <li>Magnus Nordin (Electronic Arts), David King (DICE), and Stefan Posthuma (Electronic Arts)</li> <li><a href="https://www.youtube.com/watch?v=CBIhu_9OolY">https://www.youtube.com/watch?v=CBIhu_9OolY</a></li> </ul> </li> <li>Ensuring Exception Safety Through Testing <ul> <li>CppCon 2018; Jon Cohen</li> <li><a href="https://www.youtube.com/watch?v=XPzHNSUnTc4">https://www.youtube.com/watch?v=XPzHNSUnTc4</a></li> </ul> </li> </ul> <h2>Talks: 2017</h2> <ul> <li>C++ Unit testing - the good, the bad &amp; the ugly <ul> <li>NDC 2017; Dror Helper</li> <li><a href="https://www.youtube.com/watch?v=EiN3cF_y3vM">https://www.youtube.com/watch?v=EiN3cF_y3vM</a></li> <li><a href="https://vimeo.com/223984399">https://vimeo.com/223984399</a></li> <li><a href="https://www.slideshare.net/dhelper/c-unit-testing-the-good-the-bad-the-ugly">https://www.slideshare.net/dhelper/c-unit-testing-the-good-the-bad-the-ugly</a></li> </ul> </li> <li>Microcontrollers in Micro-increments: A Test-driven C++ Workflow for Embedded Systems <ul> <li>CppCon 2017: Mike Ritchie</li> <li><a href="https://www.youtube.com/watch?v=XuHlDtWYeD8">https://www.youtube.com/watch?v=XuHlDtWYeD8</a></li> </ul> </li> <li>Mocking C++ <ul> <li>C++Now 2017; Peter Bindels</li> <li><a href="https://www.youtube.com/watch?v=t0wLm2iiEH0">https://www.youtube.com/watch?v=t0wLm2iiEH0</a></li> </ul> </li> <li>Mocking Frameworks considered harmful <ul> <li>CppCon 2017; Peter Sommerlad</li> <li><a href="https://www.youtube.com/watch?v=uhuHZXTRfH4">https://www.youtube.com/watch?v=uhuHZXTRfH4</a></li> </ul> </li> </ul> <h2>Talks: 2016</h2> <ul> <li>Assessing software quality from first principles <ul> <li>UCIBrenICS Seminar 2016</li> <li>Reid Holmes, University of British Columbia</li> <li>code coverage and mutation testing</li> <li><a href="https://www.youtube.com/watch?v=RIVLHQjpTck">https://www.youtube.com/watch?v=RIVLHQjpTck</a></li> </ul> </li> </ul> <h2>Talks: 2015</h2> <ul> <li>Advanced Unit Testing in C &amp; C++ <ul> <li>CppCon 2015; Matt Hargett</li> <li><a href="https://www.youtube.com/watch?v=Wmy6g-aVgZI">https://www.youtube.com/watch?v=Wmy6g-aVgZI</a></li> </ul> </li> <li>All Your Tests are Terrible: Tales from the Trenches <ul> <li>CppCon 2015; T. Winters &amp; H. Wright</li> <li><a href="https://www.youtube.com/watch?v=u5senBJUkPc">https://www.youtube.com/watch?v=u5senBJUkPc</a></li> </ul> </li> <li>Fast Iteration Tools in the Production of the Talos Principle <ul> <li>GDC Europe 2015; Alen Ladavac</li> <li><a href="http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in-the">http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in-the</a></li> </ul> </li> </ul> <h2>Talks: 2014</h2> <ul> <li>Pragmatic Unit Testing in C++ <ul> <li>CppCon 2014; Matt Hargett</li> <li><a href="https://www.youtube.com/watch?v=Y8YVSohnlgY">https://www.youtube.com/watch?v=Y8YVSohnlgY</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/static-verification/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/static-verification/index.html <ul> <li> <p>Formal Verification to Ensuring the Memory Safety of C++ Programs</p> <ul> <li>2020 M.Sc. Thesis; Felipe R. Monteiro</li> <li><a href="https://feliperodri.github.io/papers/msc-manuscript.pdf">https://feliperodri.github.io/papers/msc-manuscript.pdf</a></li> <li><a href="https://feliperodri.github.io/talks/msc-presentation.pdf">https://feliperodri.github.io/talks/msc-presentation.pdf</a></li> <li>Apply model checking techniques to ensuring memory safety of C++ programs: <ul> <li>(i) Provide a logical formalization of essential features that the C++ programming language offers, such as templates, sequential and associative containers, inheritance, polymorphism, and exception handling.</li> <li>(ii) Provide a set of abstractions to the Standard C++ Libraries that reflects their semantics, in order to enable the verification of functional properties related to the use of these libraries.</li> <li>(iii) Extend an existing verifier to handle the verification of C++ programs based on (i) and (ii) and evaluate its efficiency and effectiveness in comparison to similar state-of-the-art approaches.</li> </ul> </li> </ul> </li> <li> <p>Model Checking a C++ Software Framework, a Case Study</p> <ul> <li>European Software Engineering Conference / Foundations of Software Engineering (ESEC/FSE) 2019</li> <li>John Lång, I.S.W.B. Prasetya</li> <li><a href="https://arxiv.org/abs/1907.00172">https://arxiv.org/abs/1907.00172</a></li> </ul> </li> </ul> <h2>Verification: Software</h2> <ul> <li> <p>CMBC: C Bounded Model Checker</p> <ul> <li>CBMC is a Bounded Model Checker for C and C++ programs.</li> <li><a href="http://www.cprover.org/cbmc/">http://www.cprover.org/cbmc/</a></li> <li><a href="https://github.com/diffblue/cbmc">https://github.com/diffblue/cbmc</a></li> </ul> </li> <li> <p>DIVINE</p> <ul> <li>DIVINE is a modern, explicit-state model checker. Based on the LLVM toolchain, it can verify programs written in multiple real-world programming languages, including C and C++.</li> <li><a href="https://divine.fi.muni.cz/">https://divine.fi.muni.cz/</a></li> </ul> </li> <li> <p>ESBMC</p> <ul> <li>ESBMC is an open source, permissively licensed, context-bounded model checker based on satisfiability modulo theories for the verification of single- and multi-threaded C/C++ programs.</li> <li><a href="http://www.esbmc.org/">http://www.esbmc.org/</a></li> </ul> </li> <li> <p>Symbiotic</p> <ul> <li>Tool for verifying computer programs based on instrumentation, program slicing and symbolic executor KLEE.</li> <li><a href="http://staticafi.github.io/symbiotic/">http://staticafi.github.io/symbiotic/</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/tracing/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/tracing/index.html <p><a href="https://jvns.ca/blog/2014/04/20/debug-your-programs-like-theyre-closed-source/">Debug your programs like they're closed source! (strace, ltrace)</a></p> <p><a href="https://queue.acm.org/detail.cfm?id=1117401">Hidden in Plain Sight</a>: ACM Queue 4(1) (2006); Bryan Cantrill</p> <p><a href="https://publications.polymtl.ca/2980/">Multi-scale navigation of large trace data: A survey</a><br> - Concurrency and Computation: Practice and Experience, 29(10) 2017<br> - Ezzati-Jivan, N. &amp; Dagenais, M. R</p> <p><a href="https://dl.acm.org/citation.cfm?id=3158644">Survey and Analysis of Kernel and Userspace Tracers on Linux: Design, Implementation, and Overhead</a><br> - ACM Computing Surveys (CSUR) Volume 51 Issue 2, March 2018<br> - Mohamad Gebai, Michel R. Dagenais</p> <p><a href="https://github.com/VANDAL/SynchroTrace-gem5">SynchroTrace</a>:<br> - <a href="http://vlsi.ece.drexel.edu/index.php?title=SynchroTrace">http://vlsi.ece.drexel.edu/index.php?title=SynchroTrace</a><br> - <a href="https://sites.tufts.edu/tcal/current-research-projects/synchrotrace/">https://sites.tufts.edu/tcal/current-research-projects/synchrotrace/</a><br> - Synchronization-Aware Architecture-Agnostic Traces for Lightweight Multicore Simulation of CMP and HPC Workloads<br> - 2018 ACM Transactions on Architecture and Code Optimization (TACO) 15(1)<br> - <a href="http://vlsi.ece.drexel.edu/images/b/b1/ST_2018.pdf">http://vlsi.ece.drexel.edu/images/b/b1/ST_2018.pdf</a><br> - SynchroTrace: Synchronization-aware Architecture-agnostic Traces for Light-Weight Multicore Simulation<br> - 2015 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)<br> - <a href="http://dpac.ece.drexel.edu/wp-content/uploads/2013/04/SynchroTrace.pdf">http://dpac.ece.drexel.edu/wp-content/uploads/2013/04/SynchroTrace.pdf</a></p> <h2>Hardware Assistance / Processor Tracing</h2> <ul> <li>Hardware-assisted instruction profiling and latency detection <ul> <li>The Journal of Engineering, Volume 2016, Issue 10</li> <li>Suchakrapani Datt Sharma, Michel Dagenais</li> <li><a href="https://digital-library.theiet.org/content/journals/10.1049/joe.2016.0127">https://digital-library.theiet.org/content/journals/10.1049/joe.2016.0127</a></li> </ul> </li> <li>Hardware-assisted software event tracing <ul> <li>Concurrency and Computation: Practice and Experience, 29(10) 2017</li> <li>Vergé, A., Ezzati-Jivan, N. &amp; Dagenais, M. R.</li> <li><a href="https://publications.polymtl.ca/2981/">https://publications.polymtl.ca/2981/</a></li> </ul> </li> <li>Hardware trace reconstruction of runtime compiled code <ul> <li>Software: Practice and Experience 2018</li> <li>Sharma, S.D. &amp; Dagenais, M.R.</li> <li><a href="https://doi.org/10.1002/spe.2567">https://doi.org/10.1002/spe.2567</a></li> </ul> </li> <li>Hardware Tracing for Fast and Precise Performance Analysis <ul> <li>2015; Suchakra Sharma</li> <li><a href="https://thenewstack.io/hardware-tracing-fast-precise-performance-analysis/">https://thenewstack.io/hardware-tracing-fast-precise-performance-analysis/</a></li> </ul> </li> <li>In-Depth System Analysis Using Hardware-Assisted Tracing <ul> <li><a href="https://www.sigarch.org/in-depth-system-analysis-using-hardware-assisted-tracing/">https://www.sigarch.org/in-depth-system-analysis-using-hardware-assisted-tracing/</a></li> </ul> </li> <li>Low-Impact System Performance Analysis Using Hardware Assisted Tracing Techniques <ul> <li>2017 PhD thesis; Sharma, S. D.</li> <li><a href="https://publications.polymtl.ca/2485/">https://publications.polymtl.ca/2485/</a></li> <li><a href="https://publications.polymtl.ca/2485/1/2017_SuchakrapaniDattSharma.pdf">https://publications.polymtl.ca/2485/1/2017_SuchakrapaniDattSharma.pdf</a></li> </ul> </li> <li>Low Overhead Hardware-Assisted Virtual Machine Analysis and Profiling <ul> <li>2016 IEEE Globecom Workshops</li> <li>Sharma, S. D., Bastien, G., Nemati, H. &amp; Dagenais, M.</li> <li><a href="https://publications.polymtl.ca/2975/1/2016_Sharma_Low_overhead_hardware-assisted_virtual_machive.pdf">https://publications.polymtl.ca/2975/1/2016_Sharma_Low_overhead_hardware-assisted_virtual_machive.pdf</a></li> </ul> </li> <li>libbts.c: minimal BTS tracing wrapper for Linux Perf <ul> <li><a href="https://gist.github.com/pkhuong/1ce34e33c6df4b9be3bc9beb22415a47">https://gist.github.com/pkhuong/1ce34e33c6df4b9be3bc9beb22415a47</a></li> </ul> </li> <li>Andi Kleen's Intel Processor Trace resources <ul> <li><a href="http://halobates.de/blog/p/406">http://halobates.de/blog/p/406</a></li> <li><a href="http://web.archive.org/http://halobates.de/blog/p/406">http://web.archive.org/http://halobates.de/blog/p/406</a></li> <li>Cheat sheet for Intel Processor Trace with Linux perf and gdb <ul> <li><a href="https://halobates.de/blog/p/410">https://halobates.de/blog/p/410</a></li> <li><a href="http://web.archive.org/http://halobates.de/blog/p/410">http://web.archive.org/http://halobates.de/blog/p/410</a></li> </ul> </li> <li>Simple-PT: Simple Intel CPU processor tracing on Linux <ul> <li><a href="https://github.com/andikleen/simple-pt">https://github.com/andikleen/simple-pt</a></li> </ul> </li> </ul> </li> <li>Intel Processor Trace <ul> <li>Enhance performance analysis with Intel Processor Trace - <a href="https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace">https://easyperf.net/blog/2019/08/23/Intel-Processor-Trace</a></li> <li>Better debugging experience - <a href="https://easyperf.net/blog/2019/08/30/Intel-PT-part2">https://easyperf.net/blog/2019/08/30/Intel-PT-part2</a></li> <li>Analyzing performance glitches - <a href="https://easyperf.net/blog/2019/09/06/Intel-PT-part3">https://easyperf.net/blog/2019/09/06/Intel-PT-part3</a></li> <li>Better profiling experience - <a href="https://easyperf.net/blog/2019/09/13/Intel-PT-part4">https://easyperf.net/blog/2019/09/13/Intel-PT-part4</a></li> </ul> </li> <li>Failure Sketches: A Better Way to Debug <ul> <li>HotOS 2015</li> <li>Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, Madan Musuvathi, George Candea</li> <li><a href="https://www.usenix.org/conference/hotos15/workshop-program/presentation/kasikci">https://www.usenix.org/conference/hotos15/workshop-program/presentation/kasikci</a></li> <li><a href="http://dslab.epfl.ch/pubs/failure-sketches.pdf">http://dslab.epfl.ch/pubs/failure-sketches.pdf</a></li> <li><a href="https://web.eecs.umich.edu/~barisk/public/fs-slides.pdf">https://web.eecs.umich.edu/~barisk/public/fs-slides.pdf</a></li> </ul> </li> <li>Failure Sketching: A Technique for Automated Root Cause Diagnosis of In-Production Failures <ul> <li>Symposium on Operating Systems Principles (SOSP) 2015</li> <li>Baris Kasikci, Benjamin Schubert, Cristiano Pereira, Gilles Pokam, George Candea</li> <li><a href="https://www.youtube.com/watch?v=99hXVFe33w8">https://www.youtube.com/watch?v=99hXVFe33w8</a></li> <li><a href="http://dslab.epfl.ch/proj/gist/">http://dslab.epfl.ch/proj/gist/</a></li> <li><a href="http://dslab.epfl.ch/pubs/gist.pdf">http://dslab.epfl.ch/pubs/gist.pdf</a></li> <li><a href="https://web.eecs.umich.edu/~barisk/public/gist-slides.pdf">https://web.eecs.umich.edu/~barisk/public/gist-slides.pdf</a></li> <li><a href="https://blog.acolyer.org/2015/10/12/failure-sketching-a-technique-for-automated-root-cause-diagnosis-of-in-production-failures/">https://blog.acolyer.org/2015/10/12/failure-sketching-a-technique-for-automated-root-cause-diagnosis-of-in-production-failures/</a></li> </ul> </li> <li>Fuzzing <ul> <li>Internals of Hongfuzz - Intel PT <ul> <li><a href="https://tunnelshade.in/blog/2018/09/hongfuzz-intel-pt-instrumentation/">https://tunnelshade.in/blog/2018/09/hongfuzz-intel-pt-instrumentation/</a></li> </ul> </li> <li>PTrix: Efficient Hardware-Assisted Fuzzing for COTS Binary <ul> <li>2019 Asia Conference on Computer and Communications Security (AsiaCCS)</li> <li>Yaohui Chen, Dongliang Mu, Jun Xu, Zhichuang Sun, Wenbo Shen, Xinyu Xing, Long Lu, Bing Mao</li> <li><a href="https://arxiv.org/abs/1905.10499">https://arxiv.org/abs/1905.10499</a></li> <li><a href="https://github.com/junxzm1990/afl-pt">https://github.com/junxzm1990/afl-pt</a></li> </ul> </li> <li>WinAFL Intel PT mode <ul> <li><a href="https://github.com/googleprojectzero/winafl/blob/master/readme_pt.md">https://github.com/googleprojectzero/winafl/blob/master/readme_pt.md</a></li> </ul> </li> </ul> </li> <li>Intel PT Micro Tutorial - MICRO-48 (2015) <ul> <li><a href="https://sites.google.com/site/intelptmicrotutorial/">https://sites.google.com/site/intelptmicrotutorial/</a></li> </ul> </li> <li>libipt - an Intel(R) Processor Trace decoder library <ul> <li><a href="https://github.com/01org/processor-trace">https://github.com/01org/processor-trace</a></li> <li><a href="https://github.com/intel/libipt">https://github.com/intel/libipt</a></li> </ul> </li> <li>Linux perf Documentation: Intel Processor Trace <ul> <li><a href="https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/intel-pt.txt">https://github.com/torvalds/linux/blob/master/tools/perf/Documentation/intel-pt.txt</a></li> </ul> </li> <li>Inferring Fine-grained Control Flow Inside SGX Enclaves with Branch Shadowing <ul> <li><a href="https://arxiv.org/abs/1611.06952">https://arxiv.org/abs/1611.06952</a></li> <li><a href="https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/lee-sangho">https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/lee-sangho</a></li> <li>"Table 1: Measuring branch misprediction penalty with RDTSCP, Intel PT CYC packet, and LBR elapsed cycle (10,000 times). We put 120 NOP instructions at the fall-through path. The LBR elapsed cycle is less noisy than RDTSCP and Intel PT."</li> </ul> </li> <li>POMP: Postmortem Program Analysis with Hardware-Enhanced Post-Crash Artifacts <ul> <li>USENIX Security 2017</li> <li>Jun Xu, Dongliang Mu, Xinyu Xing, Peng Liu, Ping Chen, Bing Mao</li> <li><a href="https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/xu-jun">https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/xu-jun</a></li> <li><a href="https://github.com/junxzm1990/pomp">https://github.com/junxzm1990/pomp</a></li> </ul> </li> <li>Processor-Tracing Guided Region Formation in Dynamic Binary Translation <ul> <li>ACM Transactions on Architecture and Code Optimization (TACO) 15(4):52 (2018)</li> <li>Ding-Yong Hong, Jan-Jan Wu, Yu-Ping Liu, Sheng-Yu Fu, Wei-Chung Hsu</li> <li><a href="https://dl.acm.org/citation.cfm?id=3281664">https://dl.acm.org/citation.cfm?id=3281664</a></li> <li>paper: <a href="https://www.iis.sinica.edu.tw/papers/wuj/22203-F.pdf">https://www.iis.sinica.edu.tw/papers/wuj/22203-F.pdf</a></li> <li>slides: <a href="https://www.iis.sinica.edu.tw/papers/wuj/22189-A.pdf">https://www.iis.sinica.edu.tw/papers/wuj/22189-A.pdf</a></li> </ul> </li> <li>SATT Software Analyze Trace Tool <ul> <li>Experimental Linux SW tool to trace, process and analyze full stack SW traces utilizing Intel HW tracing block Intel PT (Intel Processor Trace).</li> <li><a href="https://github.com/01org/satt">https://github.com/01org/satt</a></li> </ul> </li> <li>Towards Production-Run Heisenbugs Reproduction on Commercial Hardware <ul> <li>USENIX Annual Technical Conference (ATC) 2017</li> <li>Shiyou Huang, Bowen Cai, Jeff Huang</li> <li><a href="https://www.usenix.org/conference/atc17/technical-sessions/presentation/huang">https://www.usenix.org/conference/atc17/technical-sessions/presentation/huang</a></li> </ul> </li> <li>WinIPT: Windows Library for Intel Process Trace <ul> <li><a href="https://github.com/ionescu007/winipt">https://github.com/ionescu007/winipt</a></li> </ul> </li> <li>Windows Intel PT Support <ul> <li>This driver implements the Intel Processor Trace functionality in Intel Skylake architecture for Microsoft Windows.</li> <li><a href="https://github.com/intelpt/WindowsIntelPT">https://github.com/intelpt/WindowsIntelPT</a></li> <li>Harnessing Intel Processor Trace on Windows for Vulnerability Discovery <ul> <li>Richard Johnson - <a href="http://moflow.org/">http://moflow.org/</a></li> <li><a href="http://moflow.org/Presentations/Harnessing%20Intel%20Processor%20Trace%20on%20Windows%20for%20Vulnerability%20Discovery%20-%20rjohnson.pdf">http://moflow.org/Presentations/Harnessing%20Intel%20Processor%20Trace%20on%20Windows%20for%20Vulnerability%20Discovery%20-%20rjohnson.pdf</a></li> <li>HITB2017AMS</li> <li><a href="https://www.youtube.com/watch?v=r8lzui24Cdw">https://www.youtube.com/watch?v=r8lzui24Cdw</a></li> <li><a href="http://conference.hitb.org/hitbsecconf2017ams/materials/D1T1%20-%20Richard%20Johnson%20-%20Harnessing%20Intel%20Processor%20Trace%20on%20Windows%20for%20Vulnerability%20Discovery.pdf">http://conference.hitb.org/hitbsecconf2017ams/materials/D1T1%20-%20Richard%20Johnson%20-%20Harnessing%20Intel%20Processor%20Trace%20on%20Windows%20for%20Vulnerability%20Discovery.pdf</a></li> </ul> </li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>barectf: Generator of ANSI C tracers which output CTF <ul> <li>a command-line generator of ANSI C tracers which output Common Trace Format packets natively</li> <li><a href="https://github.com/efficios/barectf">https://github.com/efficios/barectf</a></li> </ul> </li> <li>Drltrace: library calls tracer for Windows and Linux applications <ul> <li><a href="https://github.com/mxmssh/drltrace">https://github.com/mxmssh/drltrace</a></li> <li><a href="https://www.slideshare.net/MaximShudrak/tricky-sample-hack-it-easy-applying-dynamic-binary-inastrumentation-to-lightweight-malware-behaviour-analysis">https://www.slideshare.net/MaximShudrak/tricky-sample-hack-it-easy-applying-dynamic-binary-inastrumentation-to-lightweight-malware-behaviour-analysis</a></li> </ul> </li> <li>DTrace <ul> <li>Awesome DTrace - <a href="http://awesome-dtrace.com/">http://awesome-dtrace.com/</a></li> <li>DTrace Safety - <a href="http://dtrace.org/blogs/bmc/2005/07/19/dtrace-safety/">http://dtrace.org/blogs/bmc/2005/07/19/dtrace-safety/</a></li> <li>DTrace Internals: Digging into DTrace <ul> <li>BSDCan 2017; Arun Thomas</li> <li><a href="https://www.youtube.com/watch?v=Vstwq5l_dwY">https://www.youtube.com/watch?v=Vstwq5l_dwY</a></li> <li><a href="http://www.bsdcan.org/2017/schedule/events/793.en.html">http://www.bsdcan.org/2017/schedule/events/793.en.html</a></li> </ul> </li> <li>DTrace for Linux <ul> <li><a href="https://github.com/dtrace4linux/linux">https://github.com/dtrace4linux/linux</a></li> <li>FOSDEM 2018; Tomáš Jedlička <ul> <li><a href="https://fosdem.org/2018/schedule/event/debugging_tools_dtrace/">https://fosdem.org/2018/schedule/event/debugging_tools_dtrace/</a></li> </ul> </li> </ul> </li> <li>DTrace on Windows <ul> <li><a href="https://github.com/opendtrace/opendtrace/tree/windows">https://github.com/opendtrace/opendtrace/tree/windows</a></li> <li><a href="https://github.com/microsoft/DTrace-on-Windows">https://github.com/microsoft/DTrace-on-Windows</a></li> <li><a href="https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902">https://techcommunity.microsoft.com/t5/Windows-Kernel-Internals/DTrace-on-Windows/ba-p/362902</a></li> </ul> </li> <li>DTrace Review <ul> <li>Google Tech Talks; August 15, 2007; Bryan Cantrill</li> <li><a href="https://www.youtube.com/watch?v=TgmA48fILq8">https://www.youtube.com/watch?v=TgmA48fILq8</a></li> <li><a href="https://www.youtube.com/watch?v=6chLw2aodYQ">https://www.youtube.com/watch?v=6chLw2aodYQ</a></li> </ul> </li> </ul> </li> <li>minitrace: Simple C/C++ library for producing JSON traces suitable for Chrome's built-in trace viewer (about:tracing) <ul> <li><a href="https://github.com/hrydgard/minitrace">https://github.com/hrydgard/minitrace</a></li> </ul> </li> <li>Phosphor: High performance event tracing <ul> <li><a href="http://couchbase.github.io/phosphor/">http://couchbase.github.io/phosphor/</a></li> <li><a href="https://github.com/couchbase/phosphor">https://github.com/couchbase/phosphor</a></li> </ul> </li> <li>PIEtrace: Platform Independent Executable Trace <ul> <li><a href="https://sites.google.com/site/pitprj12/">https://sites.google.com/site/pitprj12/</a></li> </ul> </li> </ul> <h2>Software: Viewers</h2> <ul> <li>Babeltrace <ul> <li>The Babeltrace project provides trace read and write libraries, as well as a trace converter. Plugins can be created for any trace format to allow its conversion to/from another trace format.</li> <li><a href="http://diamon.org/babeltrace/">http://diamon.org/babeltrace/</a></li> <li><a href="https://github.com/efficios/babeltrace">https://github.com/efficios/babeltrace</a></li> <li>Common Trace Format <ul> <li><a href="https://diamon.org/ctf/">https://diamon.org/ctf/</a></li> </ul> </li> </ul> </li> <li>FlameGraph: Stack trace visualize <ul> <li><a href="https://github.com/brendangregg/FlameGraph">https://github.com/brendangregg/FlameGraph</a></li> <li><a href="http://www.brendangregg.com/flamegraphs.html">http://www.brendangregg.com/flamegraphs.html</a></li> </ul> </li> <li>LTTng Scope <ul> <li>a trace viewer and analyzer for CTF traces, with a focus on LTTng kernel and user space traces.</li> <li><a href="https://github.com/lttng/lttng-scope">https://github.com/lttng/lttng-scope</a></li> <li><a href="https://lttng.org/blog/2018/03/21/lttng-scope-0.3-a-basic-tutorial/">https://lttng.org/blog/2018/03/21/lttng-scope-0.3-a-basic-tutorial/</a></li> </ul> </li> <li>rgat: An instruction trace visualisation tool for dynamic program analysis <ul> <li><a href="https://github.com/ncatlin/rgat">https://github.com/ncatlin/rgat</a></li> </ul> </li> <li>Trace Compass: an open-source (EPL-licensed) trace viewer and analyzer <ul> <li><a href="http://tracecompass.org/">http://tracecompass.org/</a></li> <li><a href="https://github.com/tracecompass">https://github.com/tracecompass</a></li> <li><a href="https://wiki.eclipse.org/Trace_Compass">https://wiki.eclipse.org/Trace_Compass</a></li> </ul> </li> </ul> <h2>Software: Linux</h2> <ul> <li>Linux tracing systems &amp; how they fit together <ul> <li><a href="https://jvns.ca/blog/2017/07/05/linux-tracing-systems/">https://jvns.ca/blog/2017/07/05/linux-tracing-systems/</a></li> </ul> </li> <li>KUtrace <ul> <li>Low-overhead tracing of all Linux kernel-user transitions, for serious performance analysis. Includes kernel patches, loadable module, and post-processing software. Output is HTML/SVG per-CPU-core timeline that you can pan/zoom down to the nanosecond.</li> <li><a href="https://github.com/dicksites/KUtrace">https://github.com/dicksites/KUtrace</a></li> <li>KUTrace: Where have all the nanoseconds gone?, Richard Sites <ul> <li>Tracing Summit 2017; Richard Sites</li> <li><a href="https://www.youtube.com/watch?v=UYwWollxzAk">https://www.youtube.com/watch?v=UYwWollxzAk</a></li> <li><a href="https://tracingsummit.org/wiki/TracingSummit2017KUTrace">https://tracingsummit.org/wiki/TracingSummit2017KUTrace</a></li> <li><a href="https://tracingsummit.org/w/images/3/30/TS17-kutrace.pdf">https://tracingsummit.org/w/images/3/30/TS17-kutrace.pdf</a></li> </ul> </li> <li>Benchmarking "Hello, World!" <ul> <li>Six different views of the execution of "Hello, World!" show what is often missing in today's tools</li> <li>ACM Queue 16, 5 (2018); Richard L. Sites</li> <li><a href="https://queue.acm.org/detail.cfm?id=3291278">https://queue.acm.org/detail.cfm?id=3291278</a></li> </ul> </li> <li>KUtrace 2020 <ul> <li>Stanford EE380 Computer Systems Colloquium (2020); Dick Sites</li> <li><a href="https://www.youtube.com/watch?v=2HE7tSZGna0">https://www.youtube.com/watch?v=2HE7tSZGna0</a></li> <li><a href="https://ee.stanford.edu/event/seminar/ee380-computer-systems-colloquium-presents-kutrace-2020">https://ee.stanford.edu/event/seminar/ee380-computer-systems-colloquium-presents-kutrace-2020</a></li> </ul> </li> </ul> </li> <li>ltrace: Debugging program to track runtime library calls in dynamically linked programs <ul> <li>ltrace intercepts and records dynamic library calls which are called by an executed process and the signals received by that process. It can also intercept and print the system calls executed by the program.</li> <li><a href="http://ltrace.org/">http://ltrace.org/</a></li> <li><a href="https://gitlab.com/cespedes/ltrace">https://gitlab.com/cespedes/ltrace</a></li> <li>How does ltrace work? - <a href="https://blog.packagecloud.io/eng/2016/03/14/how-does-ltrace-work/">https://blog.packagecloud.io/eng/2016/03/14/how-does-ltrace-work/</a></li> </ul> </li> <li>StackTrack — Linux Call graph visualization and execution tracking <ul> <li><a href="https://stacktrack.github.io/">https://stacktrack.github.io/</a></li> </ul> </li> <li>uftrace: Function (graph) tracer for user-space <ul> <li><a href="https://github.com/namhyung/uftrace">https://github.com/namhyung/uftrace</a></li> <li>Understanding the runtime behaviors of C++ programs using uftrace tool <ul> <li>CppCon 2017; Honggyu Kim</li> <li><a href="https://www.youtube.com/watch?v=s0B8hV2O8ps">https://www.youtube.com/watch?v=s0B8hV2O8ps</a></li> </ul> </li> </ul> </li> </ul> <h3>Software: Linux - BCC/BPF</h3> <h4>Software: Linux - BCC/BPF - Readings</h4> <ul> <li>Dive into BPF: a list of reading material <ul> <li><a href="https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/">https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/</a></li> </ul> </li> <li>The BSD Packet Filter <ul> <li>USENIX Winter 1993; Steven McCanne and Van Jacobson</li> <li><a href="https://www.usenix.org/conference/usenix-winter-1993-conference/presentation/bsd-packet-filter-new-architecture-user-level-">https://www.usenix.org/conference/usenix-winter-1993-conference/presentation/bsd-packet-filter-new-architecture-user-level-</a></li> <li>PWL 2017 talk by Suchakra Sharma <ul> <li><a href="https://speakerdeck.com/tuxology/the-bsd-packet-filter">https://speakerdeck.com/tuxology/the-bsd-packet-filter</a></li> <li><a href="http://step.polymtl.ca/~suchakra/PWL-Jun28-MTL.pdf">http://step.polymtl.ca/~suchakra/PWL-Jun28-MTL.pdf</a></li> <li>Notes on BPF &amp; eBPF - <a href="https://jvns.ca/blog/2017/06/28/notes-on-bpf---ebpf/">https://jvns.ca/blog/2017/06/28/notes-on-bpf---ebpf/</a></li> </ul> </li> </ul> </li> <li>BPF and XDP Reference Guide <ul> <li><a href="http://docs.cilium.io/en/stable/bpf/">http://docs.cilium.io/en/stable/bpf/</a></li> </ul> </li> <li>Brendan D. Gregg <ul> <li>Linux Enhanced BPF (eBPF) Tracing Tools - <a href="http://www.brendangregg.com/ebpf.html">http://www.brendangregg.com/ebpf.html</a></li> <li>BPF Performance Tools: Linux System and Application Observability <ul> <li><a href="http://www.brendangregg.com/bpf-performance-tools-book.html">http://www.brendangregg.com/bpf-performance-tools-book.html</a></li> <li><a href="https://github.com/brendangregg/bpf-perf-tools-book">https://github.com/brendangregg/bpf-perf-tools-book</a></li> </ul> </li> </ul> </li> <li>Ferris Ellis <ul> <li>eBPF, part 1: Past, Present, and Future - <a href="https://ferrisellis.com/content/ebpf_past_present_future/">https://ferrisellis.com/content/ebpf_past_present_future/</a></li> <li>eBPF, part 2: Syscall and Map Types - <a href="https://ferrisellis.com/content/ebpf_syscall_and_maps/">https://ferrisellis.com/content/ebpf_syscall_and_maps/</a></li> </ul> </li> <li>Linux Tracing Workshops Materials - Sasha Goldshtein <ul> <li><a href="https://github.com/goldshtn/linux-tracing-workshop">https://github.com/goldshtn/linux-tracing-workshop</a></li> </ul> </li> </ul> <h4>Software: Linux - BCC/BPF - Software</h4> <ul> <li>BPFtrace: High-level tracing language for Linux eBPF <ul> <li><a href="https://github.com/iovisor/bpftrace">https://github.com/iovisor/bpftrace</a></li> <li><a href="http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html">http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html</a></li> </ul> </li> <li>BCC - Tools for BPF-based Linux IO analysis, networking, monitoring, and more <ul> <li><a href="http://iovisor.github.io/bcc/">http://iovisor.github.io/bcc/</a></li> <li><a href="https://github.com/iovisor/bcc">https://github.com/iovisor/bcc</a></li> <li><a href="https://www.iovisor.org/technology/bcc">https://www.iovisor.org/technology/bcc</a></li> <li><a href="https://github.com/iovisor/bcc/blob/master/LINKS.md">https://github.com/iovisor/bcc/blob/master/LINKS.md</a></li> </ul> </li> <li>uBPF: Userspace eBPF VM <ul> <li><a href="https://github.com/iovisor/ubpf">https://github.com/iovisor/ubpf</a></li> </ul> </li> <li>ply: Dynamic Tracing in Linux <ul> <li>A light-weight dynamic tracer for Linux that leverages the kernel's BPF VM in concert with kprobes and tracepoints to attach probes to arbitrary points in the kernel.</li> <li><a href="https://wkz.github.io/ply/">https://wkz.github.io/ply/</a></li> <li><a href="https://github.com/iovisor/ply">https://github.com/iovisor/ply</a></li> </ul> </li> </ul> <h4>Software: Linux - BCC/BPF - Talks</h4> <ul> <li>Tools and mechanisms to debug BPF programs <ul> <li>FOSDEM 2020; Quentin Monnet</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_bpf/">https://fosdem.org/2020/schedule/event/debugging_bpf/</a></li> </ul> </li> </ul> <h3>Software: Linux - ftrace</h3> <ul> <li>Ftrace - <a href="https://alex.dzyoba.com/blog/ftrace/">https://alex.dzyoba.com/blog/ftrace/</a></li> <li>Ftrace: The hidden light switch - <a href="https://lwn.net/Articles/608497/">https://lwn.net/Articles/608497/</a></li> <li>ftrace: Where modifying a running kernel all started <ul> <li>Kernel Recipes 2019; Steven Rostedt</li> <li><a href="https://www.youtube.com/watch?v=93uE_kWWQjs">https://www.youtube.com/watch?v=93uE_kWWQjs</a></li> </ul> </li> <li>Debugging realtime application with Ftrace <ul> <li>FOSDEM 2018; Pierre Ficheux</li> <li><a href="https://fosdem.org/2018/schedule/event/debugging_tools_ftrace/">https://fosdem.org/2018/schedule/event/debugging_tools_ftrace/</a></li> </ul> </li> <li>Hooking Linux Kernel Functions <ul> <li>Part 1: Looking for the Perfect Solution - <a href="https://www.apriorit.com/dev-blog/544-hooking-linux-functions-1">https://www.apriorit.com/dev-blog/544-hooking-linux-functions-1</a></li> <li>Part 2: How to Hook Functions with Ftrace - <a href="https://www.apriorit.com/dev-blog/546-hooking-linux-functions-2">https://www.apriorit.com/dev-blog/546-hooking-linux-functions-2</a></li> <li>Part 3: What Are the Main Pros and Cons of Ftrace? - <a href="https://www.apriorit.com/dev-blog/547-hooking-linux-functions-3">https://www.apriorit.com/dev-blog/547-hooking-linux-functions-3</a></li> <li>ftrace-hook: Linux kernel module demonstrating usage of ftrace framework for function hooking <ul> <li><a href="https://github.com/ilammy/ftrace-hook">https://github.com/ilammy/ftrace-hook</a></li> </ul> </li> </ul> </li> <li>KernelShark <ul> <li><a href="http://rostedt.homelinux.com/kernelshark/">http://rostedt.homelinux.com/kernelshark/</a></li> <li>Using KernelShark to analyze the real-time scheduler - <a href="https://lwn.net/Articles/425583/">https://lwn.net/Articles/425583/</a></li> <li>KernelShark 1.0; What’s new and what’s coming - <a href="https://kernel-recipes.org/en/2018/talks/kernelshark-1-0-whats-new-and-whats-coming/">https://kernel-recipes.org/en/2018/talks/kernelshark-1-0-whats-new-and-whats-coming/</a></li> </ul> </li> <li>perf-tools: Performance analysis tools based on Linux perf_events (aka perf) and ftrace <ul> <li><a href="https://github.com/brendangregg/perf-tools">https://github.com/brendangregg/perf-tools</a></li> </ul> </li> <li>Tracing the Linux kernel with ftrace <ul> <li><a href="https://embeddedbits.org/tracing-the-linux-kernel-with-ftrace/">https://embeddedbits.org/tracing-the-linux-kernel-with-ftrace/</a></li> </ul> </li> <li>Understanding the Linux Kernel via Ftrace - Kernel Recipes 2017 - Steven Rostedt <ul> <li><a href="https://kernel-recipes.org/en/2017/talks/understanding-the-linux-kernel-via-ftrace/">https://kernel-recipes.org/en/2017/talks/understanding-the-linux-kernel-via-ftrace/</a></li> <li><a href="https://www.youtube.com/watch?v=2ff-7UTg5rE">https://www.youtube.com/watch?v=2ff-7UTg5rE</a></li> </ul> </li> <li>See what your computer is doing with Ftrace utilities <ul> <li>linux.conf.au 2019; Steven Rostedt</li> <li><a href="https://www.youtube.com/watch?v=68osT1soAPM">https://www.youtube.com/watch?v=68osT1soAPM</a></li> <li><a href="https://2019.linux.conf.au/schedule/presentation/130/">https://2019.linux.conf.au/schedule/presentation/130/</a></li> </ul> </li> </ul> <h3>Software: Linux - LTTng</h3> <ul> <li>LTTng: Linux Trace Toolkit Next Generation <ul> <li>an open source tracing framework for Linux</li> <li><a href="https://lttng.org/">https://lttng.org/</a></li> <li><a href="https://github.com/lttng">https://github.com/lttng</a></li> </ul> </li> <li>LTTng analyses: Analyses scripts for LTTng kernel and user-space traces <ul> <li><a href="https://github.com/lttng/lttng-analyses">https://github.com/lttng/lttng-analyses</a></li> </ul> </li> <li>LTTng live streaming example <ul> <li><a href="http://pavelmakhov.com/2017/01/lttng-streaming">http://pavelmakhov.com/2017/01/lttng-streaming</a></li> </ul> </li> </ul> <h3>Software: Linux - ptrace</h3> <ul> <li>ptrace - process trace - <a href="http://man7.org/linux/man-pages/man2/ptrace.2.html">http://man7.org/linux/man-pages/man2/ptrace.2.html</a></li> <li>Adventures into ptrace(2) Hell - <a href="https://www.cyphar.com/blog/post/20160703-remainroot-ptrace-hell">https://www.cyphar.com/blog/post/20160703-remainroot-ptrace-hell</a></li> <li>DetTrace: A Reproducible Container Abstraction <ul> <li>A determinizing tracer using Ptrace</li> <li><a href="https://github.com/dettrace/dettrace">https://github.com/dettrace/dettrace</a></li> <li>Reproducible Containers <ul> <li>Architectural Support for Programming Languages and Operating Systems (ASPLOS) 2020</li> <li>Omar S. Navarro Leija, Kelly Shiptoski, Ryan G. Scott, Baojun Wang, Nicholas Renner, Ryan R. Newton, Joseph Devietti</li> <li><a href="https://doi.org/10.1145/3373376.3378519">https://doi.org/10.1145/3373376.3378519</a></li> <li><a href="https://gatowololo.github.io/resources/publications/dettrace.pdf">https://gatowololo.github.io/resources/publications/dettrace.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=YkmS-vf12nE">https://www.youtube.com/watch?v=YkmS-vf12nE</a></li> </ul> </li> </ul> </li> <li>Hiding Call To Ptrace - <a href="https://github.com/yellowbyte/analysis-of-anti-analysis/blob/master/research/hiding_call_to_ptrace/hiding_call_to_ptrace.md">https://github.com/yellowbyte/analysis-of-anti-analysis/blob/master/research/hiding_call_to_ptrace/hiding_call_to_ptrace.md</a></li> <li>Intercepting and Emulating Linux System Calls with Ptrace - <a href="http://nullprogram.com/blog/2018/06/23/">http://nullprogram.com/blog/2018/06/23/</a></li> <li>Ptrace-burrito: A plugin style wrapper around ptrace on Linux. <ul> <li><a href="https://github.com/brainsmoke/ptrace-burrito">https://github.com/brainsmoke/ptrace-burrito</a></li> </ul> </li> <li>Using Ptrace For Fun And Profit <ul> <li><a href="https://eklitzke.org/ptrace">https://eklitzke.org/ptrace</a></li> <li><a href="https://eklitzke.org/ptrace-continued">https://eklitzke.org/ptrace-continued</a></li> <li>Using ptrace(2) To Call a Userspace Function: Example of how to use the ptrace(2) system call to call a userspace method. <ul> <li><a href="https://github.com/eklitzke/ptrace-call-userspace">https://github.com/eklitzke/ptrace-call-userspace</a></li> </ul> </li> </ul> </li> </ul> <h3>Software: Linux - strace</h3> <ul> <li>strace - the linux syscall tracer <ul> <li><a href="https://github.com/strace/strace">https://github.com/strace/strace</a></li> </ul> </li> <li>Strace little book <ul> <li><a href="https://github.com/NanXiao/strace-little-book">https://github.com/NanXiao/strace-little-book</a></li> <li><a href="https://nanxiao.gitbooks.io/strace-little-book/">https://nanxiao.gitbooks.io/strace-little-book/</a></li> </ul> </li> <li>Julia Evans <ul> <li>A zine about strace <ul> <li><a href="https://jvns.ca/blog/2015/04/14/strace-zine/">https://jvns.ca/blog/2015/04/14/strace-zine/</a></li> <li><a href="https://wizardzines.com/zines/strace/">https://wizardzines.com/zines/strace/</a></li> </ul> </li> <li>strace posts: <a href="https://jvns.ca/categories/strace/">https://jvns.ca/categories/strace/</a></li> </ul> </li> <li>How does strace work? <ul> <li><a href="https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/">https://blog.packagecloud.io/eng/2016/02/29/how-does-strace-work/</a></li> </ul> </li> <li>My Favourite Secret Weapon – strace <ul> <li><a href="https://zwischenzugs.com/2011/08/29/my-favourite-secret-weapon-strace/">https://zwischenzugs.com/2011/08/29/my-favourite-secret-weapon-strace/</a></li> </ul> </li> <li>Strace -- The Sysadmin's Microscope <ul> <li><a href="https://blogs.oracle.com/ksplice/strace-the-sysadmins-microscope">https://blogs.oracle.com/ksplice/strace-the-sysadmins-microscope</a></li> </ul> </li> <li>strace cheat sheet <ul> <li><a href="https://blog.packagecloud.io/eng/2015/11/15/strace-cheat-sheet/">https://blog.packagecloud.io/eng/2015/11/15/strace-cheat-sheet/</a></li> </ul> </li> <li>Strace: The Lost Chapter <ul> <li><a href="http://price.mit.edu/blog/2010/08/strace-the-lost-chapter/">http://price.mit.edu/blog/2010/08/strace-the-lost-chapter/</a></li> </ul> </li> </ul> <h4>Software: Linux - strace - Talks</h4> <ul> <li>strace talks <ul> <li><a href="https://github.com/strace/strace-talks">https://github.com/strace/strace-talks</a></li> </ul> </li> <li>Modern strace <ul> <li>DevConf.CZ 2019; Dmitry Levin</li> <li><a href="https://www.youtube.com/watch?v=cwDFpzQuWSU">https://www.youtube.com/watch?v=cwDFpzQuWSU</a></li> <li><a href="https://devconfcz2019.sched.com/event/JchD">https://devconfcz2019.sched.com/event/JchD</a></li> </ul> </li> <li>Postmodern strace <ul> <li>FOSDEM 2020; Dmitry Levin</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_strace_modern/">https://fosdem.org/2020/schedule/event/debugging_strace_modern/</a></li> </ul> </li> <li>strace --seccomp-bpf: a look under the hood <ul> <li>FOSDEM 2020; Paul Chaignon</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_strace_bpf/">https://fosdem.org/2020/schedule/event/debugging_strace_bpf/</a></li> <li><a href="https://pchaigno.github.io/strace/2019/10/02/introducing-strace-seccomp-bpf.html">https://pchaigno.github.io/strace/2019/10/02/introducing-strace-seccomp-bpf.html</a></li> </ul> </li> <li>Strace: Monitoring The Kernel-User-Space Conversation <ul> <li>NDC TechTown 2018; Michael Kerrisk</li> <li><a href="https://www.youtube.com/watch?v=oFt6V56BOlo">https://www.youtube.com/watch?v=oFt6V56BOlo</a></li> <li><a href="http://man7.org/conf/ndctechtown2018/system_call_tracing_with_strace-NDC-TechTown-Kerrisk.pdf">http://man7.org/conf/ndctechtown2018/system_call_tracing_with_strace-NDC-TechTown-Kerrisk.pdf</a></li> </ul> </li> <li>strace: new features <ul> <li>FOSDEM 2018; Dmitry Levin</li> <li><a href="https://fosdem.org/2018/schedule/event/debugging_tools_strace_features/">https://fosdem.org/2018/schedule/event/debugging_tools_strace_features/</a></li> </ul> </li> </ul> <h2>Software: Windows</h2> <ul> <li>ATrace: a tool for tracing execution of binaries on Windows <ul> <li><a href="https://github.com/K2/EhTrace">https://github.com/K2/EhTrace</a></li> </ul> </li> <li>Branch Monitoring Project <ul> <li>Branch Monitor is an alternative for runtime process monitoring on modern (Windows) systems. Our approach makes use of Branch Trace Store (BTS) from Intel's processors to implement a dynamic, transparent framework. The framework provide many analysis facilities, such as function call tracing and Control Flow Graph (CFG) reconstruction.</li> <li><a href="https://github.com/marcusbotacin/BranchMonitoringProject">https://github.com/marcusbotacin/BranchMonitoringProject</a></li> </ul> </li> <li>DIMCT: Dirty Inter Module Calls Tracer <ul> <li>Allows tracing inter module calls for a specific module within a Windows process.</li> <li><a href="https://github.com/amossys/dimct">https://github.com/amossys/dimct</a></li> </ul> </li> <li>libptrace: An event driven multi-core process debugging, tracing, and manipulation framework <ul> <li><a href="https://github.com/immunityinc/libptrace">https://github.com/immunityinc/libptrace</a></li> </ul> </li> <li>MemTrace: Memory Tracing Software <ul> <li><a href="https://github.com/deplinenoise/ig-memtrace">https://github.com/deplinenoise/ig-memtrace</a></li> </ul> </li> <li>tiny_tracer: A Pin Tool for tracing API calls and transition between sections of the traced module <ul> <li><a href="https://github.com/hasherezade/tiny_tracer">https://github.com/hasherezade/tiny_tracer</a></li> <li>Tracing executables with a Pin Tool (tiny_tracer) - <a href="https://www.youtube.com/watch?v=kurzaoWuSHA">https://www.youtube.com/watch?v=kurzaoWuSHA</a></li> </ul> </li> <li>tracectory: a tool to analyze and visualize x86 instruction traces <ul> <li><a href="https://bitbucket.org/oebeling/tracectory/wiki/Home">https://bitbucket.org/oebeling/tracectory/wiki/Home</a></li> <li><a href="https://bling.kapsi.fi/blog/x86-memory-access-visualization.html">https://bling.kapsi.fi/blog/x86-memory-access-visualization.html</a></li> </ul> </li> <li>WinIPT: The Windows Library for Intel Process Trace <ul> <li><a href="https://ionescu007.github.io/winipt/">https://ionescu007.github.io/winipt/</a></li> <li><a href="https://github.com/ionescu007/winipt">https://github.com/ionescu007/winipt</a></li> </ul> </li> </ul> <h3>Software: Windows - ETW</h3> <ul> <li>ETW Central <ul> <li><a href="https://randomascii.wordpress.com/2015/09/24/etw-central/">https://randomascii.wordpress.com/2015/09/24/etw-central/</a></li> </ul> </li> <li>etrace: Command-line tool for ETW tracing on files and real-time events <ul> <li><a href="https://github.com/goldshtn/etrace">https://github.com/goldshtn/etrace</a></li> </ul> </li> <li>EtwConsumerNT <ul> <li>Simple project that demonstrates how an ETW consumer can be created just by using NTDLL</li> <li><a href="https://github.com/wbenny/EtwConsumerNT">https://github.com/wbenny/EtwConsumerNT</a></li> </ul> </li> <li>KrabsETW: a modern C++ wrapper and a .NET wrapper around the low-level ETW trace consumption functions <ul> <li><a href="https://github.com/microsoft/krabsetw">https://github.com/microsoft/krabsetw</a></li> </ul> </li> <li>LiveStacks: Collect, aggregate, and display live stack traces for ETW events, including CPU sampling, of native and .NET processes. <ul> <li><a href="https://github.com/goldshtn/LiveStacks">https://github.com/goldshtn/LiveStacks</a></li> </ul> </li> <li>Hidden Treasure: Detecting Intrusions with ETW <ul> <li>Derbycon 2017; Zac Brown</li> <li><a href="http://www.irongeek.com/i.php?page=videos/derbycon7/t208-hidden-treasure-detecting-intrusions-with-etw-zac-brown">http://www.irongeek.com/i.php?page=videos/derbycon7/t208-hidden-treasure-detecting-intrusions-with-etw-zac-brown</a></li> </ul> </li> <li>UIforETW: User interface for recording and managing ETW traces <ul> <li><a href="https://randomascii.wordpress.com/2015/04/14/uiforetw-windows-performance-made-easier/">https://randomascii.wordpress.com/2015/04/14/uiforetw-windows-performance-made-easier/</a></li> <li><a href="https://github.com/google/UIforETW">https://github.com/google/UIforETW</a></li> </ul> </li> <li>wtrace: Command line tracing tool for Windows, based on ETW <ul> <li><a href="https://github.com/lowleveldesign/wtrace">https://github.com/lowleveldesign/wtrace</a></li> <li><a href="https://lowleveldesign.org/2017/10/25/wtrace-2-2/">https://lowleveldesign.org/2017/10/25/wtrace-2-2/</a></li> </ul> </li> </ul> <hr> <h1>Talks</h1> <ul> <li>Tracing Summit - <a href="http://tracingsummit.org/">http://tracingsummit.org/</a></li> </ul> <h2>2017</h2> <ul> <li>Now You See Me Too: Visual Tooling for Advanced System Analysis <ul> <li>LISA 2017; Suchakrapani Sharma</li> <li>Video: <a href="https://www.youtube.com/watch?v=ihauGhgWZgQ">https://www.youtube.com/watch?v=ihauGhgWZgQ</a></li> <li>Slides (Speaker Deck): <a href="https://speakerdeck.com/tuxology/visual-tooling-for-advanced-systems-analysis">https://speakerdeck.com/tuxology/visual-tooling-for-advanced-systems-analysis</a></li> <li>Slides (PDF): <a href="http://step.polymtl.ca/~suchakra/LISA-Nov2-SF.pdf">http://step.polymtl.ca/~suchakra/LISA-Nov2-SF.pdf</a></li> </ul> </li> </ul> <h2>2016</h2> <ul> <li>Designing Tracing Tools <ul> <li>Sysdig CCWFS 2016; Brendan Gregg</li> <li><a href="https://www.youtube.com/watch?v=uibLwoVKjec">https://www.youtube.com/watch?v=uibLwoVKjec</a></li> <li><a href="http://www.slideshare.net/brendangregg/designing-tracing-tools-67693476">http://www.slideshare.net/brendangregg/designing-tracing-tools-67693476</a></li> </ul> </li> <li>Give me 15 minutes and I'll change your view of Linux tracing <ul> <li>USENIX/LISA 2016; Brendan Gregg</li> <li><a href="https://www.youtube.com/watch?v=GsMs3n8CB6g">https://www.youtube.com/watch?v=GsMs3n8CB6g</a></li> </ul> </li> <li>Low-Level Tracing for Latency Analysis: From Baremetal to Hardware Tracing Blocks <ul> <li>Tracing Summit 2016</li> <li>Suchakrapani Datt Sharma (Ecole Polytechnique Montreal)</li> <li><a href="https://www.youtube.com/watch?v=D1nggb9OgD0">https://www.youtube.com/watch?v=D1nggb9OgD0</a></li> <li><a href="http://tracingsummit.org/wiki/TracingSummit2016LowLevelTracing">http://tracingsummit.org/wiki/TracingSummit2016LowLevelTracing</a></li> <li><a href="https://hsdm.dorsal.polymtl.ca/system/files/Suchakra-tracingsummit2016.pdf">https://hsdm.dorsal.polymtl.ca/system/files/Suchakra-tracingsummit2016.pdf</a></li> </ul> </li> <li>Using Linux tracing tools <ul> <li>O'Reilly Velocity Conference 2016; Sasha Goldshtein</li> <li><a href="https://www.oreilly.com/learning/using-linux-tracing-tools">https://www.oreilly.com/learning/using-linux-tracing-tools</a></li> </ul> </li> </ul> <h2>2014</h2> <ul> <li>What is That Process Doing? <ul> <li>Greg Price</li> <li><a href="http://price.mit.edu/tracing-w2014/">http://price.mit.edu/tracing-w2014/</a></li> <li>Open Source Bridge 2013 - <a href="https://www.youtube.com/watch?v=kYIDMIekgyU">https://www.youtube.com/watch?v=kYIDMIekgyU</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/swe-at-google/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/swe-at-google/index.html <p><a href="https://abseil.io/resources/swe-book">Website</a> | Online-available</p> <ul> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html">Preface</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#programming_over_time">Programming Over Time</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#googleapostrophes_perspective">Google’s Perspective</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#what_this_book_isnapostrophet">What This Book Isn’t</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#parting_remarks">Parting Remarks</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#_conventions_used_in_this_book">Conventions Used in This Book</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#_safari_books_online">O'Reilly Online Learning</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#_how_to_contact_us">How to Contact Us</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/pr01.html#_acknowledgments">Acknowledgments</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/part1.html">Thesis</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch01.html">What Is Software Engineering?</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#time_and_change">Time and Change</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#hyrumapostrophes_law">Hyrum’s Law</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#example_hash_ordering">Example: Hash Ordering</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#why_not_just_aim_for_quotation_marknoth">Why Not Just Aim for “Nothing Changes”?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#scale_and_efficiency">Scale and Efficiency</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#policies_that_donapostrophet_scale">Policies That Don’t Scale</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#policies_that_scale_well">Policies That Scale Well</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#example_compiler_upgrade">Example: Compiler Upgrade</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#shifting_left">Shifting Left</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#tradeoffs_and_costs">Trade-offs and Costs</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#example_markers">Example: Markers</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#inputs_to_decision_making">Inputs to Decision Making</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#example_distributed_builds">Example: Distributed Builds</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#example_deciding_between_time_and_scale">Example: Deciding Between Time and Scale</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#revisiting_decisionscomma_making_mistak">Revisiting Decisions, Making Mistakes</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch01.html#software_engineering_versus_programming">Software Engineering Versus Programming</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch01.html#conclusion">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch01.html#tlsemicolondr">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/part2.html">Culture</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch02.html">How to Work Well on Teams</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch02.html#help_me_hide_my_code">Help Me Hide My Code</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch02.html#the_genius_myth">The Genius Myth</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch02.html#hiding_considered_harmful">Hiding Considered Harmful</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#early_detection">Early Detection</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#the_bus_factor">The Bus Factor</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#pace_of_progress">Pace of Progress</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#in_shortcomma_donapostrophet_hide">In Short, Don’t Hide</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#itapostrophes_all_about_the_team">It’s All About the Team</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#the_three_pillars_of_social_interaction">The Three Pillars of Social Interaction</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#why_do_these_pillars_matterquestion_mar">Why Do These Pillars Matter?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#humilitycomma_respectcomma_and_trust_in">Humility, Respect, and Trust in Practice</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#blameless_postmortem_culture">Blameless Post-Mortem Culture</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#being_googley">Being Googley</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch02.html#conclusion-id00006">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch02.html#tlsemicolondr-id00171">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch03.html">Knowledge Sharing</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch03.html#challenges_to_learning">Challenges to Learning</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch03.html#philosophy">Philosophy</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch03.html#setting_the_stage_psychological_safety">Setting the Stage: Psychological Safety</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#mentorship">Mentorship</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#psychological_safety_in_large_groups">Psychological Safety in Large Groups</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#growing_your_knowledge">Growing Your Knowledge</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#ask_questions">Ask Questions</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#understand_context">Understand Context</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#scaling_your_questions_ask_the_communit">Scaling Your Questions: Ask the Community</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#group_chats">Group Chats</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#mailing_lists">Mailing Lists</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#yaqs_question_and_answer_platform">YAQS: Question-and-Answer Platform</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#scaling_your_knowledge_you_always_have">Scaling Your Knowledge: You Always Have Something to Teach</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#office_hours">Office Hours</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#tech_talks_and_classes">Tech Talks and Classes</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#documentation-id00039">Documentation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#code">Code</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#scaling_your_organizationapostrophes_kn">Scaling Your Organization’s Knowledge</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#cultivating_a_knowledge-sharing_culture">Cultivating a Knowledge-Sharing Culture</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#establishing_canonical_sources_of_infor">Establishing Canonical Sources of Information</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#staying_in_the_loop">Staying in the Loop</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#readability_standardized_mentorship_thr">Readability: Standardized Mentorship Through Code Review</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#what_is_the_readability_processquestion">What Is the Readability Process?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#why_have_this_processquestion_mark">Why Have This Process?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch03.html#conclusion-id00007">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch03.html#tlsemicolondrs-id00101">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html">Engineering for Equity</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#bias_is_the_default">Bias Is the Default</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#understanding_the_need_for_diversity">Understanding the Need for Diversity</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#building_multicultural_capacity">Building Multicultural Capacity</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#making_diversity_actionable">Making Diversity Actionable</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#reject_singular_approaches">Reject Singular Approaches</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#challenge_established_processes">Challenge Established Processes</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#values_versus_outcomes">Values Versus Outcomes</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#stay_curiouscomma_push_forward">Stay Curious, Push Forward</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#conclusion-id00008">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch04.html#tlsemicolondrs-id00103">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch05.html">How to Lead a Team</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#managers_and_tech_leads_left_parenthesi">Managers and Tech Leads (and Both)</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_engineering_manager">The Engineering Manager</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_tech_lead">The Tech Lead</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_tech_lead_manager">The Tech Lead Manager</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#moving_from_an_individual_contributor_r">Moving from an Individual Contributor Role to a Leadership Role</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_only_thing_to_fear_issemicolonwellc">The Only Thing to Fear Is…Well, Everything</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#servant_leadership">Servant Leadership</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_engineering_manage">The Engineering Manager</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#manager_is_a_four-letter_word">Manager Is a Four-Letter Word</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#todayapostrophes_engineering_manager">Today’s Engineering Manager</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipatterns">Antipatterns</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_hire_pushovers">Antipattern: Hire Pushovers</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_ignore_low_performers">Antipattern: Ignore Low Performers</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_ignore_human_issues">Antipattern: Ignore Human Issues</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_be_everyoneapostrophes_frie">Antipattern: Be Everyone’s Friend</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_compromise_the_hiring_bar">Antipattern: Compromise the Hiring Bar</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#antipattern_treat_your_team_like_childr">Antipattern: Treat Your Team Like Children</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#positive_patterns">Positive Patterns</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#lose_the_ego-id00072">Lose the Ego</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#be_a_zen_master">Be a Zen Master</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#be_a_catalyst">Be a Catalyst</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#remove_roadblocks">Remove Roadblocks</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#be_a_teacher_and_a_mentor">Be a Teacher and a Mentor</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#set_clear_goals">Set Clear Goals</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#be_honest">Be Honest</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#track_happiness">Track Happiness</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#the_unexpected_question">The Unexpected Question</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch05.html#other_tips_and_tricks">Other Tips and Tricks</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch05.html#people_are_like_plants">People Are Like Plants</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#intrinsic_versus_extrinsic_motivation">Intrinsic Versus Extrinsic Motivation</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch05.html#conclusion-id00009">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch05.html#tlsemicolondrs">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch06.html">Leading at Scale</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#always_be_deciding">Always Be Deciding</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#the_parable_of_the_airplane">The Parable of the Airplane</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#identify_the_blinders">Identify the Blinders</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#identify_the_key_trade-offs">Identify the Key Trade-Offs</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#decidecomma_then_iterate">Decide, Then Iterate</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#always_be_leaving">Always Be Leaving</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#your_mission_build_a_quotation_markself">Your Mission: Build a “Self-Driving” Team</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#dividing_the_problem_space">Dividing the Problem Space</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#always_be_scaling">Always Be Scaling</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#the_cycle_of_success">The Cycle of Success</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#important_versus_urgent">Important Versus Urgent</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#learn_to_drop_balls">Learn to Drop Balls</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#protecting_your_energy">Protecting Your Energy</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch06.html#conclusion-id00010">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch06.html#tlsemicolondrs-id00106">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html">Measuring Engineering Productivity</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#why_should_we_measure_engineering_produ">Why Should We Measure Engineering Productivity?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#triage_is_it_even_worth_measuringquesti">Triage: Is It Even Worth Measuring?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#selecting_meaningful_metrics_with_goals">Selecting Meaningful Metrics with Goals and Signals</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#goals">Goals</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#signals">Signals</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#metrics">Metrics</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#using_data_to_validate_metrics">Using Data to Validate Metrics</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#taking_action_and_tracking_results">Taking Action and Tracking Results</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#conclusion-id00011">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch07.html#tlsemicolondrs-id00108">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/part3.html">Processes</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch08.html">Style Guides and Rules</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch08.html#why_have_rulesquestion_mark">Why Have Rules?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch08.html#creating_the_rules">Creating the Rules</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#guiding_principles">Guiding Principles</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#the_style_guide">The Style Guide</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#changing_the_rules">Changing the Rules</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#the_process">The Process</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#the_style_arbiters">The Style Arbiters</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#exceptions">Exceptions</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#guidance">Guidance</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch08.html#applying_the_rules">Applying the Rules</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#error_checkers">Error Checkers</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#code_formatters">Code Formatters</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch08.html#conclusion-id00012">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch08.html#tlsemicolondrs-id00109">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch09.html">Code Review</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch09.html#code_review_flow-id00005">Code Review Flow</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch09.html#how_code_review_works_at_google">How Code Review Works at Google</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch09.html#code_review_benefits">Code Review Benefits</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#code_correctness">Code Correctness</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#comprehension_of_code">Comprehension of Code</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#code_consistency">Code Consistency</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#psychological_and_cultural_benefits">Psychological and Cultural Benefits</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#knowledge_sharing-id00052">Knowledge Sharing</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#code_review_best_practices">Code Review Best Practices</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#be_polite_and_professional">Be Polite and Professional</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#write_small_changes">Write Small Changes</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#write_good_change_descriptions">Write Good Change Descriptions</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#keep_reviewers_to_a_minimum">Keep Reviewers to a Minimum</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#automate_where_possible">Automate Where Possible</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#types_of_code_reviews">Types of Code Reviews</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#greenfield_code_reviews">Greenfield Code Reviews</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#behavioral_changescomma_improvementscom">Behavioral Changes, Improvements, and Optimizations</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#bug_fixes_and_rollbacks">Bug Fixes and Rollbacks</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#refactorings_and_large-scale_changes">Refactorings and Large-Scale Changes</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch09.html#conclusion-id00013">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch09.html#tlsemicolondrs-id00110">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html">Documentation</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#what_qualifies_as_documentationquestion">What Qualifies as Documentation?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#why_is_documentation_neededquestion_mar">Why Is Documentation Needed?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#documentation_is_like_code">Documentation Is Like Code</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#know_your_audience">Know Your Audience</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#types_of_audiences">Types of Audiences</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#documentation_types">Documentation Types</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#reference_documentation">Reference Documentation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#design_docs">Design Docs</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#tutorials">Tutorials</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#conceptual_documentation">Conceptual Documentation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#landing_pages">Landing Pages</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#documentation_reviews">Documentation Reviews</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#documentation_philosophy">Documentation Philosophy</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#whocomma_whatcomma_whencomma_wherecomma">WHO, WHAT, WHEN, WHERE, and WHY</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#the_beginningcomma_middlecomma_and_end">The Beginning, Middle, and End</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#the_parameters_of_good_documentation">The Parameters of Good Documentation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#deprecating_documents">Deprecating Documents</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch10.html#when_do_you_need_technical_writersquest">When Do You Need Technical Writers?</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#conclusion-id00014">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch10.html#tlsemicolondrs-id00112">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch11.html">Testing Overview</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#why_do_we_write_testsquestion_mark">Why Do We Write Tests?</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#the_story_of_google_web_server">The Story of Google Web Server</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#testing_at_the_speed_of_modern_developm">Testing at the Speed of Modern Development</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#writecomma_runcomma_react">Write, Run, React</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#benefits_of_testing_code">Benefits of Testing Code</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#designing_a_test_suite">Designing a Test Suite</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#test_size">Test Size</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#test_scope">Test Scope</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#the_beyonceacutesemicolon_rule">The Beyoncé Rule</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#a_note_on_code_coverage">A Note on Code Coverage</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#testing_at_google_scale">Testing at Google Scale</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#the_pitfalls_of_a_large_test_suite">The Pitfalls of a Large Test Suite</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#history_of_testing_at_google">History of Testing at Google</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#orientation_classes">Orientation Classes</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#test_certified">Test Certified</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#testing_on_the_toilet">Testing on the Toilet</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#testing_culture_today">Testing Culture Today</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch11.html#the_limits_of_automated_testing">The Limits of Automated Testing</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch11.html#conclusion-id00015">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch11.html#tlsemicolondrs-id00113">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch12.html">Unit Testing</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch12.html#the_importance_of_maintainability">The Importance of Maintainability</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch12.html#preventing_brittle_tests">Preventing Brittle Tests</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#strive_for_unchanging_tests">Strive for Unchanging Tests</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#test_via_public_apis">Test via Public APIs</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#test_statecomma_not_interactions">Test State, Not Interactions</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#writing_clear_tests">Writing Clear Tests</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#make_your_tests_complete_and_concise">Make Your Tests Complete and Concise</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#test_behaviorscomma_not_methods">Test Behaviors, Not Methods</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#donapostrophet_put_logic_in_tests">Don’t Put Logic in Tests</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#write_clear_failure_messages">Write Clear Failure Messages</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#tests_and_code_sharing_dampcomma_not_dr">Tests and Code Sharing: DAMP, Not DRY</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#shared_values">Shared Values</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#shared_setup">Shared Setup</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#shared_helpers_and_validation">Shared Helpers and Validation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#defining_test_infrastructure">Defining Test Infrastructure</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch12.html#conclusion-id00016">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch12.html#tlsemicolondrs-id00114">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch13.html">Test Doubles</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch13.html#the_impact_of_test_doubles_on_software">The Impact of Test Doubles on Software Development</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch13.html#test_doubles_at_google">Test Doubles at Google</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch13.html#basic_concepts">Basic Concepts</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#an_example_test_double">An Example Test Double</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#seams">Seams</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#mocking_frameworks">Mocking Frameworks</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#techniques_for_using_test_doubles">Techniques for Using Test Doubles</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#faking-id00042">Faking</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#stubbing-id00089">Stubbing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#interaction_testing">Interaction Testing</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#real_implementations">Real Implementations</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#prefer_realism_over_isolation">Prefer Realism Over Isolation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#how_to_decide_when_to_use_a_real_implem">How to Decide When to Use a Real Implementation</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#faking-id00043">Faking</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#why_are_fakes_importantquestion_mark">Why Are Fakes Important?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#when_should_fakes_be_writtenquestion_ma">When Should Fakes Be Written?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#the_fidelity_of_fakes">The Fidelity of Fakes</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#fakes_should_be_tested">Fakes Should Be Tested</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#what_to_do_if_a_fake_is_not_available">What to Do If a Fake Is Not Available</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#stubbing-id00091">Stubbing</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#the_dangers_of_overusing_stubbing">The Dangers of Overusing Stubbing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#when_is_stubbing_appropriatequestion_ma">When Is Stubbing Appropriate?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#interaction_testing-id00047">Interaction Testing</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#prefer_state_testing_over_interaction_t">Prefer State Testing Over Interaction Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#when_is_interaction_testing_appropriate">When Is Interaction Testing Appropriate?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#best_practices_for_interaction_testing">Best Practices for Interaction Testing</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch13.html#conclusion-id00017">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch13.html#tlsemicolondrs-id00116">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch14.html">Larger Testing</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#what_are_larger_testsquestion_mark">What Are Larger Tests?</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#fidelity">Fidelity</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#common_gaps_in_unit_tests">Common Gaps in Unit Tests</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#why_not_have_larger_testsquestion_mark">Why Not Have Larger Tests?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#larger_tests_at_google">Larger Tests at Google</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#larger_tests_and_time">Larger Tests and Time</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#larger_tests_at_google_scale">Larger Tests at Google Scale</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#structure_of_a_large_test">Structure of a Large Test</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#the_system_under_test">The System Under Test</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#test_data">Test Data</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#verification">Verification</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#types_of_larger_tests">Types of Larger Tests</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#functional_testing_of_one_or_more_inter">Functional Testing of One or More Interacting Binaries</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#browser_and_device_testing">Browser and Device Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#performancecomma_loadcomma_and_stress_t">Performance, Load, and Stress testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#deployment_configuration_testing">Deployment Configuration Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#exploratory_testing">Exploratory Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#asolidusb_diff_regression_testing">A/B Diff Regression Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#uat">UAT</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#probers_and_canary_analysis">Probers and Canary Analysis</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#disaster_recovery_and_chaos_engineering">Disaster Recovery and Chaos Engineering</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#user_evaluation">User Evaluation</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#large_tests_and_the_developer_workflow">Large Tests and the Developer Workflow</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#authoring_large_tests">Authoring Large Tests</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#running_large_tests">Running Large Tests</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#owning_large_tests">Owning Large Tests</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch14.html#conclusion-id00018">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch14.html#tlsemicolondrs-id00118">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch15.html">Deprecation</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch15.html#why_deprecatequestion_mark">Why Deprecate?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch15.html#why_is_deprecation_so_hardquestion_mark">Why Is Deprecation So Hard?</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#deprecation_during_design">Deprecation During Design</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#types_of_deprecation">Types of Deprecation</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#advisory_deprecation">Advisory Deprecation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#compulsory_deprecation">Compulsory Deprecation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#deprecation_warnings">Deprecation Warnings</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#managing_the_deprecation_process">Managing the Deprecation Process</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#process_owners">Process Owners</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#milestones">Milestones</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#deprecation_tooling">Deprecation Tooling</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch15.html#conclusion-id00019">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch15.html#tlsemicolondrs-id00120">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/part4.html">Tools</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch16.html">Version Control and Branch Management</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#what_is_version_controlquestion_markone">What Is Version Control?</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#why_is_version_control_importantquestio">Why Is Version Control Important?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#centralized_vcs_versus_distributed_vcs">Centralized VCS Versus Distributed VCS</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#source_of_truth">Source of Truth</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#version_control_versus_dependency_manag">Version Control Versus Dependency Management</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#branch_management">Branch Management</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#work_in_progress_is_akin_to_a_branch">Work in Progress Is Akin to a Branch</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#dev_branches">Dev Branches</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#release_branches">Release Branches</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#version_control_at_google">Version Control at Google</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#one_version">One Version</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#scenario_multiple_available_versions">Scenario: Multiple Available Versions</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#the_quotation_markone_versionquotation">The “One-Version” Rule</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#left_parenthesisnearlyright_parenthesis">(Nearly) No Long-Lived Branches</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#what_about_release_branchesquestion_mar">What About Release Branches?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch16.html#monorepos">Monorepos</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch16.html#future_of_version_control">Future of Version Control</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch16.html#conclusion-id00020">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch16.html#tlsemicolondrs-id00122">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch17.html">Code Search</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch17.html#the_code_search_u">The Code Search UI</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch17.html#how_do_googlers_use_code_searchquestion">How Do Googlers Use Code Search?</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#wherequestion_mark">Where?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#whatquestion_mark">What?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#howquestion_mark">How?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#whyquestion_mark">Why?</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#who_and_whenquestion_mark">Who and When?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#why_a_separate_web_toolquestion_mark">Why a Separate Web Tool?</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#scale">Scale</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#zero_setup_global_code_view">Zero Setup Global Code View</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#specialization">Specialization</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#integration_with_other_developer_tools">Integration with Other Developer Tools</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#api_exposure">API Exposure</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#impact_of_scale_on_design">Impact of Scale on Design</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#search_query_latency">Search Query Latency</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#index_latency">Index Latency</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#googleapostrophes_implementation">Google’s Implementation</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#search_index">Search Index</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#ranking">Ranking</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#selected_trade-offs">Selected Trade-Offs</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#completeness_repository_at_head">Completeness: Repository at Head</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#completeness_all_versus_most-relevant_r">Completeness: All Versus Most-Relevant Results</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#completeness_head_versus_branches_versu">Completeness: Head Versus Branches Versus All History Versus Workspaces</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#expressiveness_token_versus_substring_v">Expressiveness: Token Versus Substring Versus Regex</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch17.html#conclusion-id00021">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch17.html#tlsemicolondrs-id00123">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch18.html">Build Systems and Build Philosophy</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch18.html#purpose_of_a_build_system">Purpose of a Build System</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch18.html#what_happens_without_a_build_systemques">What Happens Without a Build System?</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#but_all_i_need_is_a_compilerexclamation">But All I Need Is a Compiler!</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#shell_scripts_to_the_rescuequestion_mar">Shell Scripts to the Rescue?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#modern_build_systems">Modern Build Systems</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#itapostrophes_all_about_dependencies">It’s All About Dependencies</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#task-based_build_systems">Task-Based Build Systems</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#artifact-based_build_systems">Artifact-Based Build Systems</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#distributed_builds">Distributed Builds</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#timecomma_scalecomma_trade-offs">Time, Scale, Trade-Offs</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#dealing_with_modules_and_dependencies">Dealing with Modules and Dependencies</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#using_fine-grained_modules_and_the_oneo">Using Fine-Grained Modules and the 1:1:1 Rule</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#minimizing_module_visibility">Minimizing Module Visibility</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#managing_dependencies">Managing Dependencies</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch18.html#conclusion-id00022">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch18.html#tlsemicolondrs-id00124">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html">Critique: Google’s Code Review Tool</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html#code_review_tooling_principles">Code Review Tooling Principles</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html#code_review_flow">Code Review Flow</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#notifications">Notifications</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#stage_one_creating_a_change">Stage 1: Create a Change</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#diffing">Diffing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#analysis_results-id00001">Analysis Results</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#tight_tool_integration">Tight Tool Integration</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#stage_two_request_review">Stage 2: Request Review</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html#stage_three_and_four_understanding_and">Stages 3 and 4: Understanding and Commenting on a Change</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#commenting">Commenting</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#understanding_the_state_of_a_change">Understanding the State of a Change</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#stage_five_change_approvals_left_parent">Stage 5: Change Approvals (Scoring a Change)</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html#stage_six_commiting_a_change">Stage 6: Committing a Change</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#after_commit_tracking_history">After Commit: Tracking History</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch19.html#conclusion-id00023">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch19.html#tlsemicolondrs-id00125">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch20.html">Static Analysis</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#characteristics_of_effective_static_ana">Characteristics of Effective Static Analysis</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#scalability">Scalability</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#usability">Usability</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#key_lessons_in_making_static_analysis_w">Key Lessons in Making Static Analysis Work</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#focus_on_developer_happiness">Focus on Developer Happiness</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#make_static_analysis_a_part_of_the_core">Make Static Analysis a Part of the Core Developer Workflow</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#empower_users_to_contribute">Empower Users to Contribute</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#tricorder_googleapostrophes_static_anal">Tricorder: Google’s Static Analysis Platform</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#integrated_tools">Integrated Tools</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#integrated_feedback_channels">Integrated Feedback Channels</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#suggested_fixes">Suggested Fixes</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#per-project_customization">Per-Project Customization</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#presubmits">Presubmits</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#compiler_integration">Compiler Integration</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#analysis_while_editing_and_browsing_cod">Analysis While Editing and Browsing Code</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch20.html#conclusion-id00024">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch20.html#tlsemicolondrs-id00126">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch21.html">Dependency Management</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#why_is_dependency_management_so_difficu">Why Is Dependency Management So Difficult?</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#conflicting_requirements_and_diamond_de">Conflicting Requirements and Diamond Dependencies</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#importing_dependencies">Importing Dependencies</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#compatibility_promises">Compatibility Promises</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#considerations_when_importing">Considerations When Importing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#how_google_handles_importing_dependende">How Google Handles Importing Dependencies</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#dependency_managementcomma_in_theory">Dependency Management, In Theory</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#nothing_changes_left_parenthesisaka_the">Nothing Changes (aka The Static Dependency Model)</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#semantic_versioning">Semantic Versioning</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#bundled_distribution_models">Bundled Distribution Models</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#live_at_head">Live at Head</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#the_limitations_of_semver">The Limitations of SemVer</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#semver_might_overconstrain">SemVer Might Overconstrain</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#semver_might_overpromise">SemVer Might Overpromise</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#motivations-id00080">Motivations</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#minimum_version_selection">Minimum Version Selection</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#socomma_does_semver_workquestion_mark">So, Does SemVer Work?</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#dependency_management_with_infinite_res">Dependency Management with Infinite Resources</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#exporting_dependencies">Exporting Dependencies</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch21.html#conclusion-id00025">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch21.html#tlsemicolondrs-id00127">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch22.html">Large-Scale Changes</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch22.html#what_is_a_large-scale_changequestion_ma">What Is a Large-Scale Change?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch22.html#who_deals_with_lscsquestion_mark">Who Deals with LSCs?</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch22.html#barriers_to_atomic_changes">Barriers to Atomic Changes</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#technical_limitations">Technical Limitations</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#merge_conflicts">Merge Conflicts</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#no_haunted_graveyards">No Haunted Graveyards</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#heterogeneity">Heterogeneity</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#testing-id00096">Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#code_review">Code Review</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#lsc_infrastructure">LSC Infrastructure</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#policies_and_culture">Policies and Culture</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#codebase_insight">Codebase Insight</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#change_management">Change Management</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#testing">Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#language_support">Language Support</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#the_lsc_process">The LSC Process</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#authorization">Authorization</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#change_creation">Change Creation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#sharding_and_submitting">Sharding and Submitting</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#cleanup">Cleanup</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch22.html#conclusion-id00026">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch22.html#tlsemicolondrs-id00128">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch23.html">Continuous Integration</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#ci_concepts">CI Concepts</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#fast_feedback_loops">Fast Feedback Loops</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#automation">Automation</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#continuous_testing">Continuous Testing</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#ci_challenges">CI Challenges</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#hermetic_testing">Hermetic Testing</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#ci_at_google">CI at Google</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#ci_case_study_google_takeout">CI Case Study: Google Takeout</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#but_i_canapostrophet_afford_ci">But I Can’t Afford CI</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch23.html#conclusion-id00027">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch23.html#tlsemicolondrs-id00129">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html">Continuous Delivery</a></p> <ol> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#idioms_of_continuous_delivery_at_google">Idioms of Continuous Delivery at Google</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#velocity_is_a_team_sport_how_to_break_u">Velocity Is a Team Sport: How to Break Up a Deployment into Manageable Pieces</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#evaluating_changes_in_isolation_flag-gu">Evaluating Changes in Isolation: Flag-Guarding Features</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#striving_for_agility_setting_up_a_relea">Striving for Agility: Setting Up a Release Train</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch24.html#no_binary_is_perfect">No Binary Is Perfect</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch24.html#meet_your_release_deadline">Meet Your Release Deadline</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch24.html#quality_and_user-focus_ship_only_what_g">Quality and User-Focus: Ship Only What Gets Used</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#shifting_left_making_data-driven_decisi">Shifting Left: Making Data-Driven Decisions Earlier</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#changing_team_culture_building_discipli">Changing Team Culture: Building Discipline into Deployment</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#conclusion-id00029">Conclusion</a></p> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch24.html#tlsemicolondrs-id00130">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch25.html">Compute as a Service</a></p> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#taming_the_compute_environment">Taming the Compute Environment</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#automation_of_toil">Automation of Toil</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#containerization_and_multitenancy">Containerization and Multitenancy</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#summary">Summary</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#writing_software_for_managed_compute">Writing Software for Managed Compute</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#architecting_for_failure">Architecting for Failure</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#batch_versus_serving">Batch Versus Serving</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#managing_statetwosixthree">Managing State</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#connecting_to_a_service">Connecting to a Service</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#one-off_code">One-Off Code</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#caas_over_time_and_scale">CaaS Over Time and Scale</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#containers_as_an_abstraction">Containers as an Abstraction</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#one_service_to_rule_them_all">One Service to Rule Them All</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#submitted_configuration">Submitted Configuration</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#choosing_a_compute_service">Choosing a Compute Service</a> <ol> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#centralization_versus_customization">Centralization Versus Customization</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#level_of_abstraction_serverless">Level of Abstraction: Serverless</a></li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#public_versus_private">Public Versus Private</a></li> </ol> </li> <li><a href="https://abseil.io/resources/swe-book/html/ch25.html#conclusion-id00030">Conclusion</a></li> <li> <p><a href="https://abseil.io/resources/swe-book/html/ch25.html#tlsemicolondrs-id00131">TL;DRs</a></p> </li> </ol> </li> <li> <p><a href="https://abseil.io/resources/swe-book/html/part5.html">Conclusion</a></p> </li> </ul> https://tedneward.github.io/Research/reading/economist-a-z/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/economist-a-z/index.html <p><a href="https://www.economist.com/international-relations-a-to-z">Page</a></p> https://tedneward.github.io/Research/reading/development/technical-debt/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/technical-debt/index.html <p>Technical debt is the implied cost of rework caused by choosing an easier, quicker solution now instead of a better approach that would take longer. While it can speed up initial delivery, this "debt" slows down future development, increases costs, and lowers software quality over time, similar to financial debt. It can be incurred intentionally for short-term gains or unintentionally through poor practices, and includes issues like inadequate code, design, or documentation.</p> <p>Causes of technical debt:</p> <ul> <li>Intentional shortcuts: Making a quick decision to meet a deadline, even if a better but longer solution exists.</li> <li>Unintentional mistakes: Inexperienced developers, poor architectural choices, or insufficient knowledge can lead to debt.</li> <li>Insufficient documentation: Lack of documentation makes it difficult for others to understand and maintain the code.</li> <li>Lack of refactoring: Not regularly addressing and improving existing code can lead to it becoming outdated and difficult to manage.</li> <li>Rapid changes: Unforeseen or last-minute changes to features can introduce bugs and complexities that are not properly addressed.</li> <li>Pressure to minimize development time/schedule pressure</li> <li>Unexpected and ill-defined features and changes: The implementation of last-minute specification changes or changes that are insufficiently documented or tested</li> <li>Gaps in knowledge or skills. May manifest as a lack of process understanding, insufficient knowledge, poor technological leadership, or inadequate mentoring or knowledge sharing practices.</li> <li>Issues in the development process: Such as sub-optimal solutions, insufficient requirements (from process inefficiencies), conflicting requirements on parallel branches, deferred refactoring, or delayed upstream contributions.</li> <li>Non-compliance with best practice: Such as insufficient software documentation, poor collaboration practices, lack of ownership, rewrites for outsourced software, inadequate attention to code quality, tightly coupled components, lack of a test suite, or failure to align to standards (including ignoring industry standard frameworks).</li> </ul> <p>Fowler's thoughts:</p> <ul> <li><a href="https://martinfowler.com/bliki/TechnicalDebt.html">Technical Debt</a> <ul> <li><a href="https://martinfowler.com/bliki/TechnicalDebtQuadrant.html">Technical Debt Quadrant</a></li> </ul> </li> </ul> <h2>Thoughts</h2> <ul> <li>Capturing tribal and tacit knowledge - <a href="https://leaddev.com/software-quality/building-knowledge-transfer-strategy-manage-technical-debt">https://leaddev.com/software-quality/building-knowledge-transfer-strategy-manage-technical-debt</a></li> <li><a href="https://systemic.engineering/on-fragmentation-tech-debt-coherence/">On Fragmentation: Tech Debt Is a Coherence Problem</a></li> </ul> <h2>Tools/Tooling</h2> <ul> <li><a href="https://dev.to/elshadhu/i-built-a-tool-to-stop-wasting-time-on-toxic-open-source-projects-5h12">I Built a Tool to Stop Wasting Time on Toxic OSS Projects</a> <a href="https://github.com/ElshadHu/repo-health">Source</a>: My system uses a Hybrid Approach that combines a deterministic formula grounded in industry standards with a qualitative Language Model Judge to account for real-world context. The Algorithm (0-100 Score) is calculated using a custom weighted average that I designed, inspired by standardized CHAOSS Metrics. I tuned the weights myself based on what I believe indicates a healthy modern project: <pre><code>Score = (0.3 × Activity) + (0.25 × Maintenance) + (0.2 × Community) + (0.25 × Docs) </code></pre> <p>Activity (30%): Frequency of commits + Recency of updates + Unique authors.<br> Maintenance (25%): Issue response time + Open issue ratio + Repository age.<br> Community (20%): Logarithmic scale of Stars &amp; Forks.<br> Documentation (25%): Existence of README, LICENSE, and CONTRIBUTING files.</p> </li> </ul> <h3>High-level categories</h3> <ul> <li><strong>Requirement Debt:</strong></li> <li><strong>People Debt:</strong></li> <li><strong>Architectural debt:</strong> Issues in the software architecture, such as strongly coupled components, non-scalability of the components, and poor performance, are considered architectural debts. These debts mostly result from hurried decisions or lack of foresight.</li> <li><strong>Design debt:</strong> Design debt refers to all those UI or UX issues: inconsistency in styling, bad usability, and a lack of accessibility. It usually occurs when design decisions or designs are made hastily and without proper user research.</li> <li><strong>Process debt:</strong> stems from poor collaboration, unclear workflows and missing documentation, causing delays in feature delivery and increasing onboarding challenges. Companies that neglect agile methodologies or fail to integrate scrum principles often struggle with backlog accumulation, making it difficult to track and resolve issues efficiently.</li> <li><strong>Build Debt:</strong></li> <li><strong>Code debt:</strong> Code debt is the problem in the source code. It includes bad structuring, duplication, or a lack of comments. Common reasons for code debt include rapid developments or not correctly following the best practices for coding.</li> <li><strong>Testing debt:</strong> Testing debt means the need for more extensive and automated testing. When a project has testing debt, the software is unreliable, and there will be more manual testing activities. This might be because of rushed development or poor emphasis on testing.</li> <li><strong>Test Automation Debt:</strong> <em>(Could be seen as a subcategory of Testing Debt)</em></li> <li><strong>Defect Debt:</strong></li> <li><strong>Documentation debt:</strong> Missing or outdated documentation is commonly referred to as documentation debt, which makes it hard for a newcomer to the team to understand the codebase. It generally happens when development is rushed or when documentation is not prioritized.</li> <li><strong>Infrastructure Debt:</strong> Infrastructure debt refers to problems in the infrastructure itself, such as underpowered servers, unautomated tasks, and a lack of good monitoring. This debt generally results from hurried decisions about infrastructure or from maintaining infrastructure taking a backseat.</li> <li><strong>Service Debt:</strong></li> <li><strong>Security debt:</strong> Arises when teams cut corners on encryption, authentication or vulnerability patching, leaving software exposed to cyberthreats and compliance risks. A lack of automated security testing increases the burden on teams, making it more difficult to maintain secure systems. <em>(Could be seen as a subcategory of Infrastructure.)</em></li> </ul> <h3>Metrics</h3> <p>Tracking technical debt with metrics: Keeping track of technical debt is essential for maintaining the health and sustainability of a software project.</p> <ul> <li> <p>Documentation and record keeping</p> <ul> <li>Document Technical Debt: Whenever technical debt is incurred, document it. This includes the reasons for the debt, the expected impact, and any plans for resolution.</li> <li>Maintain a Technical Debt Register: Create a centralized document or tool where all instances of technical debt are recorded and tracked. This should include details like the nature of the debt, associated code or modules, and any deadlines for resolution.</li> </ul> </li> <li> <p>Code reviews and analysis:</p> <ul> <li>Regular Code Reviews: Implement a process where code is regularly reviewed by peers. This helps in identifying potential technical debt early.</li> <li>Static Code Analysis Tools: Use tools that automatically analyze code for potential issues, such as code smells, complexity, duplication, and other indicators of technical debt.</li> </ul> </li> <li> <p>Integration with Issue Tracking Systems:</p> <ul> <li>Link Debt to Issues: When technical debt results in bugs or issues, link these in your issue tracking system. This provides a clear connection between the debt and its consequences.</li> <li>Prioritize Debt Issues: Make sure that technical debt issues are prioritized alongside other work items in your backlog.</li> </ul> </li> <li> <p>Testing and Coverage Metrics:</p> <ul> <li>Monitor Test Coverage: Regularly review test coverage metrics to ensure that your codebase is adequately tested. Low test coverage can be an indicator of technical debt.</li> <li>Automated Testing: Implement automated testing to continuously monitor the health of your codebase and catch issues early.</li> </ul> </li> <li> <p>Regular Refactoring Sessions:</p> <ul> <li>Dedicated Refactoring Time: Allocate regular time slots for refactoring and addressing technical debt. This ensures that debt is actively being paid down.</li> <li>Refactor as You Go: Encourage developers to refactor and improve code as part of their regular development process, not just during dedicated sessions.</li> </ul> </li> <li> <p>Metrics and Dashboards:</p> <ul> <li>Use Metrics: Track key metrics to get a quantitative view of your technical debt.</li> <li>Dashboards: Create dashboards that visualize technical debt metrics. This helps in quickly assessing the health of the project and identifying areas that need attention.</li> </ul> </li> </ul> <hr> <h4>Tech Debt Index (TDI)</h4> <p><em>Measurement: Metric</em> (Goal will depend on how the metric is calcuated)</p> <p>A composite metric that combines various indicators of technical debt into a single index.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Combining various technical debt metrics (like code complexity, bug count, etc.) into a single index.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Provides a holistic view of technical debt in the project.</li> <li>Can be used to track changes in technical debt over time.</li> <li>Offers a high-level overview of technical debt, useful for management and stakeholder communication.</li> </ul> <p><strong><em>Risks:</em></strong></p> <p>The construction of the index can be subjective, depending on which metrics are included.<br> May oversimplify complex aspects of technical debt.</p> <hr> <h4>Tech Debt Ratio (TDR)</h4> <p><em>Goal: Low</em> <em>Measurement: percentage</em></p> <p>TDR is a measure of the cost to fix the technical debt relative to the size of the codebase. Use TDR to assess and communicate the overall health of the codebase to both technical and non-technical stakeholders.</p> <p><strong><em>Mechanism:</em></strong></p> <ol> <li><strong>Identifying components.</strong> Decide which parts of the application are to be evaluated for technical debt.</li> <li><strong>Code analysis.</strong> Use static code analysis tools suitable for the programming languages and frameworks used in the fintech app (e.g., ESLint for JavaScript, SonarQube for a range of languages).</li> <li><strong>Calculating remediation cost.</strong> Estimate the time required to address the code issues identified. Consider the complexity of the fintech environment, which might require specialized skills. Assume an average cost per developer hour specific to fintech development expertise (e.g., $80/hour). Example: If it takes 15 hours to refactor the complex code and update libraries, the cost is 15 hours * $80/hour = $1,200.</li> <li><strong>Calculating development cost.</strong> Estimate the total cost of developing the app or its specific modules, including design, coding, testing, deployment, and any other expenses. Example: The development cost of the investment portfolio manager might be $30,000.</li> <li><strong>Calculating Technical Debt Ratio.</strong> Technical Debt Ratio = (Remediation Cost / Development Cost) * 100<br> Calculation: For the investment portfolio manager, the ratio is ($1,200 / $30,000) * 100 = 4%.</li> <li><strong>Interpretation and action.</strong> Interpretation: A 4% technical debt ratio indicates a moderate level of debt. Action: Prioritize fixing critical issues, especially those related to security, data integrity, and regulatory compliance.</li> </ol> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Provides a quick snapshot of the quality of code being produced</li> <li>Provides a monetary perspective on the impact of technical debt</li> <li>Helps in prioritizing debt reduction based on cost</li> <li>A higher ratio suggests a significant portion of development effort goes into addressing technical debt</li> <li>Can be used to justify investment in reducing technical debt</li> </ul> <p><strong><em>Limitations:</em></strong></p> <ul> <li>Estimating the cost to fix issues can be subjective.</li> <li>Doesn't capture non-code aspects of technical debt (like architectural issues).</li> <li>Might not reflect the impact of non-quantifiable aspects of debt.</li> </ul> <hr> <h4>Defect count</h4> <p><em>Goal: Low</em> <em>Measurement: Instances of defects within a system</em></p> <p>This metric tracks the number of open bugs. High numbers can indicate underlying quality issues contributing to technical debt.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Use a defect tracker or other system (spreadsheet).</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Useful as overall guide to health of development.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Bug counts alone don't give a complete picture of the software's health.</li> </ul> <hr> <h4>Defect age</h4> <p><em>Goal: Low</em> <em>Measurement: Time per defect</em></p> <p>Bugs that have been open for a long time can indicate that technical debt is being neglected. This can lead to a deteriorating codebase and user experience.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Use a defect tracker or other system (spreadsheet). Timestamps must be present for each defect report.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Useful for prioritizing bug fixes.</li> <li>Helps prioritize bug fixes and identify areas needing improvement.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>The severity and impact of bugs vary; not all bugs contribute equally to technical debt.</li> </ul> <hr> <h4>Defect growth rate</h4> <p><em>Goal: Low, preferably negative</em> <em>Measurement: instance delta (number of new bugs reported versus the number of bugs that have been resolved or closed over a specific period)</em></p> <p><strong><em>Mechanism:</em></strong></p> <p>Track the number of new bugs reported and the number of bugs closed in your issue tracking or project management software over regular intervals (e.g., weekly, monthly).</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li><strong>Balance of Bug Creation and Resolution:</strong> A high number of new bugs compared to closed ones could indicate growing technical debt, as it suggests issues are being introduced faster than they are being resolved.</li> <li><strong>Software Stability:</strong> A sudden spike in new bugs might indicate problems introduced in recent updates, reflecting the impact of rushed development or insufficient testing, both of which can contribute to technical debt.</li> <li><strong>Project Health Monitoring:</strong> Regularly comparing new and closed bugs can help in assessing the health and progress of a project. A project with decreasing technical debt should ideally show a trend of decreasing new bugs or an increasing rate of bug closure.</li> <li><strong>Prioritization and Resource Allocation:</strong> This metric can help in prioritizing bug fixing and allocating resources effectively, which is crucial for managing and reducing technical debt.</li> <li>Helps in understanding the dynamics of bug occurrence and resolution in a project.</li> <li>Can signal when technical debt might be increasing, requiring attention to areas of the codebase that are generating new bugs.</li> <li>Assists in resource planning for bug fixing and technical debt reduction efforts.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li><strong>Not All Bugs are Equal:</strong> This metric doesn't account for the severity or complexity of the bugs. Therefore, the raw numbers might be misleading without the context of bug impact.</li> <li><strong>Quality of Bug Reporting:</strong> The reliability of this metric depends on the consistency and quality of bug reporting practices. Inconsistent reporting can skew the data.</li> <li><strong>Does Not Reflect Underlying Causes:</strong> While this metric can signal potential problems, it doesn't directly point to the causes of technical debt.</li> </ul> <hr> <h4>Defect ratio, Defect density</h4> <p><em>Goal: Low</em> <em>Measurement: number of defects in a software system relative to its size</em></p> <p>Helps in tracking the quality of the software over time, indicating whether technical debt is causing an increase in defects. Can guide decisions about where to focus development efforts, particularly in identifying areas of the codebase that may be contributing disproportionately to the defect count. Assists in prioritizing technical debt reduction strategies, especially when correlated with other metrics like code complexity or code churn.</p> <p>Applied to whole systems, this is essentially a bug count. Applied to individual files or components, it highlights (heatmaps) the 20% of the code where 80% of the defects live.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li><strong>Quality vs. Debt:</strong> While a high defect ratio directly points to quality issues, it indirectly suggests technical debt. For instance, rushed features, lack of testing, or poor design decisions can lead to both more defects and more debt.</li> <li><strong>Actionable Insights:</strong> Knowing the defect ratio can help teams prioritize refactoring, improve testing strategies, or revise development practices to reduce both defects and technical debt.</li> <li><strong>Indicator of Software Quality:</strong> A high defect ratio suggests lower quality, which could be a result of accumulated technical debt due to issues like rushed development or poor design.</li> <li><strong>Maintenance and Refactoring Needs:</strong> An increasing defect ratio over time can indicate that the software is becoming more difficult to maintain, suggesting the need for refactoring and addressing technical debt.<br> Project Health Assessment: It provides a quick snapshot of the current state of the project, helping to assess whether technical debt is impacting the software negatively.</li> </ul> <p><strong><em>Limitations:</em></strong></p> <ul> <li><strong>Not All-Encompassing:</strong> The defect ratio alone doesn't capture all aspects of technical debt, such as architectural issues or code complexity.</li> <li><strong>Quality of Defect Reporting:</strong> The metric's accuracy depends on the thoroughness and consistency of defect reporting practices.</li> <li><strong>Variability in Defect Severity:</strong> Not all defects have the same impact or require the same effort to fix, which this ratio doesn't account for.</li> </ul> <p><strong><em>Mechanism:</em></strong></p> <ul> <li>Count the total number of defects (reported bugs or issues) and divide it by a measure of software size (like lines of code, function points, or modules).</li> <li>Defect Ratio = Size of the Software/Number of Defects​</li> <li>Number of Defects is the total number of defects found in the software.</li> <li>Size of the Software could be measured in different units, such as lines of code, function points, or number of modules/components.</li> </ul> <hr> <h4>Cycle Time</h4> <p><em>Goal: Low</em> <em>Measurement: Time</em></p> <p>Measures the time it takes for a unit of work to be completed from start to finish. Unlike throughput, which measures the amount of work completed in a given time period, cycle time focuses on the amount of time. This metric is useful for understanding how long it takes to complete a specific task, and can be useful for identifying bottlenecks or inefficiencies in the development process.</p> <hr> <h4>Lead time for changes, Time to Market (TTM)</h4> <p><em>Goal: Low</em> <em>Measurement: Time</em></p> <p>Lead time for changes is the time it takes to go from code committed to code successfully running in production. TTM is typically customer-facing and at largerscale than a single feature. This metric is important for measuring the speed and agility of your software delivery process. Lead time takes into account the time it takes to plan, prioritize, and start work on a unit of work, as well as the time it takes to complete it. It can be useful for understanding how long it takes to deliver new features or improvements to customers.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Measure the duration from when a feature is planned to when it is available to users.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Longer lead times can be a symptom of high technical debt, as existing issues slow down new development.</li> <li>Helps in identifying bottlenecks in the development process.</li> <li>Longer time to market can indicate inefficiencies and potential technical debt.</li> <li>Striving for shorter time to market can sometimes lead to accruing more technical debt.</li> <li>If the team is able to quickly implement new features or fix bugs and get them into production, the organization can respond more quickly to changes in the market or user needs. This can lead to a competitive advantage.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Time to market is influenced by many factors, not just technical debt.</li> <li>Short time to market isn't always positive, especially if it leads to lower quality.</li> <li>Influenced by many factors beyond technical debt, such as team size and resource allocation.</li> <li>Not all delays are due to technical debt.</li> </ul> <hr> <h4>Code Churn</h4> <p><em>Goal: Low</em> <em>Measurement: percentage of a developer's own code that is recently edited (added, modified, deleted) after being written.</em></p> <p>Code churn can be indicative of indecision, lack of clarity, or misunderstanding – all of which contribute to technical debt. High code churn, especially late in a development cycle, can be a red flag that code may be less reliable or harder to maintain. High churn may indicate unstable or problematic areas in the codebase. Helps identify features or modules that are frequently changed and might be accumulating debt.</p> <p><strong><em>Mechanism:</em></strong></p> <ul> <li>Calculate the percentage of lines of code that have been changed, added, or deleted during a specific time frame.</li> </ul> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Identifying areas of frequent changes helps to pinpoint unstable code and prevent potential future issues related to constant modifications.</li> <li>Analyzing code churn alongside bug count can provide insights into whether the churn is constructive (refactoring, improving code) or destructive (introducing more defects).</li> <li>It helps in prioritizing areas for refactoring and understanding the impact of changes.</li> </ul> <p><strong><em>Limitations:</em></strong></p> <ul> <li>High churn isn’t always negative; it can reflect healthy iterative development.</li> <li>Doesn’t directly measure code quality.</li> </ul> <hr> <h4>Development Throughput, Velocity</h4> <p><em>Goal: High</em> <em>Measurement: Instances (stories, features, etc) within time</em></p> <p>Throughput is a way of measuring how much work your team is able to complete in a specific period of time. It's a way of tracking progress and understanding how well your processes are working. By keeping an eye on throughput, you can make changes to improve the efficiency of your team’s work. Tracking throughput is like keeping a pulse on the health of the development process.</p> <p><strong><em>Mechanism:</em></strong></p> <ol> <li><strong>Define the unit of work.</strong> The first step is to define what constitutes a unit of work in your development process. For example, you may decide that a unit of work is a completed user story, a fixed bug, or a feature that meets acceptance criteria. Whatever you choose, it's important that the definition of a unit of work is consistent and clear for everyone on the team.</li> <li><strong>Track progress.</strong> Next, track how many units of work are being completed in a given time period. For example, you might track the number of completed user stories over the course of a sprint, or the number of fixed bugs over the course of a week.</li> <li><strong>Calculate the throughput.</strong> Finally, calculate the throughput by dividing the number of units of work completed by the amount of time it took to complete them. For example, if your team completed 20 user stories in a two-week sprint, the throughput would be 20 user stories / 2 weeks. Important: You can’t measure a weekly throughput by dividing this number by 2. For that, you need to assess a week as a separate entity.</li> </ol> <p>Example: Development team of 5 people completed a total of 20 user stories in 4 weeks. The team's throughput would be 20 user stories / 4 weeks.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li><strong>Early access to potential efficiency problems.</strong> Throughput shows you if your process remains efficient. It’s the first indicator of something going wrong. Then, you can get deeper and look for detailed causes of this problem. By that, you’re continuously making sure you’re on the right path to deliver value to your customers in a timely manner and can prioritize tasks accordingly.</li> <li><strong>Improved processes.</strong> Measuring throughput helps you to identify problems early on and make changes before they become bigger issues. By tracking this metric, you can identify bottlenecks, optimize the workflow, and prioritize tasks effectively. After that, you can make changes to improve efficiency and reduce waste, which entails working in accordance with lean methodology.</li> <li><strong>Better resource allocation.</strong> Throughput helps to determine the amount of work that can be completed in a given time period, which in turn helps to allocate resources effectively.</li> <li><strong>Enhanced decision-making.</strong> By tracking this metric, you can make informed decisions that drive continuous improvement and help you meet your goals.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li><strong>Oversimplification.</strong> Focusing solely on throughput can lead to an oversimplification of the complex processes involved in software development. This can result in an incomplete understanding of performance and a failure to identify other key factors that impact development outcomes.</li> <li><strong>Misinterpretation.</strong> If not properly understood, throughput metrics can be misinterpreted, leading to incorrect conclusions and misguided decisions. The issue is that throughput metrics on their own only provide a small piece of information about the overall performance of a system. If they're looked at in isolation, without considering other factors, it's possible to come to incorrect conclusions. For example, if you see an increase in throughput, it could be due to an increase in the number of resources or an improvement in processes. If you make decisions based solely on this increase, you might be misled into thinking that you're making progress, when in fact the underlying issues causing problems in the system are still there.</li> <li><strong>Lack of reference to complexity and size of tasks.</strong> Throughput doesn’t incorporate two aspects: complexity and size of tasks. That way, a team still can process tasks of various types, and in that case throughput won’t give much insight into your process.</li> <li><strong>Incentivizing negative behaviors.</strong> If throughput becomes the primary focus, team members may be incentivized to prioritize speed over quality, leading to decreased productivity and increased errors.</li> <li><strong>Neglect of important factors.</strong> Focusing on throughput to the exclusion of other important factors, such as code quality or collaboration, can lead to an unbalanced approach to software development.</li> <li><strong>Over-reliance on metrics.</strong> Relying too heavily on throughput metrics can result in a lack of attention to other important indicators of performance, such as customer satisfaction and employee engagement.</li> <li><strong>Must still be conscious while assessing performance.</strong> Assessing the performance of individual developers is controversial. Throughput as a single source of truth also isn't good for that goal since it doesn’t refer to quality and complexity of tasks. That way, developers who complete a lot of small tasks and do it carelessly would have higher results than the ones who deliver the code of the highest quality and work with complex tasks.</li> </ul> <hr> <h4>Pull Request Size</h4> <p><em>Goal: Low</em> <em>Measurement: bytes, lines of code, # of files</em></p> <p>Pull request size refers to the extent of code changes introduced within a single pull request, typically measured by the number of files modified and lines of code added or removed.</p> <p>Measuring pull request size helps streamline code reviews, accelerate feedback cycles, and improve code quality by breaking down changes into manageable units and facilitating efficient collaboration among team members.</p> <hr> <h4>Code Duplication</h4> <p><em>Goal: Low</em> <em>Measurement: percentage</em></p> <p>Code duplication often leads to maintenance challenges. If a bug is found in a piece of duplicated code, it needs to be fixed in all instances. This increases the likelihood of missed defects and future issues.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Analyze the codebase using static code analysis tools to identify and quantify duplicated blocks of code.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>High duplication indicates poor code quality and higher maintenance costs.</li> <li>Points to potential areas for code reuse and refactoring.</li> <li>Reducing code duplication enhances maintainability and reduces the likelihood of introducing inconsistencies during updates.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Some duplication might be necessary or intentional.</li> <li>Doesn’t measure other quality aspects like code complexity.</li> </ul> <hr> <h4>Cyclomatic Complexity</h4> <p><em>Goal: Low</em> <em>Measurement: instances of linearly independent paths through a unit of code</em></p> <p>High cyclomatic complexity indicates that the code may be harder to test thoroughly and more prone to defects. It can also be harder to read and understand, increasing the likelihood of introducing errors during future changes. ‍Use this metric to identify complex code that may be harder to maintain and more prone to defects. Analyzing cyclomatic complexity at a method, class, and module level can help identify specific areas that may benefit most from refactoring.</p> <p><strong><em>Mechanism:</em></strong></p> <p>‍Use static analysis tools to calculate the number of independent paths through the code.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Higher complexity often indicates harder-to-maintain code, contributing to technical debt.</li> <li>Can help identify areas that need refactoring.</li> <li>Helps in estimating the effort required for testing and maintenance.</li> <li>Evaluate the intricacy and potential risks associated with modifications. This approach provides a deeper understanding of the impact on code quality and maintainability.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Doesn’t measure code readability or maintainability directly.</li> <li>Can be misleading for certain code structures.</li> <li>Some level of duplication might be unavoidable or even preferable in certain cases.</li> <li>Doesn't indicate other aspects of code quality.</li> </ul> <hr> <h4>Test Coverage</h4> <p><em>Goal: High</em> <em>Measurement: percentage over lines of code and/or files</em></p> <p>It measures the extent to which the modified code is covered by automated tests. Higher test coverage indicates more confidence in the changes and reduces the risk of regressions. Test coverage analysis helps ensure code quality and stability. While high test coverage is not a guarantee of code quality, inadequate test coverage can mean that the codebase is not well-protected against regressions, making it riskier to address technical debt through refactoring.</p> <p><strong><em>Mechanism:</em></strong></p> <p>Use testing tools and frameworks to calculate the percentage of code executed during testing.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>High test coverage can reduce the likelihood of bugs and technical debt.</li> <li>Indicates the robustness of testing practices.</li> <li>Low test coverage can lead to undiscovered bugs and increase technical debt.</li> <li>High coverage helps ensure code changes don't introduce new issues.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>High coverage doesn't guarantee high-quality tests or code.</li> <li>Can lead to a false sense of security if tests are not well-designed.</li> </ul> <hr> <h4>Deployment Frequency</h4> <p><em>Goal: High</em> <em>Measurement: Instances within time</em></p> <p>Deployment frequency is the number of times code is deployed to production in a given period of time. This metric is important for measuring the speed and agility of a software delivery process.</p> <p>For example, if your team is able to deploy code to production frequently, it can lead to a faster feedback loop and quicker response to issues or changes. This can lead to a better product and increased user satisfaction.</p> <p>A high deployment frequency indicates that teams are able to deliver changes quickly, responding to the needs of the business.</p> <hr> <h4>Changed Failure Rate</h4> <p><em>Goal: Low</em> <em>Measurement: percentage (of changes (e.g., deployments or code commits) that fail and require immediate remediation (like a hotfix or rollback))</em></p> <p><strong><em>Mechanism:</em></strong></p> <p>Tracking the success and failure rates of deployments and code changes.</p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>A high change failure rate can indicate poor code quality or inadequate testing, contributing to technical debt.</li> <li>Helps identify the stability and reliability of the development and deployment processes.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Some failures are normal and not indicative of technical debt.</li> <li>Doesn't capture the underlying causes of the failures.</li> </ul> <hr> <h4>Number of Failed CI/CD Events</h4> <p><em>Goal: Low</em> <em>Measurement: instances (number of times continuous integration (CI) or continuous deployment (CD) processes fail.)</em></p> <p><strong><em>Mechanism:</em></strong></p> <p><strong><em>Benefits:</em></strong></p> <ul> <li>Frequent CI/CD failures can indicate underlying issues in code quality, testing, or the build process, contributing to technical debt.</li> <li>Helps in identifying the reliability and efficiency of automated processes.</li> </ul> <p><strong><em>Risks:</em></strong></p> <ul> <li>Some failures might be due to external factors unrelated to technical debt.</li> <li>Doesn't provide insights into the nature of the issues causing the failures.</li> </ul> <hr> <h4>Mean time to recovery (MTTR)</h4> <p><em>Goal: Low</em> <em>Measurement: Time</em></p> <p>MTTR is the average time it takes to repair a system after a failure. This metric is important for measuring the reliability and availability of a system.</p> <p>For example, let's say your company has a web application that experiences a failure. The MTTR is the time it takes for your team to identify the failure and fix the issue. A low MTTR means your team is able to quickly detect and resolve issues, ensuring that your application is available to users.</p> <p>A low MTTR indicates a system that is reliable and easy to recover from failures.</p> <hr> <h4>Mean time between failures (MTBF)</h4> <p><em>Goal: High</em> <em>Measurement: Time</em></p> <p>MTBF is the average time between failures in a system. This metric is important for measuring the reliability and availability of a system.</p> <p>For example, if your web application has a high MTBF, it means that your users will experience fewer issues and downtime due to system failures. This can lead to increased user satisfaction and a better overall experience with your product.</p> <p>A high MTBF indicates a system that is reliable and has fewer failures.</p> <hr> <h4>Mean time to detect (MTTD)</h4> <p><em>Goal: Low</em> <em>Measurement: Time</em></p> <p>MTTD is the average time it takes to detect a failure in a system. This metric is important for measuring the effectiveness of monitoring and alerting systems.</p> <p>For example, if you have a monitoring system set up to detect issues with your web application, the MTTD is the time it takes for your team to receive an alert after a failure occurs. A low MTTD means your team can quickly identify and address issues, minimizing the impact on users.</p> <p>A low MTTD indicates that failures are detected quickly, allowing teams to respond and recover more quickly.</p> <hr> <h4>User value and impact</h4> <p><em>Goal: High</em> <em>Measurement: opinion/human analysis</em></p> <p>It involves considering factors like user feedback, customer requests, or business goals. By prioritizing value-driven changes, your team will ensure that their efforts align with the needs and expectations of stakeholders.</p> <hr> <h4>Peer review</h4> <p><em>Goal: High</em> <em>Measurement: opinion/human analysis</em></p> <p>Instead of relying solely on metrics, emphasize the feedback and insights provided during peer code reviews. Human judgment and expertise play a crucial role in evaluating code changes. Peer reviews provide an opportunity for knowledge sharing, identifying potential issues, and enhancing overall code quality.</p> <hr> <h2>Table of Tech Debt ideas</h2> <pre><code>Metric Name,Description,Category,Primary Source,Probability of Existing,Ease of Collection,Importance (Why track this?) 1. Cyclomatic Complexity,A quantitative measure of the number of linearly independent paths through a program's source code.,Engineering,SonarQube,High,Easy (Out of box),High complexity increases defect probability and makes testing/refactoring exponentially harder. 2. Cognitive Complexity,"A measure of how difficult a unit of code is to intuitively understand (unlike Cyclomatic, which measures structural logic).",Engineering,SonarQube,Medium,Easy (Out of box),"Directly correlates to ""time to read"" and onboarding costs for new developers." 3. Code Coverage,The percentage of code lines executed by automated unit/integration tests.,Engineering,SonarQube / JaCoCo,High,Easy (CI Integration),"Low coverage implies ""Blind Spots"" where regressions can occur without detection." 4. Code Duplication %,The percentage of code blocks that appear identical or nearly identical in multiple places.,Engineering,SonarQube,High,Easy (Out of box),Violates DRY (Don't Repeat Yourself); a bug fix in one spot must be manually replicated. 5. SQALE Rating (Debt Ratio),An aggregated rating (A-E) based on the estimated time required to fix all maintainability issues.,Engineering,SonarQube,Medium,Easy (Config needed),"Provides a high-level ""Credit Score"" for codebases that executives can easily understand." 6. Code Churn,"The volume of lines added, modified, or deleted over a specific time period.",Engineering,GitHub Ent.,High,Medium (Git mining),"High churn in legacy files often indicates ""Hotspots"" of instability or unclear requirements." 7. TODO/FIXME Count,The count of code comments explicitly flagged as temporary workarounds or deferred maintenance.,Engineering,SonarQube / Grep,High,Easy (Regex search),"A literal ""IOU"" list developers have left in the code; indicates rushed features." 8. Deployment Frequency,How often an organization successfully releases to production.,Delivery (DORA),Jenkins / GitHub,Medium,Medium (Log parsing),"Low frequency often implies brittle pipelines, manual gates, or fear of breaking production." 9. Lead Time for Changes,The amount of time it takes a commit to get into production.,Delivery (DORA),Jira / GitHub,Low,Hard (Correlating tools),"Long lead times indicate pipeline bottlenecks, slow code reviews, or manual testing debt." 10. Change Failure Rate,The percentage of deployments causing a failure in production.,Delivery (DORA),ServiceNow / Jira,Medium,Hard (Manual tagging),"High failure rates indicate ""Quality Debt""—lack of automated testing or reliable staging." 11. Mean Time to Recovery,How long it takes to restore service after a failure.,Delivery (DORA),PagerDuty,High,Medium (Incident tools),"High MTTR suggests ""Observability Debt""—it takes too long to diagnose and patch issues." 12. Flaky Test Ratio,The percentage of automated tests that fail and pass without code changes.,Delivery,CI Tool,Low,Hard (Log analysis),"Destroys trust in the CI/CD pipeline, causing developers to ignore red builds." 13. Build Duration,The time it takes for a full CI/CD pipeline to run.,Delivery,CI Tool,High,Easy (CI Logs),"Slow builds break flow state and discourage frequent commits, leading to larger riskier merges." 14. Library Freshness,A measurement of how many versions behind the current stable release your dependencies are.,Cyber / Platform,Snyk / Dependabot,Medium,Easy (SaaS Tools),Outdated libraries prevent using new features and eventually become security liabilities. 15. Critical/High CVEs,Count of known security vulnerabilities in code or dependencies.,Cyber,Snyk / Veracode,High,Easy (Scanners),"Immediate risk of breach; represents ""Security Debt"" that must be paid immediately." 16. Secret Leaks Count,"Number of hardcoded secrets (API keys, passwords) detected in the codebase.",Cyber,TruffleHog,Low,Medium (Scanners),"Indicates poor security practices and requires immediate, expensive rotation of credentials." 17. IAM Over-provisioning,Percentage of permissions granted vs. permissions actually used by services.,Cyber,AWS IAM Analyzer,Low,Hard (Cloud audit),"""Least Privilege"" debt; increases blast radius if a service is compromised." 18. Infrastructure Drift,The delta between the defined IaC (Terraform) and the actual state of the cloud environment.,Infrastructure,Terraform / Driftctl,Low,Medium (State diffs),"Indicates ""ClickOps"" (manual changes) are happening, making disaster recovery unreliable." 19. Cloud Asset Utilization,"Measure of provisioned CPU/RAM vs. actual usage (e.g., idle instances).",Infrastructure,AWS Cost Explorer,High,Easy (Cloud Native),Financial debt; paying for resources that aren't delivering value due to poor optimization. 20. Legacy OS Instances,Count of servers running on End-of-Life (EOL) Operating Systems.,Infrastructure,CMDB / Console,Medium,Medium (Inventory),High operational risk; EOL systems cannot be patched against new exploits. 21. IaC Coverage %,Percentage of cloud infrastructure managed via code vs. manual console creation.,Infrastructure,Terraform,Low,Hard (Manual Audit),Low coverage means environments cannot be easily replicated or restored. 22. Cyclic Dependencies,"Number of cycles between packages/modules (A depends on B, B depends on A).",Architecture,SonarQube,Low,Medium (Static Analysis),Makes code tightly coupled; you cannot change one module without breaking the other. 23. God Class Count,"Classes that exceed a high threshold of Lines of Code (e.g., &gt;2000 LOC) or methods.",Architecture,SonarQube,High,Easy (Static Analysis),"These classes know too much, are hard to test, and usually become the bottleneck for changes." 24. API Contract Breaking,Frequency of backward-incompatible changes to internal/external APIs.,Architecture,Pact / Swagger,Low,Hard (Diffing specs),High frequency breaks downstream consumers and forces unplanned refactoring work. 25. Afferent/Efferent Coupling,Measures stability (how many rely on you) vs. instability (how many you rely on).,Architecture,ArchUnit,Low,Medium (Tooling req.),High coupling makes the architecture rigid; changing one component ripples through the system. 26. Monolith Size (LOC),Total Lines of Code in a single deployable unit (if microservices strategy is desired).,Architecture,SonarQube / Git,High,Easy (LOC Count),"If the goal is decoupling, a growing monolith represents negative architectural progress." 27. Bus Factor,The minimum number of developers required to hit by a bus before the project stalls.,Process,Git Analytics,Low,Hard (Algo required),"Low bus factor (e.g., 1) indicates knowledge silos and ""Knowledge Debt.""" 28. Defect Backlog Age,The average time known non-critical bugs sit in the backlog.,Product,Jira,High,Medium (JQL Query),"Old bugs are rarely fixed; they clutter the view and represent ""Product Debt.""" 29. Onboarding Time,"Time from ""Day 1"" to ""First PR Merged"" for a new engineer.",Process,HR + GitHub,Low,Hard (Manual data),Proxy for documentation quality and environment complexity. Long onboarding = high complexity. 30. DB Schema Version Lag,Difference between production schema version and migration scripts.,Data,Flyway,Medium,Medium (DB Tools),Indicates manual DB patches or dangerous divergence between code and database state. 31. Dead Code / Unused Tables,Methods or DB tables that have zero usage references.,Data,Sonar / Logs,Medium,Hard (Usage analysis),Clutter that confuses developers and wastes backup/storage resources. 32. Documentation Staleness,Average age of files in the /docs or Wiki compared to current date.,Process,Confluence,Medium,Medium (API Metadata),Outdated docs are worse than no docs; they mislead engineers and cause outages. 33. PR Review Time,Average time elapsed between a Pull Request being opened and merged/closed.,Process,GitHub,High,Easy (API available),Long review times block value delivery and increase merge conflicts (Process Debt). 34. Branch Lifespan,The duration a feature branch exists before being merged.,Delivery,GitHub / Git,High,Easy (Git logs),"Long-lived branches drift from main, leading to ""Merge Hell"" and integration debt." 35. Merge Conflict Rate,Percentage of PRs that require manual conflict resolution.,Delivery,GitHub / Git,Low,Medium (Git stats),High rates indicate poor communication or architectural coupling. 36. Test Execution Time,Total time to run the full test suite (local or CI).,Delivery,CI Tool,High,Easy (CI Timestamps),"If tests take 1 hour, devs won't run them locally, leading to ""Feedback Loop Debt.""" 37. Environment Parity Gap,Count of configuration differences between Staging and Production.,Infrastructure,Terraform,Low,Hard (Deep diffs),"""It worked on my machine"" syndrome; parity gaps cause production-only bugs." 38. Untagged Cloud Assets,Percentage of cloud assets missing cost allocation/owner tags.,Infrastructure,AWS Config,High,Medium (Policy scan),"""FinOps Debt""—impossible to attribute costs or identify owners of zombie resources." 39. Orphaned Storage,Count/Size of EBS volumes or snapshots not attached to any compute instance.,Infrastructure,Cloud Custodian,High,Easy (API Query),Pure waste; paying for storage that is technically disconnected from the application. 40. Container Image Size,Size of the Docker/Container images being deployed.,Platform,Artifactory,High,Easy (Registry API),Bloated images slow down scaling (autoscaling latency) and increase vulnerability surface area. 41. Pod Restart Rate,"Frequency of containers crashing and restarting (OOMKilled, etc.).",Platform,Prometheus,High,Easy (Metrics),"Indicates instability, memory leaks, or improper resource limits (Configuration Debt)." 42. Serverless Cold Starts,Average latency added when a function scales from zero.,Platform,Datadog,Medium,Medium (APM),High latency here indicates poor optimization of runtimes or dependencies. 43. License Risk,"Count of dependencies with restrictive (e.g., GPL) or unknown licenses.",Legal,FOSSA,Medium,Easy (Scanners),"""Legal Debt""—risk of having to open-source proprietary code or face lawsuits." 44. TLS Protocol Lag,Percentage of endpoints supporting deprecated protocols (TLS 1.0/1.1).,Cyber,Qualys,Medium,Medium (Net scan),Security compliance debt; modern browsers/clients will eventually reject connections. 45. Missing Security Headers,"Endpoints missing standard headers (HSTS, CSP, X-Frame-Options).",Cyber,OWASP ZAP,Medium,Easy (Curl/Scan),"Low-hanging fruit for attackers; indicates lack of ""Security by Design.""" 46. Dependency Tree Depth,The average depth of transitive dependencies in the project.,Architecture,Maven / npm,Low,Medium (Graph analysis),Deep trees make vulnerability patching difficult (you rely on a library that relies on a library...). 47. Layer Violations,"Instances where lower layers call upper layers (e.g., Domain layer calling UI).",Architecture,ArchUnit,Low,Hard (Custom rules),Breaks separation of concerns; creates spaghetti code that is hard to refactor. 48. Interface Segregation,Classes forced to implement methods they do not use (bloated interfaces).,Architecture,SonarQube,Low,Medium (Static Analysis),Indicates poor abstraction; makes mocking and testing specific behaviors difficult. 49. Slow Query Ratio,Percentage of database queries exceeding a specific latency threshold.,Data,DB Insights,High,Medium (DB Logs),"""Performance Debt""—often due to missing indexes or N+1 query problems." 50. Data Null Rate,Percentage of unexpected nulls or format errors in critical data columns.,Data,dbt,Low,Hard (Data testing),"""Data Trust Debt""—if data is dirty, downstream analytics and ML models are worthless." 51. ETL Failure Rate,Frequency of data pipeline job failures requiring manual retry.,Data,Airflow,High,Easy (Orchestrator),High failure rates indicate fragile data ingestion and lack of idempotency. 52. Core Web Vitals,"Scores for LCP (Loading), FID (Interactivity), CLS (Visual Stability).",Engineering,Lighthouse,Medium,Easy (CI or Browser),"""Frontend Debt""—poor scores hurt SEO and user retention." 53. Accessibility Violations,"Count of automated accessibility errors (missing alt tags, contrast).",Engineering,axe-core,Low,Easy (Scanner),Legal and ethical debt; excludes users with disabilities and risks lawsuits. 54. App Binary Size,Total size of the IPA/APK downloaded by users.,Mobile,Store Connect,High,Easy (Store Metadata),"Bloat leads to lower install conversion rates, especially in low-bandwidth markets." 55. Crash-Free Sessions,Percentage of mobile/web sessions that do not end in a crash.,Mobile,Crashlytics,High,Easy (SDK),The ultimate measure of stability; low rates destroy brand reputation. 56. API 5xx Error Rate,Percentage of server-side errors returned to clients.,Platform,Load Balancer,High,Easy (Logs),Indicates unhandled exceptions and poor error management in backend services. 57. Log Coverage %,Percentage of critical business paths that emit traceable logs.,Platform,Splunk / ELK,Low,Hard (Manual Audit),"""Observability Debt""—if you can't see it, you can't debug it during an outage." 58. Mean Time To Detection,Average time between an issue starting and an alert firing.,Delivery,PagerDuty,Low,Hard (Incident Review),"If MTTD is high, your customers are your monitoring system." 59. Sprint Spillover %,Percentage of story points committed vs. completed in a sprint.,Process,Jira,High,Medium (Jira Reports),"High spillover indicates ""Planning Debt""—poor estimation or unclear requirements." 60. Meeting Load,Average hours per day engineers spend in meetings vs. coding blocks.,Process,Calendar API,Low,Medium (API Integration),"""Focus Debt""—engineers cannot pay down technical debt if they have no deep work time." </code></pre> <hr> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li> <p>"Towards an Ontology of Terms on Technical Debt": "Technical debt is a term that has been used to describe the increased cost of changing or maintaining a system due to shortcuts taken during its development. As technical debt is a recent research area, its different types and their indicators are not organized yet. Therefore, this paper proposes an ontology of terms on technical debt in order to organize a common vocabulary for the area. The organized concepts derived from the results of a systematic literature mapping. The proposed ontology was evaluated in two steps. In the first one, some ontology design quality criteria were used. For the second one, a specialist in the area performed an initial evaluation. This work contributes to evolve the Technical Debt Landscape through the organization of the different types of technical debt and their indicators. We consider this an important contribution for both researchers and practitioners because this information was spread out in the literature hindering their use in research and development activities." <a href="https://www.researchgate.net/publication/286010286_Towards_an_Ontology_of_Terms_on_Technical_Debt">ResearchGate</a> <a href="/reading/development/Towards_an_Ontology_of_Terms_on_Technical_Debt.pdf">PDF</a> <a href="https://pdfs.semanticscholar.org/3ab6/bd6fd72110a18f2d9d442cab03369a6017c5.pdf">slides</a></p> </li> <li> <p>Martin Fowler:</p> <ul> <li><a href="https://martinfowler.com/bliki/TechnicalDebt.html">Technical Debt</a></li> <li><a href="https://martinfowler.com/bliki/TechnicalDebtQuadrant.html">Technical Debt Quadrant</a></li> </ul> </li> <li> <p>SQALE (<a href="https://en.wikipedia.org/wiki/SQALE">Wikipedia</a>): (Software Quality Assessment based on Lifecycle Expectations) is a method to support the evaluation of a software application source code. It is a generic method, independent of the language and source code analysis tools. The method is based on 4 main concepts: The quality model; The analysis model; The indices; The indicators. <em>(Website appears down--Wikipedia links go to 404s)</em></p> </li> <li> <p>LeadDev:</p> <ul> <li><a href="https://leaddev.com/technical-direction/what-leaders-wrong-about-latent-cost-technical-debt">What leaders get wrong about latent cost of technical debt</a>: <strong>Problem:</strong> "Accruing technical debt isn’t necessarily a negative issue, and usually starts with good intentions. For instance, you may have made the call to accept a tech debt trade-off with the view to finishing a feature or meeting a deadline. But what happens when this becomes the norm – an ingrained component of the culture? I have seen the process of teams normalizing broken tests, sluggish builds, and tribal knowledge to the point that no one asks questions about it anymore. The longer this debt remains unaddressed, the more it interferes with the velocity of development and cognitive load, demotivating team members." ... Operational instability: "Technical debt is not always about messy code; it is instability that sneaks into operations. Even the slightest quality problem in a regulated industry can turn into an existential threat. For example, an FAA audit conducted on Boeing found that the airplane company failed 33 out of 89 quality audits of their 737 Max production, because they were not following approved manufacturing processes. What is the cause of most of these failures? Missing documentation, haphazard tooling, and a lack of process; technical debt at scale. Similar to the weak APIs and undocumented programs in our world, these were not merely bugs, but time bombs that had systemic implications. The parallel fragility in software teams is that the <a href="https://leaddev.com/reporting/solving-mean-time-repair-problem">mean time to recovery</a> (MTTR) is slower, and more often, rollbacks or on-call escalations are required. You may not be building planes, but when a service fails and no one understands who owns the fallback logic or how it is tested, you are closer to the risk than you imagine." ... The silencing effect of accumulated complexity: "Systems are bound to become more complex as they evolve. However, when technical debt accumulates without oversight, such as a <em>lack of documentation</em>, <em>unclear ownership</em>, or <em>postponed refactoring</em>, it accelerates complexity with no structural checks in place. When I was working in a fintech company, we learned that more than 40% of their microservices lacked identifiable owners, alongside the fact that uneven test coverage was rampant. Although the engineering department had grown at a high rate, no one bothered to assess the structural debt being incurred; issues such as tightly coupled services, legacy monoliths with hardcoded integrations, or ownership gaps that made critical systems unmaintainable. These findings illustrated how entrenched silence was in the team culture. Engineers cease to raise issues since this is how things are. New employees do not challenge inconsistencies since they presume it is deliberate. This normalization is what makes technical debt so dangerous; it becomes unseen, but highly influential." ... Strategic cost isn’t just financial: "In addition to operational anarchy, debt constrains strategic options. It traps teams in unstable architectures, makes experimentation less desirable, and change more expensive. ... The moral is obvious: technical debt minimizes optionality. It denies organizations the flexibility to react to threats or innovate promptly when needed." <strong>Approach:</strong> Start with full-scope visibility: "[R]unning engineering surveys, conducting service maturity audits, and analyzing operational metrics. These efforts often revealed debt artifacts like unowned scripts, deprecated libraries, or undocumented APIs – elements that rarely show up in standard project tracking tools. Without a structured inventory like this, teams often focus their efforts on the most obvious pain points, such as slow tests or deployment delays, rather than the most strategically important ones. Full-scope visibility means going beyond surface issues to identify and document what’s genuinely slowing down delivery, scaling, or incident response. A more modern strategy for understanding the scope of your tech debt issue incorporates telemetry-driven scans. These will be able to surface broken pipelines and flaky tests. It’s also important to gather qualitative feedback: developer pain points, support tickets, and onboarding feedback. If new engineers repeatedly encounter setup failures or unclear integration steps with a specific legacy module during onboarding, that module is a visibility gap. It’s not just a one-time inconvenience; it reflects debt that directly affects developer experience and onboarding velocity. These recurring issues should be logged and scored, as they indicate systemic friction with measurable impact." Score by impact, not just frustration: "Not every debt is alike. An abandoned configuration file is an inconvenience to engineers, but a closely-coupled authentication system that drags every product update has significantly steeper consequences. I suggest a light scoring model that is determined by three factors: <em>Severity</em>: What is the downstream risk of this debt going unaddressed? <em>Frequency</em>: How frequently does it create issues? <em>Strategic impact</em>: Does the debt limit your ability to scale systems, like handling more users, data, or teams? Does it impede your ability to adapt your product direction, e.g., shift to a new architecture, integrate with new services, or launch a different feature? With a simple scoring system (e.g., 1-5), you will have a shared language to compare debts between teams and make decisions on what to work on first. Debt isn't solved by scope; it's solved by relevance." Choose the right fix: Refactor, replace, or bypass: <em>Refactor</em> when the debt compounds daily costs; things like developer frustration, poor test coverage, or sluggish performance. For instance, in one of our back-end services, a shared utility function was frequently modified and regularly broke downstream dependencies. A simple refactor to isolate concerns reduced change failure rates by over 30% in just two sprints. <em>Replace</em> when you’re scaling past its original intent, e.g., hardcoded workflows or in-memory stores. At a previous role, our real-time analytics relied on an in-memory store that had no sharding or durability guarantees. It worked at launch, but as our usage scaled 10x, data loss and throttling became common. We replaced it with a distributed store designed for high throughput and persistence. <em>Bypass</em> when the effort-to-impact ratio is too high, fix only what’s necessary, and document the rest. One team I worked with had a legacy admin portal with hardcoded permissions logic. Rewriting it would have taken months, but it was rarely used. We documented its quirks, added a banner to warn users of limitations, and created a wrapper for the one feature it still supported.<br> The lesson: don’t assume all tech debt deserves your best engineering. Sometimes, clarity and containment are more valuable than cleanup." Accountability is a team sport: "The ownership of the debt cannot rest with tech leads alone. Teams require incentives and rituals, such as quarterly debt review, shared dashboards, and making ownership based on service health scores. An organization I consulted with tied debt scores directly to performance reviews at the top management level, not as a stick, but as an indication that quality was not a choice. Within two quarters, they saw a 25% increase in resolved debt items and a measurable drop in incident frequency across critical systems, indicating that visibility and ownership alone can drive behavior change."</li> </ul> </li> <li> <p><a href="https://brainhub.eu/library">BrainHub</a>:</p> <ul> <li> <p><a href="https://brainhub.eu/library/tech-debt-puzzle">Solving the Tech Debt Puzzle</a>: A case study (Express/JS). "Approach:</p> <ul> <li>Performing a tech audit to highlight the problems, risks, and start to raise awareness.</li> <li>Connecting the technical challenges with business priorities.</li> <li>Establishing a testing strategy (unit tests didn’t fit, Testing Trophy was a solution).</li> <li>Adopting a structured approach that leverages testing, refactoring, and careful planning.</li> <li>Encapsulating legacy code within the existing system and developing new features in the Nest framework.</li> <li>Making incremental improvements through minor rewrites.</li> </ul> </li> <li> <p><a href="https://brainhub.eu/library/cost-of-technical-debt">Business Costs of Technical Debt</a></p> </li> <li><a href="https://brainhub.eu/library/refactoring-legacy-code-strategy">Refactoring Legacy Code</a></li> <li><a href="https://brainhub.eu/library/refactoring-best-practices">Top 7 Refactoring Techniques</a></li> <li><a href="https://brainhub.eu/library/technical-debt-in-microservices">Technical Debt in Microservices</a></li> <li><a href="https://brainhub.eu/library/prioritizing-technical-debt-repayment">Technical Debt Repayment</a></li> <li><a href="https://brainhub.eu/library/avoid-technical-bankruptcy-tips">How to Avoid Technical Bankruptcy</a></li> <li><a href="https://brainhub.eu/library/technical-debt-metrics">Tech Debt Metrics</a></li> <li><a href="https://brainhub.eu/library/technical-debt-ratio">Tech Debt Ratio</a></li> <li><a href="https://brainhub.eu/library/how-to-deal-with-technical-debt">Managing Tech Debt</a></li> <li><a href="https://brainhub.eu/library/tools-to-measure-technical-debt">Tools to Measure Tech Debt</a></li> <li><a href="https://brainhub.eu/library/technical-debt-examples">Examples</a></li> <li><a href="https://brainhub.eu/library/application-modernization-challenges">Application Modernization Challenges</a></li> <li><a href="https://brainhub.eu/library/legacy-app-modernization-roadmap">Legacy Application Modernization Roadmap - a Step-by-Step Guide</a></li> <li><a href="https://brainhub.eu/library/legacy-systems-examples">Legacy Systems Examples</a></li> <li><a href="https://brainhub.eu/library/legacy-application-migration-to-cloud">Legacy Application Migration to Cloud</a></li> <li><a href="https://brainhub.eu/library/cost-of-technical-debt">Business Costs of Technical Debt</a></li> <li><a href="https://brainhub.eu/library/red-green-refactor-legacy-systems">Red Green Refactor</a></li> <li><a href="https://brainhub.eu/library/strangler-pattern-legacy-modernization">The Strangler Pattern for Legacy Modernization</a></li> <li><a href="https://brainhub.eu/library/monolith-to-microservices-using-strangler-pattern">Monolith to Microservices Using Strangler Pattern</a></li> <li><a href="https://brainhub.eu/library/big-bang-migration-vs-trickle-migration">Big Bang Migration vs Trickle Migration</a></li> <li><a href="https://brainhub.eu/library/top-data-migration-tools">Top 15 Data Migration Tools</a></li> <li><a href="https://brainhub.eu/library/data-migration-challenges-risks-legacy-modernization">Data Migration Challenges</a></li> <li><a href="https://brainhub.eu/library/data-migration-strategy-legacy-app">Data Migration Strategy for a Legacy App</a></li> <li><a href="https://brainhub.eu/library/microservices-vs-monolith-which-to-use-when">Microservices vs Monolith</a></li> <li><a href="https://brainhub.eu/library/monolith-better-than-microservices">When Monolith Is Better Than Microservices</a></li> <li><a href="https://brainhub.eu/library/modular-monolith-architecture">Modular Monolith – When to Choose It</a></li> <li><a href="https://brainhub.eu/library/master-data-management-strategy">Master Data Management Strategy</a></li> <li><a href="https://brainhub.eu/library/cloud-migration-challenges">Cloud Migration Challenges</a></li> <li><a href="https://brainhub.eu/library/mean-time-between-failures-mtbf">Mean Time Between Failures</a></li> <li><a href="https://brainhub.eu/library/how-to-measure-lead-time-for-changes">Lead Time for Changes</a></li> <li><a href="https://brainhub.eu/library/improve-software-delivery-speed">Software Delivery Speed</a></li> <li><a href="https://brainhub.eu/library/how-to-measure-deployment-frequency">How to Measure and Improve Deployment Frequency</a></li> <li><a href="https://brainhub.eu/library/change-failure-rate-best-practices">Change Failure Rate</a></li> <li><a href="https://brainhub.eu/library/mean-time-to-detect-mttd-explained">Mean Time to Detect (MTTD)</a></li> <li><a href="https://brainhub.eu/library/pull-request-size-best-practices">Pull Request Size</a></li> </ul> </li> <li> <p><a href="https://fastwonderblog.com/2025/10/23/assessing-the-viability-of-open-source-projects/">Assessing the Viability of Open Source Projects</a>: "Open source software is found in almost every codebase, but some open source projects are more viable than others over the long term. Many companies don’t have a rigorous process for selecting the most viable dependencies. Often product teams, or even individual software developers, select open source projects because they fill a particular technical need without any assessment of the viability of the project or the risks they might be taking by using it. Assessing the viability of open source projects, especially ones that have the potential to impact your business, is a good first step toward managing risk and reducing the chances of potential business disruptions."</p> </li> </ul> https://tedneward.github.io/Research/reading/how-to-read-a-book/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/how-to-read-a-book/index.html <p><em>Some notes derived from <a href="https://fs.blog/how-to-read-a-book/">https://fs.blog/how-to-read-a-book/</a></em></p> <p><a href="https://www.amazon.com/How-Read-Book-Classic-Intelligent/dp/0671212095">Book (Amazon)</a></p> <h3>The Four Levels of Reading</h3> <ul> <li>Elementary Reading (reading to entertain) - elementary school</li> <li>Inspectional Reading (reading to inform) - high school/college</li> <li>Analytical Reading (reading to understand)</li> <li>Syntopical Reading (reading to master)</li> </ul> <p>If you don’t learn the last two levels, you’ll go through life like a one-legged man in an ass-kicking contest.</p> <p>This doesn’t mean that everything should be read the same way. It shouldn’t. Why you are reading should match how you are reading. Your level of effort needs to match the importance of the material. If you read a romance novel and legal document the same way, you’re in for a life of misery.</p> <h3>Reading to Entertain</h3> <p>This is the level of reading taught in our elementary schools. If you’re reading this website, you already know how.</p> <h3>Reading to Inform</h3> <p>Some people think skimming and reading quickly is a bad idea. It’s not. The point of a quick skim is to determine if the book is worth reading deeply in the first place. Most are not.</p> <p>There are two kinds of inspectional reading:</p> <ul> <li>Systematic skimming: Quickly skim the book. Read the introduction, table of contents, index, and inside cover. This shows you the key chapters and ideas. Jump in and out, reading a paragraph here and there.</li> <li>Superficial reading: This is a quick read. Don’t stop to think deeply. Don’t write things in the margin. Don’t look things up; keep going if you don’t understand something. Read most of the book, but read it quickly.</li> </ul> <p>At this point, you have the main idea. Often, that’s enough. Think about all the pop-psychology books. Simply skimming the introduction gives you 90% of the value.</p> <p>Inspectional reading gives you the structure and an overview of the author’s arguments and opinions. Because you didn’t read deeply, you won’t understand all the nuances, but you’ll get the gist. Most times, you can stop right here. Put the book down and go on to the next one. But sometimes, you’ll want to know more about the book.</p> <h3>Reading to Understand</h3> <p>Francis Bacon once remarked, “some books are to be tasted, others to be swallowed, and some few to be chewed and digested.” You can think of analytical reading as doing that chewing and digesting. This is the real work of reading. If inspectional reading is the best you can do quickly, analytical reading is the best you can do with time. At this point, you start to engage your mind and do the work required to understand what’s being said. I recommend writing in your book and using our Blank Sheet notetaking method.</p> <p>There are four rules to Analytical Reading</p> <ul> <li>Classify the book according to its kind and subject matter</li> <li>State what the whole book is about with the utmost brevity</li> <li>Enumerate its major parts in their order and relation, and outline these parts</li> <li>Define the problem or problems the author is trying to solve</li> </ul> <p>You’ll probably notice that while those sound pretty easy, they involve a lot of work. Luckily, the inspectional reading you did earlier has primed you for this and made it easier. If you are reading to understand, you need to integrate the ideas into your head. You do the work of digesting the material and making it your own. Done correctly, this helps you understand the book. That does not mean, however, that you understand the broader subject. You can’t simply read one book or article and claim to be an expert. If you really want to master the subject, you need to dive into the subject entirely reading multiple books and comparing and contrasting them. That’s where syntopical reading comes in.</p> <h3>Reading to Master</h3> <p>Syntopical reading is the most demanding and challenging reading of all. Reading to master something involves reading many books on the same subject and comparing and contrasting ideas, vocabulary, and arguments. To do this, you must identify relevant passages, translate terminology, define the issues, explore issues through multiple lenses, and frame questions that need answering. The goal is not to achieve an overall understanding of any particular book but rather to understand the broader subject and develop deep fluency. Doing this means identifying and filling in your blind spots about the subject,</p> <p>When reading to master something, there are four keys to keep in mind</p> <ul> <li>Identify the relevant passages: Find the right books and the key passages. You can’t simply outsource this to others. If you want to address your blind spots, you must do the work yourself.</li> <li>Translate and synthesize: Instead of using the author’s language, establish your own terms. This exercise in translation bridges different authors’ concepts and arguments.</li> <li>Formulate Your Questions: Focus on what you want to know, not just the author’s problems. Frame questions that multiple authors can answer. Be prepared: some questions might remain unanswered if authors didn’t consider them.</li> <li>Define the Issues and Analyze the Discussion: When a question has multiple answers, you’ve defined an issue that needs further work. <a href="https://fs.blog/the-work-required-to-have-an-opinion/">Understanding various perspectives helps form an intelligent opinion.</a> <strong><em>Don’t expect unchallenged truths; your answer lies in the conflict of opposing views.</em></strong> The value lies in your informed dialogue with these authors and yourself.</li> </ul> <h3>Effective Note-Taking for Better Reading</h3> <p>To enhance your reading, Adler recommends various note-taking techniques:</p> <ul> <li>Underline major points</li> <li>Use vertical lines in the margin for longer passages</li> <li>Mark important statements with stars or asterisks</li> <li>Number sequences of points in an argument</li> <li>Cross-reference related ideas with page numbers</li> <li>Circle keywords or phrases</li> <li>Write questions and answers in the margins</li> </ul> <h3>The Demanding Reader</h3> <p>The demanding reader asks the right questions and seeks answers.</p> <p>There are four main questions you need to ask of every book:</p> <ul> <li>What is this book about?</li> <li>What is being said in detail, and how?</li> <li>Is this book true — in whole or in part?</li> <li>What of it?</li> </ul> https://tedneward.github.io/Research/reading/development/software-foundations/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/software-foundations/index.html <p><a href="https://softwarefoundations.cis.upenn.edu/">Website</a></p> <p>The principal novelty of the series is that every detail is one hundred percent formalized and machine-checked: the entire text of each volume, including the exercises, is literally a "proof script" for the Coq proof assistant.</p> <p>The exposition is intended for a broad range of readers, from advanced undergraduates to PhD students and researchers. No specific background in logic or programming languages is assumed, though a degree of mathematical maturity is helpful. A one-semester course can expect to cover Logical Foundations plus most of Programming Language Foundations or Verified Functional Algorithms, or selections from both.</p> <p>Six parts:</p> <ul> <li><a href="https://softwarefoundations.cis.upenn.edu/lf-current/index.html">Logical Foundations (Vol 1)</a> (<a href="https://softwarefoundations.cis.upenn.edu/lf-current/lf.tgz">Download</a>)</li> <li><a href="https://softwarefoundations.cis.upenn.edu/plf-current/index.html">Programming Language Foundations (Vol 2)</a> (<a href="https://softwarefoundations.cis.upenn.edu/plf-current/plf.tgz">Download</a>)</li> <li><a href="https://softwarefoundations.cis.upenn.edu/qc-current/index.html">QuickChick: Property-Based Testing in Coq (Vol 3)</a> (<a href="https://softwarefoundations.cis.upenn.edu/qc-current/qc.tgz">Download</a>)</li> <li><a href="https://softwarefoundations.cis.upenn.edu/vc-current/index.html">Verifiable C (Vol 4)</a> (<a href="https://softwarefoundations.cis.upenn.edu/vc-current/vc.tgz">Download</a>)</li> <li><a href="https://softwarefoundations.cis.upenn.edu/slf-current/index.html">Separation Logic Foundations (Vol 5)</a> (<a href="https://softwarefoundations.cis.upenn.edu/slf-current/slf.tgz">Download</a>)</li> <li><a href="https://softwarefoundations.cis.upenn.edu/vfa-current/index.html">Verified Functional Algorithms</a> (<a href="https://softwarefoundations.cis.upenn.edu/vfa-current/vfa.tgz">Download</a>)</li> </ul> https://tedneward.github.io/Research/reading/development/state-machine/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/state-machine/index.html <p><a href="http://en.wikipedia.org/wiki/Finite-state_machine">Wikipedia</a> | <a href="https://www.reddit.com/r/statemachines/">r/statemachines</a></p> <h2>Overview</h2> <h3>An introduction</h3> <p>From <a href="https://excalidraw.com/">here</a> there is a really nice summary of FSMs:</p> <ol> <li> <p>What is a state machine?</p> <p>A state machine is an abstract concept that describes</p> <ul> <li>being in only one 'finite state' at any time</li> <li>where there is an "initial state"</li> <li>and "transitions" between states triggered by "events"</li> </ul> </li> <li> <p>States: A finite state describes some "mode" or "status" that a system (e.g., an app or component) is in. State machines always start in an initial state. Final states mark that a machine is "done". Extended state describes contextual data that is not finite, but relevant to the machine. Don't confuse finite ("countable") states with extended ("infinite") states!</p> </li> <li> <p>Events &amp; Transitions: An event is a "signal" that something happened. Events can trigger "transitions" between states. Transitions are always zero-time; they happen instantaneously. (Promises &amp; async can still be modeled as a state machine. Just think of "awaiting" as a finite state.) Events are distinguished by a finite set of types and sometimes carry extra data.</p> </li> <li> <p>Actions: Actions are side-effects that are executed due to events. Transition ("do") actions are executed due to transitions. (These are characterisitic of Mealy machines). Entry actions are executed whenever a state is entered. Exit actions are executed whenever a state is exited. (These are characteristic of Moore machines.)</p> </li> </ol> <h3>Another introduction</h3> <p>Finite state machines are a computation model, it consists of a machine with a finite number of states. A finite state machine has a finite number of events, to move from one state to another we have something called transitions. These transitions are triggered after a specific event, each transition expects an input, after its triggered, it will change to a specific state depending on the current state and the event.</p> <p>We can simplify a finite state machine in 4 parts:</p> <ol> <li>A finite number of states.</li> <li>A finite number of events.</li> <li>An initial state.</li> <li>A transition will be triggered after a specific event and will change to a specific state depending on the current state and the event.</li> </ol> <p>Finite state machines can be used to simulate sequential logic, let's take as an example of a traffic light.</p> <p>We know that a traffic light has only 3 possible states: green, yellow, and red. To change from one state to another, we will use it as an input integer representing seconds. When the input is 60, the machine will trigger an event and transition to another state depending on the current state and the event, in our case, the seconds.</p> <p>We can observe finite state machines not only inside the software development but in our real world as well. You can think about almost anything and use finite state machines for it. By having a finite number of states, it reduces drastically the possibility of having unexpected side-effects in your application, your application will get more consistent, well-written, and maintainable.</p> <h3>Further Reading</h3> <ul> <li><a href="https://en.wikipedia.org/wiki/Finite-state_machine">Finite-state_machine</a></li> <li><a href="https://stackabuse.com/theory-of-computation-finite-state-machines/">Theory of Computation - Finite State Machines</a></li> <li><a href="https://www.i-programmer.info/babbages-bag/223-finite-state-machines.html">Finite State Machines</a></li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-319-62533-1_4">Elements of Robotics - Finite State Machines</a></li> <li><a href="http://www.inf.ufsc.br/~joao.dovicchi/pos-ed/pos/exerc/machines2.pdf">Finite State Machines</a></li> <li><a href="https://www.smashingmagazine.com/2018/01/rise-state-machines/">The Rise Of The State Machines</a></li> </ul> <h2>Concepts</h2> <p>In the finite state machine model, we have a few different concepts that it’s very important to learn about. There are two types of finite state machines: deterministic state automaton and non-deterministic state automaton. There are a few differences between those two types.</p> <p>A deterministic state automaton is a finite state machine with a finite number of states and input symbols. A transition is defined in every state for every input symbol. Each input symbol will determine which state the machine will move. A deterministic finite automaton machine can only be in one state at a time, and transition to one state at a time.</p> <p>A non-deterministic finite automaton has a finite number of states and a finite number of input symbols. In a nondeterministic finite automaton, for each state and input symbol, it can be one or more output states. NFAs can be in one or more states at once.</p> <p>If you’re wondering, an automaton is the plural of automata. Finite state machines are also known as finite-state automata or finite-state automaton.</p> <h3>Deterministic Finite Automaton (DFAs)</h3> <p>A deterministic finite automaton finite state machine with a finite number of inputs symbols. For each input symbol, it will determine which state the machine will move.<br> A deterministic finite automaton machine can only be in one state at a time, and transition to one state at a time.</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Deterministic_finite_automaton">Deterministic finite automaton</a></li> <li><a href="https://lambda.uta.edu/cse5317/notes/node8.html">Deterministic Finite Automata (DFAs)</a></li> <li><a href="https://www.cs.uic.edu/~ajayk/c301/CS301-UIC/Lecture-02-DFA.pdf">Deterministic Finite Automata (DFAs)</a></li> <li><a href="https://www.cs.wcupa.edu/rkline/fcs/dfas.html">Deterministic Finite Automata (DFA)</a></li> <li><a href="https://www.tutorialspoint.com/automata_theory/deterministic_finite_automaton.htm">Deterministic Finite Automaton</a></li> <li><a href="http://www.cse.chalmers.se/~coquand/AUTOMATA/o2.pdf">Deterministic Finite Automata</a></li> <li><a href="https://web.cs.ucdavis.edu/~rogaway/classes/120/spring13/eric-dfa.pdf">Applications of Deterministic Finite Automata</a></li> <li><a href="https://cseweb.ucsd.edu/classes/sp15/cse191-e/lec1.html">Deterministic Finite Automata (DFA)</a></li> </ul> <h3>Non-deterministic Finite Automaton (NFAs)</h3> <p>A nondeterministic finite automaton has a finite number of states and a finite number of input symbols. In a nondeterministic finite automaton, for each state and input symbol, it can be one or more output states. NFAs can be in one or more states at once.</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Nondeterministic_finite_automaton">Nondeterministic finite automaton</a></li> <li><a href="http://cs.ru.nl/~jrot/TnA2017/lecture3.pdf">Non-deterministic Finite Automata</a></li> <li><a href="https://courses.engr.illinois.edu/cs374/sp2017/notes/models/NFAnotes.pdf">Nondeterministic Finite Automata</a></li> <li><a href="https://www.tutorialspoint.com/automata_theory/non_deterministic_finite_automaton.htm">Non-deterministic Finite Automaton</a></li> <li><a href="https://cs.uwaterloo.ca/~watrous/CS360/Lectures/03.pdf">Nondeterministic finite automata</a></li> <li><a href="http://people.seas.harvard.edu/~cs125/fall16/lec12.pdf">Nondeterministic Finite Automata</a></li> <li><a href="http://www.inf.ed.ac.uk/teaching/courses/inf1/cl/notes/Comp3.pdf">Non-deterministic Finite State Machines</a></li> <li><a href="https://people.cs.clemson.edu/~goddard/texts/theoryOfComputation/3a.pdf">Nondeterministic Finite Automata</a></li> <li><a href="https://www.javatpoint.com/non-deterministic-finite-automata">NFA (Non-Deterministic finite automata)</a></li> </ul> <h3>Mealy Machine</h3> <p>A mealy machine is a machine where the output value is determined by the input and the current state. For each input and state in a Mealy machine, one transition is possible.</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Mealy_machine">Mealy machine</a></li> <li><a href="https://www.javatpoint.com/automata-mealy-machine">Mealy Machine</a></li> <li><a href="https://ncatlab.org/nlab/show/Mealy+machine">Mealy machine</a></li> <li><a href="https://xlinux.nist.gov/dads/HTML/mealyMachine.html">Mealy machine</a></li> <li><a href="https://www.slideshare.net/arifsiyal7/mealy-state-machine">Mealy state machine</a></li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-642-05089-3_14">Inferring Mealy Machines</a></li> <li><a href="https://www.tutorialspoint.com/automata_theory/moore_and_mealy_machines.htm">Moore and Mealy Machines</a></li> <li><a href="https://www.geeksforgeeks.org/mealy-and-moore-machines-in-toc/">Mealy and Moore Machines in TOC</a></li> </ul> <h3>Moore Machine</h3> <p>A Moore machine is a machine where the output is determined by only its current state. The output depends only on the present state and it's the same output every time the machine enters that state.</p> <ul> <li><a href="https://en.wikipedia.org/wiki/Moore_machine">Moore machine</a></li> <li><a href="https://www.javatpoint.com/automata-moore-machine">Moore Machine</a></li> <li><a href="https://xlinux.nist.gov/dads/HTML/mooreMachine.html">Moore Machine</a></li> <li><a href="https://www.electronics-tutorial.net/finite-state-machines/State-Machine-Fundamentals/Moore-Finite-State-Machine/">Moore Finite State Machine</a></li> <li><a href="http://www2.elo.utfsm.cl/~lsb/elo211/aplicaciones/katz/chapter8/chapter08.doc4.html">Moore and Mealy Machine Design Procedure</a></li> </ul> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="http://blog.manuvra.com/modeling-a-simple-ai-behavior-using-a-finite-state-machine/">Modeling AI</a></li> </ul> <h3>Books</h3> <ul> <li><a href="https://link.springer.com/book/10.1007/978-3-031-43973-5">Programming-Based Formal Languages and Automata Theory</a></li> </ul> <h2>Formal representation</h2> <h3>SMC</h3> <p><a href="/languages/smc">State Machine Compiler</a></p> <h3>SCXML</h3> <p><a href="/formats/scxml">State Machine notation for control abstraction specification</a>.</p> <h3>Statecharts</h3> <p>Sometimes we need something more sophisticated and complex, and that's when a finite state machine cannot help us.</p> <p>Statecharts are an extension of traditional finite state machines, the main difference of statecharts is that it can have a hierarchical state, states can contain nested state inside them. The reason for this is simple, not all applications in the world can be described as flat multi numbers of states, sometimes we need to have nested states.</p> <p>Statecharts also bring us a few extra features such as actions, entry and exit actions, guard conditions, deferred events, etc.</p> <h4>Further Reading</h4> <ul> <li><a href="https://statecharts.github.io/">Welcome to Statecharts</a></li> <li><a href="http://www.inf.ed.ac.uk/teaching/courses/seoc/2005_2006/resources/statecharts.pdf">Statecharts: A Visual Formalism for Complex Systems</a></li> <li><a href="https://help.anylogic.com/index.jsp?topic=%2Fcom.anylogic.help%2Fhtml%2Fstatecharts%2FStatecharts.html">Statecharts</a></li> <li><a href="http://homepage.cs.uiowa.edu/~fleck/181content/statecharts.pdf">The Statecharts Perspective</a></li> </ul> <h2>Languages</h2> <ul> <li><a href="https://states-language.net/">states-language</a>: Amazon States Language. A JSON-based language used to describe state machines declaratively. The state machines thus defined may be executed by software. In this document, the software is referred to as "the interpreter".</li> <li><a href="/languages/dotnet/coyote">Coyote</a></li> <li><a href="https://github.com/morazanm/fsm">fsm</a>: A DSL for the Automata Theory Classroom</li> <li><a href="/languages/p">P</a> and/or <a href="/languages/dotnet/psharp">P#</a></li> <li><a href="/languages/plaid">Plaid</a></li> <li><a href="/languages/ragel">Ragel</a></li> <li>Archived <a href="https://web.archive.org/web/20120722084226/http://www-fp.cs.st-andrews.ac.uk/hume/index.shtml">Hume</a>: Hume combines functional programming ideas with ideas from finite state automata. Automata are used to structure communicating programs into a series of "boxes", where each box maps inputs to outputs in a purely functional way using high-level pattern-matching. It is structured as a series of levels, each of which exposes different machine properties.</li> </ul> <h2>Libraries</h2> <h3>C++</h3> <ul> <li><a href="https://github.com/erikzenker/hsm">hsm</a></li> <li><a href="https://www.boost.org/doc/libs/1_73_0/libs/msm/doc/HTML/index.html">msm</a></li> <li><a href="https://boost-experimental.github.io/sml/index.html">sml</a></li> <li><a href="https://www.boost.org/doc/libs/1_73_0/libs/statechart/doc/index.html">Statechart</a></li> </ul> <h3>Go</h3> <ul> <li><a href="https://github.com/looplab/fsm">fsm</a></li> <li><a href="https://github.com/qor/transition">transition</a></li> <li><a href="https://github.com/qmuntal/stateless">stateless</a></li> <li><a href="https://github.com/ryanfaerman/fsm">fsm</a></li> <li><a href="https://github.com/bykof/stateful">stateful</a></li> <li><a href="https://github.com/aglyzov/ws-machine">ws-machine</a></li> <li><a href="https://github.com/dyrkin/fsm">fsm</a></li> <li><a href="https://github.com/Gurpartap/statemachine-go">statemachine-go</a></li> <li><a href="https://github.com/calebwin/go-sm">go-sm</a></li> <li><a href="https://github.com/theckman/go-fsm">go-fsm</a></li> <li><a href="https://github.com/igorrius/go-fsm">go-fsm</a></li> </ul> <h3>Java</h3> <ul> <li><a href="https://github.com/hekailiang/squirrel">squirrel-foundation</a></li> <li><a href="https://github.com/spring-projects/spring-statemachine">Spring Statemachine</a></li> <li><a href="https://github.com/stateless4j/stateless4j">stateless4j</a></li> <li><a href="https://github.com/Beh01der/EasyFlow">EasyFlow</a></li> <li><a href="https://github.com/statefulj/statefulj">StatefulJ</a></li> <li><a href="https://github.com/j-easy/easy-states">Easy States</a></li> <li><a href="https://github.com/UnquietCode/JState">JState</a></li> <li><a href="https://github.com/doridori/Engine">Engine</a></li> <li><a href="https://github.com/thehiflyer/Fettle">Fettle</a></li> <li><a href="https://github.com/camelot-framework/yatomata">Yatomata</a></li> <li><a href="https://github.com/zevada/stateful">stateful</a></li> <li><a href="https://github.com/davidmoten/state-machine">state-machine</a></li> </ul> <h3>JavaScript</h3> <ul> <li><a href="https://github.com/davidkpiano/xstate">XState</a></li> <li><a href="https://github.com/nickuraltsev/finity">finity</a></li> <li><a href="https://github.com/vstirbu/fsm-as-promised">fsm-as-promised</a></li> <li><a href="https://github.com/StoneCypher/jssm">jssm</a></li> <li><a href="https://github.com/jakesgordon/javascript-state-machine">JavaScript State Machine</a></li> <li><a href="https://github.com/brucou/kingly">kingly</a></li> <li><a href="https://github.com/ifandelse/machina.js">Machina</a></li> <li><a href="https://github.com/tomas2387/maquinaria">maquinaria</a></li> <li><a href="https://github.com/bredele/mood">mood</a></li> <li><a href="https://github.com/choojs/nanostate">nanostate</a></li> <li><a href="https://github.com/MicheleBertoli/react-automata">React Automata</a></li> <li><a href="https://github.com/matthewp/robot">Robot</a></li> <li><a href="https://github.com/ericelliott/redux-dsm">redux-dsm</a></li> <li><a href="https://github.com/mheiber/redux-machine">redux-machine</a></li> <li><a href="https://github.com/steelbreeze/state">state</a></li> <li><a href="https://github.com/davestewart/javascript-state-machine">State Machine</a></li> <li><a href="https://github.com/shuckster/statebot">Statebot</a></li> <li><a href="https://github.com/fschaefer/Stately.js">Stately</a></li> <li><a href="https://github.com/krasimir/stent">Stent</a></li> </ul> <h3>Kotlin</h3> <ul> <li><a href="https://github.com/KStateMachine/kstatemachine">KStateMachine</a></li> <li><a href="https://open.jumpco.io/projects/kfsm.html">https://open.jumpco.io/projects/kfsm.html</a></li> </ul> <h3>Python</h3> <ul> <li><a href="https://github.com/pytransitions/transitions">transitions</a></li> <li><a href="https://github.com/DLR-RM/RAFCON">RAFCON</a></li> <li><a href="https://github.com/viewflow/django-fsm">django-fsm</a></li> <li><a href="https://github.com/glyph/automat">automat</a></li> <li><a href="https://github.com/jtushman/state_machine">state_machine</a></li> <li><a href="https://github.com/rbarrois/xworkflows">xworkflows</a></li> <li><a href="https://github.com/vikingco/django-states2">django-states2</a></li> <li><a href="https://github.com/fgmacedo/python-statemachine">python-statemachine</a></li> <li><a href="https://github.com/ScatterHQ/machinist">machinist</a></li> <li><a href="https://github.com/nsi-iff/fluidity">fluidity</a></li> <li><a href="https://github.com/openstack/automaton">automaton</a></li> <li><a href="https://github.com/pgularski/pysm">pysm</a> <a href="https://pysm.readthedocs.io/en/latest/">Docs</a></li> <li><a href="https://github.com/thomasquintana/declarative-fsm">declarative-fsm</a></li> <li><a href="https://github.com/nazavode/automaton">automaton</a></li> </ul> <h3>Ruby</h3> <ul> <li><a href="https://github.com/aasm/aasm">aasm</a></li> <li><a href="https://github.com/pluginaweek/state_machine">state_machine</a></li> <li><a href="https://github.com/gocardless/statesman">statesman</a></li> <li><a href="https://github.com/piotrmurach/finite_machine">finite_machine</a></li> <li><a href="https://github.com/state-machines/state_machines">state_machines</a></li> <li><a href="https://github.com/troessner/transitions">transitions</a></li> <li><a href="https://github.com/amatsuda/stateful_enum">stateful_enum</a></li> <li><a href="https://github.com/soveran/micromachine">micromachine</a></li> <li><a href="https://github.com/ryan-allen/workflow">workflow</a></li> <li><a href="https://github.com/rtwomey/stately">stately</a></li> <li><a href="https://github.com/opyh/motion-state-machine">motion-state-machine</a></li> <li><a href="https://github.com/emancu/aquam">aquam</a></li> </ul> <h3>Shell</h3> <ul> <li><a href="https://github.com/shuckster/statebot-sh">Statebot-sh</a></li> </ul> <h3>Swift</h3> <ul> <li><a href="https://github.com/sergebouts/hsm">hsm</a></li> </ul> https://tedneward.github.io/Research/reading/development/static-analysis-benchmarking/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/static-analysis-benchmarking/index.html <ul> <li>A Comparison of Static Analysis Tools for Vulnerability Detection in C/C++ Code <ul> <li>SYNASC 2017</li> <li>Andrei Arusoaie, Ștefan Ciobâcă, Vlad Crăciun, Dragoș Gavriluț, Dorel Lucanu</li> <li><a href="https://profs.info.uaic.ro/~stefan.ciobaca/synasc2017.pdf">https://profs.info.uaic.ro/~stefan.ciobaca/synasc2017.pdf</a></li> </ul> </li> <li>Analyzing the State of Static Analysis: A Large-Scale Evaluation in Open Source Software <ul> <li>Software Analysis, Evolution, and Reengineering (SANER) 2016</li> <li><a href="http://ieeexplore.ieee.org/document/7476667/">http://ieeexplore.ieee.org/document/7476667/</a></li> <li><a href="https://www.slideshare.net/inventitech/analyzing-the-state-of-static-analysis-a-largescale-evaluation-in-open-source-software-60473247">https://www.slideshare.net/inventitech/analyzing-the-state-of-static-analysis-a-largescale-evaluation-in-open-source-software-60473247</a></li> <li>2015 Master’s Thesis: <a href="https://repository.tudelft.nl/islandora/object/uuid:3d834130-8dd7-420a-9af9-6e77761cdad6/">https://repository.tudelft.nl/islandora/object/uuid:3d834130-8dd7-420a-9af9-6e77761cdad6/</a></li> </ul> </li> <li>Automated Customized Bug-Benchmark Generation <ul> <li>IEEE SCAM 2019</li> <li>Vineeth Kashyap, Jason Ruchti, Lucja Kot, Emma Turetsky, Rebecca Swords, David Melski, Eric Schulte</li> <li><a href="https://arxiv.org/abs/1901.02819">https://arxiv.org/abs/1901.02819</a></li> <li><a href="https://www.dropbox.com/sh/f3p1wbomv93fz78/AABzXcLzqp5zkx75nQym6gJPa">https://www.dropbox.com/sh/f3p1wbomv93fz78/AABzXcLzqp5zkx75nQym6gJPa</a></li> </ul> </li> <li>Benchmark Software for Evaluation of Software Analysis Tools <ul> <li><a href="https://github.com/Toyota-ITC-SSD/Software-Analysis-Benchmark">https://github.com/Toyota-ITC-SSD/Software-Analysis-Benchmark</a></li> <li>Static analysis benchmarks from Toyota ITC - <a href="https://github.com/regehr/itc-benchmarks">https://github.com/regehr/itc-benchmarks</a></li> </ul> </li> <li>How Toyota Picks Software Tools <ul> <li>EETimes, August 11, 2015; Bernard Cole</li> <li><a href="https://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1327387">https://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1327387</a></li> <li><a href="http://web.archive.org/web/20150815120014/http://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1327387">http://web.archive.org/web/20150815120014/http://www.eetimes.com/author.asp?section_id=36&amp;doc_id=1327387</a></li> </ul> </li> <li>Quantitative Evaluation of Static Analysis Tools <ul> <li>Software Reliability Engineering Workshops (ISSREW) 2014</li> <li>Shinichi Shiraishi, Veena Mohan, Hemalatha Marimuthu</li> <li><a href="https://dl.acm.org/citation.cfm?id=2706360">https://dl.acm.org/citation.cfm?id=2706360</a></li> </ul> </li> <li>Software Assurance Reference Dataset (SARD) <ul> <li><a href="https://samate.nist.gov/SARD/">https://samate.nist.gov/SARD/</a></li> <li>Test Suites - <a href="https://samate.nist.gov/SRD/testsuite.php">https://samate.nist.gov/SRD/testsuite.php</a></li> </ul> </li> <li>Static Analysis Benchmarks <ul> <li><a href="https://blog.regehr.org/archives/1217">https://blog.regehr.org/archives/1217</a></li> </ul> </li> <li>Test Suites for Benchmarks of Static Analysis Tools <ul> <li>IEEE International Symposium on Software Reliability Engineering (ISSRE) 2015</li> <li>Shinichi Shiraishi, Veena Mohan, and Hemalatha Marimuthu</li> <li><a href="https://www.researchgate.net/publication/283548090_Test_Suites_for_Benchmarks_of_Static_Analysis_Tools">https://www.researchgate.net/publication/283548090_Test_Suites_for_Benchmarks_of_Static_Analysis_Tools</a></li> </ul> </li> <li>Towards Automatically Generating a Sound and Complete Dataset for Evaluating Static Analysis Tools <ul> <li>NDSS Workshop on Binary Analysis Research (BAR) 2019</li> <li>Aravind Machiry, Nilo Redini, Eric Gustafson, Hojjat Aghakhani, Christopher Kruegel, Giovanni Vigna</li> <li><a href="https://ruoyuwang.me/bar2019/pdfs/bar2019-final90.pdf">https://ruoyuwang.me/bar2019/pdfs/bar2019-final90.pdf</a></li> <li><a href="https://github.com/ucsb-seclab/autofacts">https://github.com/ucsb-seclab/autofacts</a></li> </ul> </li> <li>Toyota ITC Benchmark <ul> <li><a href="https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benchmark/">https://runtimeverification.com/match/1.0-SNAPSHOT/docs/benchmark/</a></li> </ul> </li> <li>Using Benchmarks to Assess Static Analysis Tools <ul> <li><a href="http://blogs.grammatech.com/using-benchmarks-to-assess-static-analysis-tools">http://blogs.grammatech.com/using-benchmarks-to-assess-static-analysis-tools</a></li> </ul> </li> <li>Differentially testing soundness and precision of program analyzers <ul> <li>International Symposium on Software Testing and Analysis 2019</li> <li>Christian Klinger, Maria Christakis, Valentin Wüstholz</li> <li><a href="https://arxiv.org/abs/1812.05033">https://arxiv.org/abs/1812.05033</a></li> </ul> </li> <li>Quickchecking Static Analysis Properties <ul> <li>Testing, Verification and Reliability 27(6) 2017</li> <li>Jan Midtgaard, Anders Møller</li> <li><a href="http://janmidtgaard.dk/papers/Midtgaard-Moeller%3aSTVR17.pdf">http://janmidtgaard.dk/papers/Midtgaard-Moeller%3aSTVR17.pdf</a></li> </ul> </li> <li>Test suites for benchmarks of static analysis tools <ul> <li>Software Reliability Engineering Workshops (ISSREW) 2015</li> <li>Shinichi Shiraishi, Veena Mohan, Hemalatha Marimuthu</li> <li><a href="http://ieeexplore.ieee.org/document/7392027/">http://ieeexplore.ieee.org/document/7392027/</a></li> </ul> </li> <li>Testing Static Analyses for Precision and Soundness <ul> <li>CGO 2020</li> <li>Jubi Taneja, Zhengyang Liu, and John Regehr</li> <li><a href="https://doi.org/10.1145/3368826.3377927">https://doi.org/10.1145/3368826.3377927</a></li> </ul> </li> <li>Testing Static Analyzers with Randomly Generated Programs <ul> <li>NFM 2012: NASA Formal Methods</li> <li>Pascal Cuoq, Benjamin Monate, Anne Pacalet, Virgile Prevosto, John Regehr, Boris Yakobowski, Xuejun Yang</li> <li><a href="http://blog.frama-c.com/public/csmith.pdf">http://blog.frama-c.com/public/csmith.pdf</a></li> </ul> </li> <li>Towards Scalable Translation Validation of Static Analyzers <ul> <li>ROSAEC Technical Report ROSAEC-2014-003, Nov 2014</li> <li>Jeehoon Kang, Sungkeun Cho, Joonwon Choi, Chung-Kil Hur, Kwangkeun Yi</li> <li><a href="http://sf.snu.ac.kr/gil.hur/publications/ROSAEC_2014_003.pdf">http://sf.snu.ac.kr/gil.hur/publications/ROSAEC_2014_003.pdf</a></li> </ul> </li> <li>Systematic Approaches for Increasing Soundness and Precision of Static Analyzers <ul> <li>State Of the Art in Program Analysis (SOAP) 2017</li> <li>Esben Sparre Andreasen, Anders Møller, Benjamin Barslev Nielsen</li> <li><a href="https://pldi17.sigplan.org/event/soap-2017-papers-systematic-approaches-for-increasing-soundness-and-precision-of-static-analyzers">https://pldi17.sigplan.org/event/soap-2017-papers-systematic-approaches-for-increasing-soundness-and-precision-of-static-analyzers</a></li> <li><a href="https://cs.au.dk/~amoeller/papers/tajsexperience/paper.pdf">https://cs.au.dk/~amoeller/papers/tajsexperience/paper.pdf</a></li> </ul> </li> <li>Verasco, a formally verified C static analyzer <ul> <li>A Formally-Verified C Static Analyzer <ul> <li>POPL 2015</li> <li>Jacques-Henri Jourdan, Vincent Laporte, Sandrine Blazy, Xavier Leroy, David Pichardie</li> <li><a href="https://hal.inria.fr/hal-01078386">https://hal.inria.fr/hal-01078386</a></li> <li><a href="https://xavierleroy.org/publi/verasco-popl2015.pdf">https://xavierleroy.org/publi/verasco-popl2015.pdf</a></li> </ul> </li> <li>Microsoft Research 2016; Jacques Henri Jourdan <ul> <li>"the design and soundness proof of Verasco, a formally verified static analyzer for most of the ISO C99 language (excluding recursion and dynamic allocation), developed using the Coq proof assistant."</li> <li><a href="https://www.youtube.com/watch?v=0pUueg3Dslo">https://www.youtube.com/watch?v=0pUueg3Dslo</a></li> </ul> </li> <li>Checking a Checker - Verasco: a Formally Verified C Static Analyzer <ul> <li>2016 PhD Dissertation; Jacques-Henri Jourdan</li> <li><a href="https://jhjourdan.mketjh.fr/thesis_jhjourdan.pdf">https://jhjourdan.mketjh.fr/thesis_jhjourdan.pdf</a></li> </ul> </li> </ul> </li> <li>Verified Translation Validation of Static Analyses <ul> <li>Computer Security Foundations Symposium 2017</li> <li>Gilles Barthe, Sandrine Blazy, Vincent Laporte, David Pichardie, Alix Trieu</li> <li><a href="https://hal.inria.fr/hal-01588422/">https://hal.inria.fr/hal-01588422/</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/operating-systems/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/operating-systems/index.html <h2>General</h2> <ul> <li><a href="http://markburgess.org/os/os.pdf">A short introduction to operating systems (2001)</a> - Mark Burgess (PDF)</li> <li><a href="http://www.bottomupcs.com">Computer Science from the Bottom Up</a></li> <li><a href="https://github.com/SamyPesse/How-to-Make-a-Computer-Operating-System">How to Make a Computer Operating System</a> (:construction: <em>in process</em>)</li> <li><a href="http://mikeos.sourceforge.net/write-your-own-os.html">How to write a simple operating system in assembly language</a> - Mike Saunders (HTML)</li> <li><a href="https://www.digitalocean.com/community/books/sysadmin-ebook-making-servers-work">Making Servers Work: A Practical Guide to Linux System Administration</a> - Jamon Camisso (PDF, EPUB)</li> <li><a href="https://gustavus.edu/mcs/max/os-book/">Operating Systems and Middleware</a> (PDF and LaTeX)</li> <li><a href="http://pages.cs.wisc.edu/~remzi/OSTEP/">Operating Systems: Three Easy Pieces</a> (PDF)</li> <li><a href="http://www.nobius.org/~dbg/practical-file-system-design.pdf">Practical File System Design: The Be File System</a> - Dominic Giampaolo (PDF)</li> <li><a href="http://people.inf.ethz.ch/wirth/ProjectOberon/index.html">Project Oberon: The Design of an Operating System, a Compiler, and a Computer</a> - Niklaus Wirth &amp; Jürg Gutknecht (PDF)</li> <li><a href="http://catb.org/esr/writings/taoup/html/">The Art of Unix Programming</a> - Eric S. Raymond</li> <li><a href="http://www.fixup.fi/misc/rumpkernel-book/">The Design and Implementation of the Anykernel and Rump Kernels</a> - Antti Kantee</li> <li><a href="https://littleosbook.github.io">The little book about OS development</a> (<a href="https://littleosbook.github.io/book.pdf">PDF</a>)- Erik Helin, Adam Renberg - This is not your typical dry academic textbook. It’s a hands-on, step-by-step guide aimed at hackers, tinkerers, and developers who want to demystify kernel programming. The book walks you through setting up your environment, bootstrapping your OS, handling interrupts, implementing virtual memory, and even tackling system calls and multitasking. It provides just enough detail to get you started but leaves room for exploration – because, let’s be honest, half the fun is in figuring things out yourself.</li> <li><a href="http://greenteapress.com/semaphores/">The Little Book of Semaphores</a> - Allen B. Downey</li> <li><a href="http://www.greenteapress.com/thinkos/index.html">Think OS: A Brief Introduction to Operating Systems</a> - Allen B. Downey (PDF)</li> <li><a href="http://www.compsci.hunter.cuny.edu/~sweiss/course_materials/unix_lecture_notes.php">UNIX Application and System Programming, lecture notes</a> - Prof. Stewart Weiss (PDF)</li> <li><a href="http://www.cs.bham.ac.uk/~exr/lectures/opsys/10_11/lectures/os-dev.pdf">Writing a Simple Operating System from Scratch</a> - Nick Blundell (PDF)</li> </ul> <h2>Linux</h2> <ul> <li><a href="https://www.linuxfromscratch.org/lfs/view/stable/">Linux From Scratch</a> - Gerard Beekmans, Bruce Dubbs, Ken Moffat, Pierre Labastie et al. (HTML, <a href="https://www.linuxfromscratch.org/lfs/downloads/stable/">PDF, downloads...</a>)</li> </ul> https://tedneward.github.io/Research/reading/development/pragmatic-programmer/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/pragmatic-programmer/index.html <p><em>(by Andy Hunt, Dave Thomas (Addison-Wesley) )</em></p> <h1>Checklists</h1> <h2>The WISDOM acrostic</h2> <ul> <li>What:&nbsp;What do you want them to learn?</li> <li>Interest:&nbsp;What is their interest in what you've got to say?</li> <li>Sophistication:&nbsp;How sophisticated are they?</li> <li>Detail:&nbsp;How much detail do they want?</li> <li>Own:&nbsp;Whom do you want to own the information?</li> <li>Motivate:&nbsp;How can you motivate them to listen to you?&nbsp;</li> </ul> <h2>How to maintain orthogonality</h2> <ul> <li>Design independent, well-defined components.</li> <li>Keep your code decoupled.</li> <li>Avoid global data.</li> <li>Refactor similar functions.</li> </ul> <h2>Things to prototype</h2> <ul> <li>Architecture</li> <li>New functionality in an existing system</li> <li>Structure or contents of external data</li> <li>Third-party tools or components</li> <li>Performance issues</li> <li>User interface design</li> </ul> <h2>Architectural questions</h2> <ul> <li>Are responsibilities well defined?</li> <li>Are the collaborations well defined?</li> <li>Is coupling minimized?</li> <li>Can you identify potential duplication?</li> <li>Are interface definitions and constrains acceptable?</li> <li>Can modules access needed data--when needed?</li> </ul> <h2>Debugging</h2> <ul> <li>Is the problem being reported a direct result of the underlying bug, or merely a symptom?</li> <li>Is the bug really in the compiler? Is it in the OS? Or is it in your code?</li> <li>If you explained this problem in detail to a coworker, what would you say?</li> <li>If the suspect code passes its unit tests, are the tests complete enough? What happens if you run the unit test with "this" data?</li> <li>Do the conditions that caused this bug exist anywhere else in the system?</li> </ul> <h2>Law of Demeter for functions</h2> <p>An object's method should call only methods belonging to:</p> <ul> <li>Itself</li> <li>Any parameters passed in</li> <li>Objects it creates</li> <li>Component objects</li> </ul> <h2>How to Program Deliberately</h2> <ul> <li>Stay aware of what you're doing</li> <li>Don't code blindfolded</li> <li>Proceed from a plan</li> </ul> <h2>Rely only on reliable things</h2> <ul> <li>Document your assumptions</li> <li>Test assumptions as well as code</li> <li>Prioritize your effort</li> <li>Don't be a slave to history</li> </ul> <h2>When to Refactor</h2> <ul> <li>You discover a violation of DRY</li> <li>You find things that could be more orthogonal</li> <li>Your knowledge improves</li> <li>The requirements evolve</li> <li>You need to improve performance</li> </ul> <h2>Cutting the Gordian Knot</h2> <ul> <li>Is there an easier way?</li> <li>Am I solving the right problem?</li> <li>Why is this a problem?</li> <li>What makes it hard?</li> <li>Do I have to do it this way?</li> <li>Does it have to be done at all?</li> </ul> <h2>Aspects of Testing</h2> <ul> <li>Unit testing</li> <li>Integration testing</li> <li>Validation and verification</li> <li>Resource exhaustion, errors and recovery</li> <li>Performance testing</li> <li>Usability testing</li> <li>Testing the tests themselves</li> </ul> <h1>Principles</h1> <ol> <li> <p><strong>Care About Your Craft.</strong> Why spend your life developing software unless you care about doing it well?</p> </li> <li> <p><strong>Think! About Your Work.</strong>&nbsp;Turn off the autopilot and take control. Constantly critique and appraise your work.</p> </li> <li> <p><strong>Provide Options, Don't Make Lame Excuses.</strong>&nbsp;Instead of excuses, provide options. Don't say it can't be done; say what can be done.</p> </li> <li> <p><strong>Don't Live with Broken Windows.</strong>&nbsp;Fix bad designs, wrong decisions, and poor code when you see them.</p> </li> <li> <p><strong>Be a Catalyst for Change.</strong>&nbsp;You can't force change on people. Instead, show them how the future might be and help them participate in creating it.</p> </li> <li> <p><strong>Remember the Big Picture.</strong>&nbsp;Don't get so engrossed in the details that you forget to check what's happening around you.</p> </li> <li> <p><strong>Make Quality a Requirements Issue.</strong>&nbsp;Involve your users in determining the project's real quality requirements.</p> </li> <li> <p><strong>Invest Regularly in your Knowledge Portfolio.</strong>&nbsp;Make learning a habit.</p> </li> <li> <p><strong>Critically Analyze What You Read and Hear.</strong>&nbsp;Don't be swayed by vendors, media hype or dogma. Analyze information in terms of you and your project.</p> </li> <li> <p><strong>It's Both What You Say and How You Say It.</strong>&nbsp;There's no point in having great ideas if you don't communicate them effectively.</p> </li> <li> <p><strong>DRY--Don't Repeat Yourself.</strong>&nbsp;Every piece of knowledge must have a single, unambiguous, authoritative representation within a system.</p> </li> <li> <p><strong>Make It Easy to Reuse.</strong>&nbsp;If it's easy to reuse, people will. Create an environment that supports reuse.</p> </li> <li> <p><strong>Eliminate Effects Between Unrelated Things.</strong>&nbsp;Design components that are self-contained, independent, and have a single, well-defined purpose.</p> </li> <li> <p><strong>There Are No Final Decisions.</strong>&nbsp;No decision is cast in stone. Instead, consider each as being written in the sand at the beach, and plan for change.</p> </li> <li> <p><strong>Use Tracer Bullets to Find the Target.</strong>&nbsp;Tracer bullets let you home in on your target by trying things and seeing how close they land.</p> </li> <li> <p><strong>Prototype to Learn.</strong>&nbsp;Prototyping is a learning experience. Its value lies not in the code you produce, but in the lessons you learn.</p> </li> <li> <p><strong>Program Close to the Problem Domain.</strong>&nbsp;Design and code in your user's language.</p> </li> <li> <p><strong>Estimate to Avoid Surprises.</strong>&nbsp;Estimate before you start. You'll spot potential problems up front.</p> </li> <li> <p><strong>Iterate the Schedule with the Code.</strong>&nbsp;Use experience you gain as you implement to refine the project time scales.</p> </li> <li> <p><strong>Keep Knowledge in Plain Text.</strong>&nbsp;Plain text won't become obsolete. It helps leverage your work and simplifies debugging and testing.</p> </li> <li> <p><strong>Use the Power of Command Shells.</strong>&nbsp;Use the shell when GUIs don't cut it.</p> </li> <li> <p><strong>Use a Single Editor Well.</strong>&nbsp;The editor should be an extension of your fingertips. make sue your editor is configurable, extensible, and programmable.</p> </li> <li> <p><strong>Always use Source Code Control.</strong>&nbsp;Source code control is a time machine for your work--you can go back.</p> </li> <li> <p><strong>Fix the Problem, Not the Blame.</strong>&nbsp;It doesn't really matter whether the bug is your fault or someone else's--it is still your problem, and it still needs to be fixed.</p> </li> <li> <p><strong>Don't Panic When Debugging.</strong>&nbsp;Take a deep breath and THINK! about what could be causing the bug.</p> </li> <li> <p><strong>"select" Isn't Broken.</strong>&nbsp;It is rare to find a bug in the OS or the compiler, or even a third-party product or library. The bug is most likely in the application.</p> </li> <li> <p><strong>Don't Assume It--Prove It.</strong>&nbsp;Prove your assumptions in the actual environment--with real data and boundary conditions.</p> </li> <li> <p><strong>Learn a Text Manipulation Language.</strong>&nbsp;You spend a large part of each day working with text. Why not have the computer do some of it for you?</p> </li> <li> <p><strong>Write Code That Writes Code.</strong>&nbsp;Code generators increase your productivity and help avoid duplication.</p> </li> <li> <p><strong>You Can't Write Perfect Software.</strong>&nbsp;Software can't be perfect. Protect your code and users from the inevitable errors.</p> </li> <li> <p><strong>Design With Contracts.</strong>&nbsp;Use contracts to document and verify that code does no more and no less than it claims to do.</p> </li> <li> <p><strong>Crash Early.</strong>&nbsp;A dead program normally does a lot less damage than a crippled one.</p> </li> <li> <p><strong>Use Assertions to Prevent the Impossible.</strong>&nbsp;Assertions validate your assumptions. Use them to protect your code from an uncertain world.</p> </li> <li> <p><strong>Use Exceptions for Exceptional Problems.</strong>&nbsp;Exceptions can suffer from all the readability and maintainability problems of classic spaghetti code. Reserve exceptions for exceptional things.</p> </li> <li> <p><strong>Finish What You Start.</strong>&nbsp;Where possible, the routine or object that allocates a resource should be responsible for deallocating it.</p> </li> <li> <p><strong>Minimize Coupling Between Modules.</strong>&nbsp;Avoid coupling by writing "shy" code and applying the Law of Demeter.</p> </li> <li> <p><strong>Configure, Don't Integrate.</strong>&nbsp;Implement technology choices for an application as configuration options, not through integration or engineering.</p> </li> <li> <p><strong>Put Abstractions in Code, Details in Metadata.</strong>&nbsp;Program for the general case, and put the specifics outside the compiled code base.</p> </li> <li> <p><strong>Analyze Workflow to Improve Concurrency.</strong>&nbsp;Exploit concurrency in your users' workflow.</p> </li> <li> <p><strong>Design Using Services.</strong>&nbsp;Design in terms of services--independent, concurrent objects behind well-defined, consistent interfaces.</p> </li> <li> <p><strong>Always Design for Concurrency.</strong>&nbsp;Allow for concurrency, and you'll design cleaner interfaces with fewer assumptions.</p> </li> <li> <p><strong>Separate Views from Models.</strong>&nbsp;Gain flexibility at low cost by designing your application in terms of models and views.</p> </li> <li> <p><strong>Use Blackboards to Coordinate Workflow.</strong>&nbsp;Use blackboards to coordinate disparate facts and agents, while maintaining independence and isolation among participants.</p> </li> <li> <p><strong>Don't Program by Coincidence.</strong>&nbsp;Rely only on reliable things. Beware of accidental complexity, and don't confuse a happy coincidence with a purposeful plan.</p> </li> <li> <p><strong>Estimate the Order of your Algorithms.</strong>&nbsp;Get a feel for how long things are likely to take before you write code.</p> </li> <li> <p><strong>Test Your Estimates.</strong>&nbsp;Mathematical analysis of algorithms doesn't tell you everything. Try timing your code in its target environment.</p> </li> <li> <p><strong>Refactor Early, Refactor Often.</strong>&nbsp;Just as you might weed and rearrange a garden, rewrite, rework, and re-architect code when it needs it. Fix the root of the problem.</p> </li> <li> <p><strong>Design to Test.</strong>&nbsp;Start thinking about testing before you write a line of code.</p> </li> <li> <p><strong>Test Your Software, or Your Users Will.</strong>&nbsp;Test ruthlessly. Don't make your users find bugs for you.</p> </li> <li> <p><strong>Don't Use Wizard Code You Don't Understand.</strong>&nbsp;Wizards can generate reams of code. Make sur eyou understand all of it before you incorporate it into your project.</p> </li> <li> <p><strong>Don't Gather Requirements--Dig for Them.</strong>&nbsp;Requirements rarely lie on the surface. They're buried deep beneath layers of assumptions, misconceptions, and politics.</p> </li> <li> <p><strong>Work With a User to Think Like a User.</strong>&nbsp;It's the best way to gain insight into how the system will really be used.</p> </li> <li> <p><strong>Abstractions Live Longer Than Details.</strong>&nbsp;Invest in the abstraction, not the implementation. Abstractions can survive the barrage of changes from different implementations and new technologies.</p> </li> <li> <p><strong>Use a Project Glossary.</strong>&nbsp;Create and maintain a single source of all the specific terms and vocabulary for the project.</p> </li> <li> <p><strong>Don't Think Outside the Box--Find the Box.</strong>&nbsp;When faced with an impossible problem, identify the real constraints. Ask yourself: "Does it have to be done this way? Does it have to be done at all?"</p> </li> <li> <p><strong>Start When You're Ready.</strong>&nbsp;You've been building experience all your life. Don't ignore niggling doubts.</p> </li> <li> <p><strong>Some Things Are Better Done Than Described.</strong>&nbsp;Don't fall into the specification spiral--at some point you need to start coding.</p> </li> <li> <p><strong>Don't Be a Slave to Formal Methods.</strong>&nbsp;Don't blindly adopt any technique without putting it into the context of your development practices and capabilities.</p> </li> <li> <p><strong>Costly Tools Don't Produce Better Designs.</strong>&nbsp;Beware of vendor hype, industry dogma, and the aura of the price tag. Judge tools on their merits.</p> </li> <li> <p><strong>Organize Teams Around Functionality.</strong>&nbsp;Don't separate designers from coders, testers from data modelers. Build teams the way you build code.</p> </li> <li> <p><strong>Don't Use Manual Procedures.</strong>&nbsp;A shell script or batch file will execute the same instructions, in the same order, time after time.</p> </li> <li> <p><strong>Test Early, Test Often, Test Automatically.</strong>&nbsp;Tests that run with every build are much more effective than test plans that will sit on a shelf.</p> </li> <li> <p><strong>Coding Ain't Done 'Till All The Tests Run.</strong></p> </li> <li> <p><strong>Use Saboteurs to Test Your Testing.</strong>&nbsp;Introduce bugs on purpose in a separate copy of the source to verify that unit testing will catch them.</p> </li> <li> <p><strong>Test State Coverage, Not Code Coverage.</strong>&nbsp;Identify and test significant program states. Just testing lines of code isn't enough.</p> </li> <li> <p><strong>Find Bugs Once.</strong>&nbsp;Once a human tester finds a bug, it should be the last time a human tester finds that bug. Automatic tests should check for it from then on.</p> </li> <li> <p><strong>English is Just a Programming Language.</strong>&nbsp;Write documents as you would write code: honor the DRY principle, use metadata, MVC, automatic generation, and so on.</p> </li> <li> <p><strong>Build Documentation In, Don't Bolt It On.</strong>&nbsp;Documentation created separately from code is less likely to be correct and up to date.</p> </li> <li> <p><strong>Gently Exceed Your Users' Expectations.</strong>&nbsp;Come to understand your users' expectations, then deliver just that little bit more.</p> </li> <li> <p><strong>Sign Your Work.</strong>&nbsp;Craftsmen of an earlier age were proud to sign their work. You should be, too.</p> </li> </ol> https://tedneward.github.io/Research/reading/development/sicp/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/sicp/index.html <p><em>(by Abelman, Sussman, and Sussman)</em></p> <p><a href="https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book.html">Book (HTML)</a></p> <p><a href="https://web.mit.edu/6.001/6.037/sicp.pdf">PDF</a> | <a href="https://github.com/sarabander/sicp-pdf">GitHub PDF?</a> (I'm not sure if this is an "official" site or a fan's port) | <a href="https://github.com/sarabander/sicp">GitHub HTML</a> (sister project to the PDF)</p> <h1>Notes</h1> <h2>1: Building Abstractions with Procedures</h2> <h2>2: Building Abstractions with Data</h2> <h2>3: Modularity, Objects, and State</h2> <h2>4: Metalinguistic Abstraction</h2> <h2>5: Computing with Register Machines</h2> https://tedneward.github.io/Research/reading/development/papers-we-love/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/papers-we-love/index.html <p><a href="https://paperswelove.org/">Website</a> | <a href="https://github.com/papers-we-love/papers-we-love">GitHub</a></p> https://tedneward.github.io/Research/reading/development/reactivemanifesto/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/reactivemanifesto/index.html <p><a href="https://www.reactivemanifesto.org/">Website</a></p> <h2>Portions lifted from the site (so that I may put notes in italics within or near the prose), visit there for full body.</h2> <p>Organisations working in disparate domains are independently discovering patterns for building software that look the same. These systems are more robust, more resilient, more flexible and better positioned to meet modern demands. <em>(I'm not 100% sure this is any more true than when we said the exact same thing about object-oriented systems thirty years ago.)</em></p> <p>These changes are happening because application requirements have changed dramatically in recent years. Only a few years ago a large application had tens of servers, seconds of response time, hours of offline maintenance and gigabytes of data. Today applications are deployed on everything from mobile devices to cloud-based clusters running thousands of multi-core processors. Users expect millisecond response times and 100% uptime. Data is measured in Petabytes. Today's demands are simply not met by yesterday’s software architectures. <em>(This is probably true, though I think there's some generalization going on here.)</em></p> <p>We believe that a coherent approach to systems architecture is needed, and we believe that all necessary aspects are already recognised individually: we want systems that are Responsive, Resilient, Elastic and Message Driven. We call these Reactive Systems. <em>(This is good, but there's other qualities we want out of a system, too, and I'm not sure the authors aren't skating quickly across some thin ice here in their effort to promote reactivity.)</em></p> <p>Systems built as Reactive Systems are more flexible, loosely-coupled and scalable. <em>(I'm not 100% sure this is 100% true 100% of the time. They can be, but that doesn't mean they will be.)</em> This makes them easier to develop and amenable to change. They are significantly more tolerant of failure and when failure does occur they meet it with elegance rather than disaster. Reactive Systems are highly responsive, giving users effective interactive feedback.</p> <p>Reactive Systems are:</p> <ul> <li> <p><strong>Responsive</strong>: The system responds in a timely manner if at all possible. Responsiveness is the cornerstone of usability and utility, but more than that, responsiveness means that problems may be detected quickly and dealt with effectively. Responsive systems focus on providing rapid and consistent response times, establishing reliable upper bounds so they deliver a consistent quality of service. This consistent behaviour in turn simplifies error handling, builds end user confidence, and encourages further interaction.</p> </li> <li> <p><strong>Resilient</strong>: The system stays responsive in the face of failure. This applies not only to highly-available, mission-critical systems — any system that is not resilient will be unresponsive after a failure. Resilience is achieved by replication, containment, isolation and delegation. Failures are contained within each component, isolating components from each other and thereby ensuring that parts of the system can fail and recover without compromising the system as a whole. Recovery of each component is delegated to another (external) component and high-availability is ensured by replication where necessary. The client of a component is not burdened with handling its failures.</p> </li> <li> <p><strong>Elastic</strong>: The system stays responsive under varying workload. Reactive Systems can react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs. This implies designs that have no contention points or central bottlenecks, resulting in the ability to shard or replicate components and distribute inputs among them. Reactive Systems support predictive, as well as Reactive, scaling algorithms by providing relevant live performance measures. They achieve elasticity in a cost-effective way on commodity hardware and software platforms.</p> </li> <li> <p><strong>Message Driven</strong>: Reactive Systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling, isolation and location transparency. This boundary also provides the means to delegate failures as messages. Employing explicit message-passing enables load management, elasticity, and flow control by shaping and monitoring the message queues in the system and applying back-pressure when necessary. Location transparent messaging as a means of communication makes it possible for the management of failure to work with the same constructs and semantics across a cluster or within a single host. Non-blocking communication allows recipients to only consume resources while active, leading to less system overhead.</p> </li> </ul> <p>Large systems are composed of smaller ones and therefore depend on the Reactive properties of their constituents. This means that Reactive Systems apply design principles so these properties apply at all levels of scale, making them composable. The largest systems in the world rely upon architectures based on these properties and serve the needs of billions of people daily. It is time to apply these design principles consciously from the start instead of rediscovering them each time.</p> https://tedneward.github.io/Research/reading/development/passionate-programmer/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/passionate-programmer/index.html <p><em>(by Chad Fowler)</em></p> <h3>Introduction</h3> <ul> <li>According to the U.S. Bureau of Labor Statistics, the average American spends half their waking time at work.</li> <li>If your life is primarily consumed by your work, then loving your work is one of the most important keys to loving your life.</li> <li>Once we have basic human needs like food and shelter taken care of, most of our goals are geared toward finding happiness.</li> </ul> <h4>Failure Is Off The Radar!</h4> <ul> <li>A person who wants to be great is far more likely to become great than someone who just wants to do their job.</li> </ul> <h4>You Own It</h4> <ul> <li>To excel at a company, you have to understand how you fit into the business's plan to make money.</li> <li>Your company is investing in you. Your challenge is to become an obviously good investment, and to judge your own performance in terms of the value you bring to the business.</li> </ul> <h3>Part 1: Choosing Your Market</h3> <ul> <li>Many of us just float down the stream of our careers, letting the current take us where it may.</li> <li>If you think of your career as a business (which it is), then your "product" is made up of the services you have to offer.</li> </ul> <h4>Tip 1: Lead or Bleed?</h4> <ul> <li>The risk-reward trade-off is an important part of making intentional choices about which technologies and domains to invest in.</li> <li>Picking a stable technology that's already widely used in production is safer, but potentially less rewarding, than picking a flashy new technology not yet deployed.</li> <li>Old systems don't just die, but are replaced. As a <em>technology hospice</em>, you are the last one left to support still-critical systems, and can pretty much call the shots.</li> </ul> <h4>Tip 2: Supply and Demand</h4> <ul> <li>The supply-and-demand model predicts how price changes will affect the number of people willing to sell and purchase a product or service.</li> <li>Offshore companies do work that is in high demand. So by focusing on niche technologies, you might change the focus of the competition from price (which you cannot compete on) to ability.</li> <li>The lower price of offshore programmers drives down the overall price, thereby increasing overall demand&nbsp;– including a higher bracket of developers.</li> <li>The most important lesson from the supply and demand model is that with increased demand comes increased price competition.</li> </ul> <h4>Tip 3: Coding Don't Cut It Anymore</h4> <ul> <li>If you want to stay relevant, you must dive into the domain of the business you're in.</li> <li>You might be "just a programmer," but being able to speak to your business clients in the language of their business domain is a critical skill.</li> <li>Business domains can become hot, just like technologies. Put the same care into selecting which industry to serve as you put into which technologies to master.</li> <li>Find industry web sites you can monitor, and think about how current trends affect your company, your division, your team, and your work.</li> </ul> <h4>Tip 4: Be the Worst</h4> <ul> <li>Being the worst guy in the band means always playing with people who are better than you.</li> <li>Similarly, being the worst guy or gal on the team has the same effect as being the worst guy or gal in the band. You find that you inexplicably get smarter.</li> </ul> <h4>Tip 5: Invest in Your Intelligence</h4> <ul> <li>Good people seek out diversity, either because they love to learn new things, or being forced into alien experiences creates more mature and well-rounded software developers.</li> <li>Exposure to fringe technologies makes you deeper, better, smarter, and more creative.</li> </ul> <h4>Tip 6: Don't Listen to Your Parents</h4> <ul> <li>Your parents will give you fear-driven advice that optimizes for <em>not losing</em>. But winners take risks.</li> <li>If your job isn't fun, then you won't do a fantastic job at it. Because passion leads to excellence, and without fun there won't be passion.</li> <li>The more domains you've seen and technical architectures you've slogged through, the more prepared you are to make the right decisions on tougher projects.</li> <li>Take calculated risks in your career, and don't let fear consume you.</li> </ul> <h4>Tip 7: Be a Generalist</h4> <ul> <li>Things change in a business, and businesspeople know that software is malleable and can be changed to meet changing requirements.</li> <li>In such a rapidly changing environment, flexible people that can solve any problem that arises will excel.</li> <li>If the programmers don't understand what a system is supposed to do, then they won't be able to create a good implementation of it – regardless of any requirements, architecture, or design documents.</li> <li>If you want to be the last person standing after layoffs, then you had better make yourself generally useful.</li> <li>If you want to stand out and be remarkable, then wrapping your head around The Big Picture is where it's at.</li> <li>When it comes to hierarchical boundary crossing, it's most often not reluctance that stops people from doing it –&nbsp;it's ability.</li> <li>Longevity requires that you are platform neutral in the workplace. We all have our preferences, but leave your ideals at home.</li> </ul> <h4>Tip 8: Be a Specialist</h4> <ul> <li>The software industry has churned out a lot of shallow specialists, who use the term <em>specialist</em> as an excuse for only knowing one thing.</li> <li>Strive to be a specialist with a deep understanding of some specific area in the software field.</li> </ul> <h4>Tip 9: Don't Put All Your Eggs in Someone Else's Basket</h4> <ul> <li>We fool ourselves into thinking <em>market leader</em> is the same thing as <em>standard</em>. This helps us rationalize making another company's product part of our identity.</li> <li>You can build a business that exists as a parasite of another, but it's risky for an individual. You don't have the bandwidth or surplus of cash to suddenly change career direction or focus like a company would.</li> <li>The <em>professional services barrier</em> is the artificial barrier that a company erects between you and the solution to a problem you may have so that it can profit from selling you support services.</li> </ul> <h4>Tip 10: Love It or Leave It</h4> <ul> <li>You have to be passionate about your work if you want to be great at your work. If you don't care, it will show.</li> <li>Many people have no idea why they are in the IT field, and their performance on the job reflects it.</li> <li>Natural talent plays a big role in ability, but we can all take a big step away from mediocrity by finding work that we are passionate about.</li> <li>You can fake it for a while, but a lack of passion will catch up with you and your work.</li> </ul> <h3>Part 2: Investing in Your Product</h3> <ul> <li>To have a great product to sell on the job market, you must invest in that product.</li> <li>In business, ideas and even talent are a dime a dozen. It's the blood, sweat, tears, and money you pour into a product that makes it really worth something.</li> </ul> <h4>Tip 11: Learn to Fish</h4> <ul> <li>Education requires both a teacher and a student, and many of us are too often reluctant to be a student.</li> <li>Don't be at the mercy of others for expertise in tools or technology. Don't take expertise of these things for granted.</li> <li>Someone ignorant of their business domain lets mistakes slip through that should be easily caught, and will be slower than someone who understands the business.</li> <li>Ask yourself "How does this work?" and "Why does this happen?" to learn about things. Moreover this puts you in a new frame of mind and generates a higher level of awareness about your work environment.</li> <li>Pick a critical tool you use, and then allot yourself time each day to learn one new thing about it that will make you more productive.</li> </ul> <h4>Tip 12: Learn How Businesses Really Work</h4> <ul> <li>The basics of business finance is one body of knowledge that is neither technical nor domain-specific and won't be outdated anytime soon.</li> <li>You work for a business, and your job is to contribute to either making or saving money for that business.</li> <li>You cannot creatively help a business be profitable if you don't even understand how the business works.</li> <li>Understand whether your organization is a cost center or a profit center.</li> </ul> <h4>Tip 13: Find a Mentor</h4> <ul> <li>The first and most important purpose that a mentor serves is that of a role model.</li> <li>A role model sets the standard for what "good" means. Without a role model, there's no incentive to get better.</li> <li>A mentor can give structure to your learning process, helping you take some of the choice out of what to focus your energies on.</li> <li>A mentor is someone you can trust enough to ask "What should be different about me as a professional?" because they'll not only criticize you, but help you improve.</li> <li>As a mentor, your role is to help someone, and so you become invested in that person's success. Their success is your success as well.</li> <li>A mentor becomes a positively reinforced connection from you to his or her network, because it's not what you know, but whom you know.</li> <li>Find a role model, and define the ten most important attributes of them. Devise concrete tasks to improve on attributes you perceive as very important but you are unskilled at.</li> </ul> <h4>Tip 14: Be a Mentor</h4> <ul> <li>There is no better way to crystallize your understanding of something than to make it understandable and teachable to someone else.</li> <li>When you teach, you have to answer questions that may never have occurred to you.</li> <li>You find a mentee not by proclaiming yourself a guru but by being knowledgeable and willing to patiently share that knowledge.</li> <li>Mentoring is about people, and so an online mentoring relationship can never compare to one that happens between two people in the same place.</li> </ul> <h4>Tip 15: Practice, Practice, Practice</h4> <ul> <li>In the computer industry, it's common to find developers stretched to their limits.</li> <li>Our industry tends to practice on the job, but like many professionals, we need to make time for practice.</li> <li>If we're going to compete based on quality, we have to stop treating our jobs as a practice session. We have to invest the time in our craft.</li> <li>If you sit down to practice coding and nothing elegant comes out, you're probably not near the edges of your capabilities, where a good practice session should place you.</li> <li>To sight read code would mean to understand what you're looking at as quickly as possible.</li> <li>To sharpen your mind and improve your improvisational coding skills is to pick a program, and to write it with self-imposed constraints.</li> </ul> <h4>Tip 16: The Way That You Do It</h4> <ul> <li>Take your eye off the process of developing software, and you risk delivering late, delivering the wrong thing, or not delivering at all.</li> <li>For a software process to have any chance of being implemented successfully, it has to be embraced by the people who are using the process.</li> <li>The best way to feel ownership of processes is to help implement them.</li> <li>The methodology world is full of buzzwords, but you can always learn <em>something</em> from the study of a software process –&nbsp;even if it's what not to do.</li> <li>The best process to follow is the one that makes your team most productive and results in the best products.</li> <li>Study the available options, pick out the pieces that make sense to you and your team, and continuously refine them based on real experience.</li> <li>It's much easier to find someone who can make software work than it is to find someone who can make the <em>making of</em> software work.</li> </ul> <h4>Tip 17: On the Shoulders of Giants</h4> <ul> <li>Studying the work of masters is an essential part of becoming a master.</li> <li>Even more important than finding solutions to specific problems is the use of existing code as a magnifying mirror to inspect our own style and capabilities.</li> <li>By looking at the code of others with a critical eye, you will start to develop your own tastes, just as you would for music, art, or literature.</li> </ul> <h4>Tip 18: Automate Yourself into a Job</h4> <ul> <li>To increase software development throughput, you can get faster people to do the work, get more people to do the work, or automate the work.</li> <li>To provably make better software faster and cheaper than your offshore competition, automate your work.</li> </ul> <h3>Part 3: Executing</h3> <ul> <li>You likely work for an institution that tries to make money. Your job is to do something to help the organization meet that goal.</li> <li>To be successful, raw ability will only get you so far. The final stretch is populated by closers –&nbsp;people who finish things.</li> </ul> <h4>Tip 19: Right Now</h4> <ul> <li>Parkinson's law states that "Work expands so as to fill the time available for its completion."</li> <li>Even a manufactured sense of urgency allows you to double or triple your productivity. You get it done instead of talking about getting it done.</li> <li>Always be the one to ask "But what can we do <em>right now</em>?"</li> </ul> <h4>Tip 20: Mind Reader</h4> <ul> <li>Doing what your customers ask for will satisfy them. But doing more than they ask for, or doing things before they even ask it, will delight them.</li> <li>To hedge, only do the guesswork that you can fit in between the cracks of your normal job so the impact is little to none.</li> <li>People who can keep a project moving in the right direction without given much guidance are highly valued and appreciated by their overworked managers and customers.</li> </ul> <h4>Tip 21: Daily Hit</h4> <ul> <li>Becoming a standout performer is more likely with some specific and intentional work. Ask how you can increase the <em>visible</em> instances of this behavior.</li> <li>When you start to search for outstanding accomplishments, you naturally evaluate and prioritize your activities based on their business value.</li> <li>Tracking hits, or accomplishments, at a reasonably high frequency will ensure that you don't get stuck. Moreover you get addicted to achieving them daily.</li> </ul> <h4>Tip 22: Remember Who You Work For</h4> <ul> <li>It's hard to align your work with the goals of the business when you're buried under so many organizational layers that you hardly know what your business is.</li> <li>Start by aligning your team, which is small and focused enough that you can conceptually wrap yourself around it.</li> <li>Or else ask your manager: In a well-structured environment, your manager's goals are your team's goals, and so solving your manager's problems solves your team's problems.</li> <li>Your manager's goals are also his or her manager's goals, and so on up to the CEO –&nbsp;and so doing your small part helps fulfill the company's goals.</li> <li>A job well done by the team is a job well done by the manager. Your manager may share credit for your work, but he or she reciprocates with performance reviews, raises, and promotions.</li> <li>Understand your manager's goals for the team over the coming month, quarter, and year. Prioritize your work based on these goals.</li> </ul> <h4>Tip 23: Be Where You're At</h4> <ul> <li>If you spend your time mentally living in your next role or your next job, you're usually doing a mediocre job in your current role.</li> <li>If you're a mediocre performer with a bad attitude and you're convinced you have "high potential," beware:&nbsp;Your company doesn't make money because of potential.</li> <li>Furthermore, your attitude makes your managers want to stop investing in you.</li> <li>Even if you finally land that promotion you've been dreaming about, you'll quickly grow tired and realize that it's not this job you were meant for –&nbsp;but the <em>next</em> one.</li> <li>Keeping your mind focused on the present gets you further toward your goals than keeping your mind focused on the goal itself.</li> <li>Focusing on the present allows you to enjoy the small victories of daily work life.</li> <li>By focusing on the present you'll feel better, those around you will feel it as well, and it will show in your work.</li> </ul> <h4>Tip 24: How Good a Job Can I Do Today?</h4> <ul> <li>Boring work is usually boring for two reasons: It isn't a creative outlet, and it's not challenging.</li> <li>To make boring work both creative and challenging, make it a goal to do the boring stuff <em>perfectly</em>.</li> <li>Look at your workday and ask yourself "How good a job can I do today?"</li> </ul> <h4>Tip 25: How Much Are You Worth?</h4> <ul> <li>Assuming you cost your employer twice your salary, ask whether your positive impact on the company's bottom line is bigger than that number.</li> <li>You work for a business, and unless you provide some real value, you are a waste of money. Salary increases are not an entitlement.</li> <li>To make how much value you add concrete, ask your manager about how to best quantify it.</li> <li>The <em>time value of money</em> says that a dollar today is worth more than a dollar next year, because the dollar today can be used to make more dollars.</li> <li>Many investments have to yield an agreed-upon percentage in an agreed-upon period of time, or they aren't made. This is called the <em>hurdle rate</em>.</li> </ul> <h4>Tip 26: A Pebble in a Bucket of Water</h4> <ul> <li>In few cases does a company feel a significant impact from an employee's departure. Even in critical positions, the effect is surprisingly low.</li> <li>If you left tomorrow, the difference would be (on the average) no more or no less impactful than if any of your co-workers left.</li> <li>The more successful you are, the more likely you are to make a fatal mistake. You become arrogant, and with arrogance you develop blind spots.</li> <li>Attempting to be irreplaceable is a defensive maneuver that creates a hostile relationship with your employer where one may not have already existed.</li> <li>For any code or task where the team depends completely on you, document, automate, or refactor it so that it can be easily understood by anyone on your team.</li> </ul> <h4>Tip 27: Learn to Love Maintenance</h4> <ul> <li>We crave "project work" over "maintenance work" because we believe it allows us to be creative and is a chance to make steps toward a promotion.</li> <li>When we don't have the constraints of legacy code and lack of funding to deal with, our managers and customers expect more from us.</li> <li>If you don't deliver on project work, you've failed, and suddenly your every move is dictated from above.</li> <li>With maintenance work, you're only expected to keep the software running smoothly and for as little money as possible.</li> <li>As long as you're keeping things running and responding to user requests in a timely fashion, maintenance mode is a place of freedom and creativity.</li> <li>In maintenance mode you can interact directly with customers. So you can build a larger base of advocates in your business, and learn the inner workings of your business.</li> <li>Moreover project work is maintenance: As soon as you've written your first line of code, each additional feature is grafted onto an existing code base.</li> </ul> <h4>Tip 28: Eight-Hour Burn</h4> <ul> <li>Rebranding "forty-hour workweek" to "eight-hour burn" conveys that you should work so relentlessly that there is no way you could work longer than eight hours.</li> <li>When we're tired, we can't think as effectively as when we're rested. We're not as creative and the quality of our work decreases dramatically.</li> <li>When you have too much time to work, your work time reduces significantly in perceived value. In a longer work week, each hour is less precious.</li> <li>With strict barriers on start and end times to your work, you naturally start to organize your tasks more effectively.</li> <li>If you work intensely every day and set boundaries, you'll find that your work doesn't follow you home.</li> </ul> <h4>Tip 29: Learn How to Fail</h4> <ul> <li>A craftsperson is really put to the test when errors arise, but learning to deal with mistakes is a skill that is both highly valuable and difficult to teach.</li> <li>We all make mistakes, so within reason, we don't judge each other on the mistakes we make. We judge each other on how we respond to mistakes.</li> <li>When a mistake happens, follow these rules:</li> <li>Raise the issue as early as you know it. The earlier you expose what you've done, the smaller the negative impact.</li> <li>Take the blame. A problem needs a resolution, and lingering on who's fault it is only prolongs the issue.</li> <li>Offer a solution. Even small and immaterial concrete and attainable goals move things from bad to good, and help rebuild credibility.</li> <li>Ask for help. Your team members, management, and customers will look at you more positively if you can maintain a healthy attitude and set your ego aside.</li> <li>A mistake handled well might make us more loyal customers than we would have been had we never experienced a service problem.</li> </ul> <h4>Tip 30: Say "No"</h4> <ul> <li>The quickest path to missing your commitments is to make commitments that you know you can't meet.</li> <li>A <em>can-do</em> attitude is different from misrepresenting one's abilities: The latter causes problems for both you and your stakeholders.</li> <li>We are programmed to want to always succeed, and saying we can't do something makes it feel like we failed.</li> <li>We know it to be true, but fail to internalize, that "yes" is not always the right answer, and "no" is seldom the wrong answer.</li> <li>If someone has the strength to say "no" when that's the truth, then you know that when they say "yes," they really mean it.</li> <li>A guru in a subject area is never afraid to admit when they don't know something, because "I don't know" is not a phrase for the insecure.</li> <li>The same courage is useful when dealing with decision from above, as people who speak up and offer better suggestions become trusted lieutenants.</li> </ul> <h4>Tip 31: Don't Panic</h4> <ul> <li>If you look back on every "disaster," you'll likely find that not a single one has made a lasting or noticeable impact on you or your career.</li> <li>Panicking creates an inability to perform at your best at the times when you really need to be performing at your best.</li> <li>When something is going wrong, we focus all of our attention on the problem. This makes the problem, no matter how small, seem more important than it is.</li> <li>To reduce panic when something bad happens, analyze the situation from a third-person perspective, and try to laugh at yourself.</li> </ul> <h4>Tip 32: Say It, Do It, Show It</h4> <ul> <li>When you have too much to do, a plan can make the difference between confused ambiguity and clear-headed confidence at the start of a workday.</li> <li>A simple plan that allows you to answer the question "What am I going to do today?" at the start of the day is a great first step.</li> <li>To create such a plan, take time in the afternoon to list in priority order everything you want to get done on the next workday.</li> <li>The larger the time span you're planning for, the higher level your plan should be.</li> <li>Start communicating your plans to management after executing one cycle of the plan. And start doing it before they ask you to do it.</li> <li>Every item in your plan must be completed, delayed, removed, or replaced. No items should go unaccounted for.</li> <li>If items show up on a plan and are never mentioned again, people will stop trusting your plans, thereby counteracting the effectiveness of planning.</li> <li>We all make mistakes. Differentiate yourself by addressing your mistakes or inabilities publicly and ask for help resolving them.</li> <li>Creating and executing plans shows that you are not just a robot typing code, but a <em>leader</em>.</li> <li>If you say what you're going to do and then you do it and show that it's done, you develop a reputation as a <em>doer</em>, and with credibility comes influence.</li> <li>Your leaders want you to have independence and ownership. Making, executing, and communicating plans will help you attain both.</li> </ul> <h3>Part 4: Executing</h3> <ul> <li>A leader must develop teams that deliver maximum value to a company. A leader can't do that without knowing who is capable of doing what kind of work.</li> <li>Manager's don't have time to keep close tabs on what each employee is doing every day. They instead focus on the big picture.</li> <li>Don't assume management knows who you are, or that just because a leader knows your name he or she has the faintest understanding of your capabilities.</li> </ul> <h4>Tip 33: Perceptions, Perschmeptions</h4> <ul> <li>You should care about what other people think of you. Perception is reality.</li> <li>There's no way to objectively measure the quality of a knowledge worker or the quality of their work, so you'll always be measured based on someone else's <em>perception</em> of you.</li> <li>People in different roles and relationships to you build their perceptions based on the qualities most important to making <em>that particular</em> relationship work well.</li> <li>Managing perception is practical: When you explicitly take note of the factors that drive other people's perceptions of you, you more firmly discover how to make them happy customers.</li> </ul> <h4>Tip 34: Adventure Tour Guide</h4> <ul> <li>It's obvious, but the most important aspect of getting the word out in the workplace is your ability to communicate.</li> <li>Customers (or managers) are looking to you to make them comfortable about the project they're working on.</li> <li>You must be your customer's tour guide through the unforgiving terrain of the information technology world. You will make your customers comfortable while guiding them through an unfamiliar place.</li> <li>When promotion and staffing decisions are being made, the best advocate for you is a customer who can't live without you.</li> <li>Don't forget that your customer represents the needs of the business, and you are paid to provide for those needs.</li> </ul> <h4>Tip 35: Me Rite Reel Nice</h4> <ul> <li>In one report, more than half of responding companies consider writing skills when making hiring and promotion decisions.</li> <li>Forty percent of surveyed companies in the service sector said that a third or fewer of their new hires had the writing skills they desired.</li> <li>Even if you're a great coder, you won't be very effective in a distributed team if you can't express yourself in words.</li> <li>Communication, especially through writing, is the bottleneck through which all your wonderful ideas must pass. You are what you can explain.</li> </ul> <h4>Tip 36: Being Present</h4> <ul> <li>Being face to face with your leaders and your customers is an advantage. Don't squander it.</li> <li>We not only see greater productivity and enhanced communication from face-to-face interactions, but we also form tighter bonds.</li> <li>In the majority of cases, the relationship built in a remote work environment will remain strictly centered around accomplishing tasks.</li> <li>The human tendency is to want to work with other humans –&nbsp;not voicemail, email, or instant messaging but actual people.</li> <li>In a distributed environment, if making regular trips for face-to-face meetings is impractical, pick up the phone and call your bosses and co-workers instead.</li> </ul> <h4>Tip 37: Suit Speak</h4> <ul> <li>People who run businesses are interested in business <em>results</em>, so marketing your accomplishments in any language other than the language of the business is ineffective.</li> <li>To sell a product to an audience, you must speak to that audience in a language they can both understand and relate to.</li> <li>As a software developer, that means framing your accomplishments in the context of the business you work for.</li> <li>To put this into practice, make a list of your recent accomplishments and write the business benefit of each.</li> </ul> <h4>Tip 38: Change the World</h4> <ul> <li>When you come to work without a mission, you do only what people tell you to do. And so the only ones who know what you've done are the ones who asked you to do it.</li> <li>You must have a mission to be a good software developer in a high-cost country. You must effect visible change through your team, organization, or company.</li> <li>You must do things you're internally <em>driven</em> to do. It's because you know things could be better, and so you <em>have</em> to change them.</li> <li>If you don't know what your crusade is, you probably don't have one.</li> </ul> <h4>Tip 39: Let Your Voice Be Heard</h4> <ul> <li>Don't think of yourself as a programmer at a specific company, but as a participating member of an industry. You are a craftsperson or an artist.</li> <li>No objective system exists for rating and employing software developers. Being good is important, but doesn't get you all the way there.</li> <li>Publishing and public speaking are the best ways for your name to be propagated and your voice to be heard.</li> <li>Write on the web as if you were writing a feature column for your favorite magazine.</li> <li>More writing leads to more writing opportunities. And all of these lead to the opportunity to speak at conferences.</li> <li>Start sooner than you think you're ready: Most people undersell themselves, and you definitely have something to teach!</li> </ul> <h4>Tip 40: Build Your Brand</h4> <ul> <li>Brand building has two parts: Making your mark so that people will recognize it, and then ensuring it's associated with positive traits.</li> <li>The most potentially destructive force for your brand, or that recognition and respect, is yourself.</li> <li>The things you choose to do and associate with yourself will have a lasting impact on what your name means to people. And Google never forgets.</li> </ul> <h4>Tip 41: Release Your Code</h4> <ul> <li>Open source developers are building a name for themselves, a reputation in the industry, and are marketing themselves in the process.</li> <li>It takes skills in leadership, release management, documentation, and product and community support to foster a community around your efforts.</li> </ul> <h4>Tip 42: Remarkability</h4> <ul> <li>Traditional marketing curricula refer to the four <em>p</em>'s of marketing: product, price, promotion, and placement.</li> <li>The goal of marketing is to create a connection between producers and consumers of a product or service.</li> <li>To be remarkable means that something is worthy of attention. You will not become a remarkable software developer by simply being better than all the other software developers you know.</li> <li>To be remarkable, you must be significantly different from those around you.</li> <li>You might be the smartest or the fastest software developer, but just <em>being</em> isn't good enough. You have to be <em>doing something</em>.</li> </ul> <h4>Tip 43: Making the Hang</h4> <ul> <li>Really good people won't mind if you want to know them, because such people like to be appreciated and like to talk about the topics they're passionate about.</li> <li>Most of us are afraid to try and associate with smart, well-connected people who can teach you things or find work – even thought its' best for you.</li> <li>The gurus are the supernodes in the social and professional network, and connecting with them just takes a little less humility.</li> <li>Some of the most influential minds in software development are readily accessible via email or real-time chat.</li> </ul> <h3>Part 5: Maintaining Your Edge</h3> <ul> <li>In your career you must research, invest, execute, market, repeat. Spending too much time in any iteration of the loop puts you at risk of becoming obsolete.</li> </ul> <h4>Tip 44: Already Obsolete</h4> <ul> <li>The more successful your business, the more comfortable you are with your business model, leaving you vulnerable to others with radical ideas.</li> <li>The more mainstream your knowledge, the greater risk you have of being left in the technology stone age.</li> <li>If you're on the bleeding edge of today's wave, you're probably behind on the next one. Start thinking <em>ahead</em> with your study.</li> <li>Learn something on the bleeding edge. Worst case you learn something enriching but not applicable later. Best case you remain ahead of the curve.</li> </ul> <h4>Tip 45: You've Already Lost Your Job</h4> <ul> <li>If your surroundings are changing and the context of your work is constantly moving, clinging to your work creates an unhealthy dissonance that infects your work.</li> <li>Have ambition but don't buy too heavily into a long, imagined future. You cannot afford tunnel vision with something too far off.</li> </ul> <h4>Tip 46: Path with No Destination</h4> <ul> <li>We are a goal-oriented society, obsessed with the <em>outcome</em> of a process. But we spend all our time <em>doing</em> things and little time actually reaching goals.</li> <li>The meat of your career is not promotions and salary advances, but the time you spend working <em>regardless</em> of the advances.</li> <li>If this is the core of your work life – the actual work – then you've already arrived at your destination.</li> <li>A goal-oriented mindset leads only from one goal to the next. It has no logical end, and what most of us fail to realize is that <em>the path</em> is the end.</li> <li>Focusing on the ending makes you forget to make the process good. And bad processes create bad products.</li> <li>Bad products also make bad processes: Once you have one of these products that is messy inside, your processes wrap around it.</li> </ul> <h4>Tip 47: Make Yourself a Map</h4> <ul> <li>Create a road map for your career like for a product, or "your personal product road map." Use it to tell whether or not you've moved in your career.</li> <li>Throw out some markers you can see in the distance, so you'll know that you've moved when you get to them. Your product "features" are these markers.</li> <li>Think of each new set of knowledge or capability as a single feature in an application. An application with one feature isn't much of an application.</li> <li>Moreover, an application with a bunch of features that aren't cohesive is going to confuse its users.</li> <li>Although it's okay to learn diverse skills&nbsp;– it expands your thinking –&nbsp;it's also a good idea to think about the story your skill set tells.</li> </ul> <h4>Tip 48: Watch the Market</h4> <ul> <li>As the conditions of the market change, not paying attention could result in money lost or money that <em>could</em> have been earned missed.</li> <li>The same holds true for your knowledge investments.</li> <li>If you are comfortable in your current job with your current set of skills, you might remain blissfully ignorant of the Next Big Thing as it rolls in.</li> <li>Monitor the alpha geeks, or those super-nerds who are always on the bloodiest tip of the bleeding edge&nbsp;– at least in their hobby activities.</li> </ul> <h4>Tip 49: That Fat Man in the Mirror</h4> <ul> <li>If you're constantly exposed to something, it's hard to see it changing unless change happens rapidly.</li> <li>There is no objective, universal scale for measuring your marketability or skill as a software developer, so you'll have to develop your own.</li> <li>An easy way to measure your progress is to use a trusted third party, such as a mentor or a close colleague.</li> <li>The most important thing is to ferret out your blind spots. You don't have to fix all of them, but you must know where they are.</li> <li>Schedule time for reviews, as you won't reflect unless you make the reflection time explicit.</li> <li>Once you have a system and have scheduled time for reviews, capture the results in writing to concretize them. Review and revise the evaluation often.</li> </ul> <h4>Tip 50: The South Indian Monkey Trap</h4> <ul> <li>Value rigidity is when you believe in the value of something so strongly that you can no longer objectively question it.</li> <li>Not all rigidly held values are good ones. And many times something that is good in one set of circumstances is not good in another.</li> <li>Since we live every day in our careers, it's easy to develop value rigidity in our career choices: We know what has worked, and we keep doing it.</li> <li>Beware of value rigidity leading you to make poor choices in the direction your career is taking or the technologies you advocate and invest in.</li> <li>Do a project in the technology you hate most. Best case you'll learn it isn't that bad, and worst case you'll have better fodder for your arguments.</li> </ul> <h4>Tip 51: Avoid Waterfall Career Planning</h4> <ul> <li>The larger and more complex a project is, the less likely it is to imagine every feature in detail well enough to create a specification.</li> <li>Changing your direction from management to programming, or vice versa, isn't hard. Neither is finding a new company to work for, or moving to a different city.</li> <li>Change is not only possible in your career, but necessary.</li> <li>Set big goals, but make constant corrections along the way. Learn from the experience, and change the goals as you go.</li> </ul> <h4>Tip 52: Better Than Yesterday</h4> <ul> <li>We become demotivated by the complexity of bigger issues and instead turn our attention to things that are easier to measure and easier to quickly fix. This is why we procrastinate.</li> <li>Focus on making whatever it is you're trying to improve better today than it was yesterday. This is an easier goal.</li> <li>Making one small improvement is motivating: You can clearly see the difference in that <em>one thing</em> you've fixed as soon as the change is made.</li> <li>For big, difficult goals, individual improvements won't often lead directly to tangible results. So don't think about <em>getting closer</em> to the goal each day, but <em>doing better</em> in your efforts each day.</li> <li>Make your improvements small but daily. Small improvements also decrease the cost of failure: If you miss a day, you have a new baseline for tomorrow.</li> <li>You can apply this approach to very tactical goals, or you can apply it to the very highest level of goals that you have.</li> </ul> <h4>Tip 53: Go Independent</h4> <ul> <li>The safety blanket of corporate hierarchy slows you down: When you can hide behind the shield of mediocrity in a corporate division, there is less incentive to excel.</li> <li>Becoming an independent contractor is an ultimate test: It's only you, your expertise, and your ability to execute.</li> <li>Going independent also forces you to learn how to market yourself and at the same time tests your choices in domain and technology to focus on.</li> <li>A startup gives you a full-time salary and the challenge of being pitted directly against the unfiltered problems of your business.</li> </ul> https://tedneward.github.io/Research/reading/development/reversing/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/reversing/index.html <h2>Books</h2> <ul> <li><a href="https://beginners.re/">Reverse Engineering for Beginners</a> (<a href="https://beginners.re/RE4B-EN.pdf">Book PDF</a>)</li> <li><a href="https://github.com/iosre/iOSAppReverseEngineering">iOS App Reverse Engineering</a> (PDF)</li> <li><a href="http://bioshacking.blogspot.co.uk/2012/02/bios-disassembly-ninjutsu-uncovered-1st.html">BIOS Disassembly Ninjutsu Uncovered 1st Edition</a> - Darmawan Salihun (PDF)</li> <li><a href="https://www.nostarch.com/xboxfree/">Hacking the Xbox: An Introduction to Reverse Engineering</a> - Andrew "bunnie" Huang</li> </ul> <h2>Articles</h2> https://tedneward.github.io/Research/reading/development/open-source/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/open-source/index.html <h2>Licenses</h2> <h2>Places</h2> <p>Places that focus on the topic of OSS itself in a variety of ways.</p> <p><a href="https://chaoss.community/">CHAOSS</a> (Community Health Analytics in Open Source Software): CHAOSS is a Linux Foundation project focused on creating metrics, metrics models, and software to better understand open source community health on a global scale. CHAOSS is an acronym for Community Health Analytics in Open Source Software. Open source software is critically important for both individuals and organizations. This importance raises questions about how we understand the health of the open-source projects we rely on. Unhealthy projects can have negative impacts on the community involved in the project as well as organizations that rely on such projects.</p> <ul> <li> <p><a href="https://chaoss.community/about-chaoss-practitioner-guides/">Practitioner Guides</a>:</p> <ul> <li><a href="https://chaoss.community/practitioner-guide-introduction/">Introduction - Things to Think about When Interpreting Metrics</a> &lt;- Please start here</li> <li><a href="https://chaoss.community/practitioner-guide-contributor-sustainability/">Getting Started with Contributor Sustainability</a></li> <li><a href="https://chaoss.community/practitioner-guide-responsiveness/">Getting Started with Responsiveness</a></li> <li><a href="https://chaoss.community/practitioner-guide-organizational-participation/">Getting Started with Organizational Participation</a></li> <li><a href="https://chaoss.community/practitioner-guide-security/">Getting Started with Security</a></li> <li><a href="https://chaoss.community/practitioner-guide-sunset">Getting Started with Sunsetting an Open Source Project</a></li> <li><a href="https://chaoss.community/practitioner-guide-diverse-leadership">Getting Started with Building Diverse Leadership</a></li> <li><a href="https://chaoss.community/practitioner-guide-demonstrating-org-value">Demonstrating Organizational Value</a></li> <li><a href="https://chaoss.community/practitioner-guide-viability">Assessing Viability</a></li> </ul> </li> <li> <p><a href="https://chaoss.community/kb-metrics-and-metrics-models/">Metrics and Models</a>:</p> <ul> <li><a href="https://chaoss.community/kb/metrics-model-development-responsiveness/">Metrics Model: Development Responsiveness</a></li> <li><a href="https://chaoss.community/kb/metrics-model-collaboration-development-index/">Metrics Model: Collaboration Development Index</a></li> <li><a href="https://chaoss.community/kb/metrics-model-dei-event-badging/">Metrics Model: DEI Event Badging</a></li> <li><a href="https://chaoss.community/kb/metrics-model-community-service-and-support/">Metrics Model: Community Service and Support</a></li> <li><a href="https://chaoss.community/kb/metrics-model-project-engagement/">Metrics Model: Project Engagement</a></li> <li><a href="https://chaoss.community/kb/metrics-model-funding/">Metrics Model: Funding</a></li> <li><a href="https://chaoss.community/kb/metrics-model-project-awareness/">Metrics Model: Project Awareness</a></li> <li><a href="https://chaoss.community/kb/metrics-model-starter-project-health/">Metrics Model: Starter Project Health</a></li> <li><a href="https://chaoss.community/kb/metrics-model-community-activity/">Metrics Model: Community Activity</a></li> <li><a href="https://chaoss.community/kb/metrics-model-business-readiness-of-an-open-source-project/">Metrics Model: Business Readiness of an Open Source Project</a></li> <li><a href="https://chaoss.community/kb/metrics-model-oss-project-viability-compliance-security/">Metrics Model: OSS Project Viability: Compliance + Security</a></li> <li><a href="https://chaoss.community/kb/metrics-model-oss-project-viability-governance/">Metrics Model: OSS Project Viability: Governance</a></li> <li><a href="https://chaoss.community/kb/metrics-model-oss-project-viability-community/">Metrics Model: OSS Project Viability: Community</a></li> <li><a href="https://chaoss.community/kb/metrics-model-oss-project-viability-strategy/">Metrics Model: OSS Project Viability: Strategy</a></li> <li><a href="https://chaoss.community/kb/metrics-model-project-viability-starter/">Metrics Model: Project Viability Starter</a></li> <li><a href="https://chaoss.community/kb/metrics-model-community-welcomingness/">Metrics Model: Community Welcomingness</a></li> <li><a href="https://chaoss.community/kb/metrics-model-safety/">Metrics Model: Safety</a></li> <li><a href="https://chaoss.community/kb/metric-conversion-rate/">Metric: Conversion Rate</a></li> <li><a href="https://chaoss.community/kb/metric-meeting-attendee-count/">Metric: Meeting Attendee Count</a></li> <li><a href="https://chaoss.community/kb/metric-chat-platform-inclusivity/">Metric: Chat Platform Inclusivity</a></li> <li><a href="https://chaoss.community/kb/metric-issue-label-inclusivity/">Metric: Issue Label Inclusivity</a></li> <li><a href="https://chaoss.community/kb/metric-self-merge-rates/">Metric: Self-Merge Rates</a></li> <li><a href="https://chaoss.community/kb/metric-collaboration-platform-activity/">Metric: Collaboration Platform Activity</a></li> <li><a href="https://chaoss.community/kb/metric-open-source-security-foundation-openssf-best-practices-badge/">Metric: Open Source Security Foundation (OpenSSF) Best Practices Badge</a></li> <li><a href="https://chaoss.community/kb/metric-test-coverage/">Metric: Test Coverage</a></li> <li><a href="https://chaoss.community/kb/metric-libyears/">Metric: Libyears</a></li> <li><a href="https://chaoss.community/kb/metric-programming-language-distribution/">Metric: Programming Language Distribution</a></li> <li><a href="https://chaoss.community/kb/metric-activity-dates-and-times/">Metric: Activity Dates and Times</a></li> <li><a href="https://chaoss.community/kb/metric-time-to-first-response/">Metric: Time to First Response</a></li> <li><a href="https://chaoss.community/kb/metric-review-cycle-duration-within-a-change-request/">Metric: Review Cycle Duration within a Change Request</a></li> <li><a href="https://chaoss.community/kb/metric-branch-lifecycle/">Metric: Branch Lifecycle</a></li> <li><a href="https://chaoss.community/kb/metric-change-requests/">Metric: Change Requests</a></li> <li><a href="https://chaoss.community/kb/metric-change-requests-accepted/">Metric: Change Requests Accepted</a></li> <li><a href="https://chaoss.community/kb/metric-occasional-contributors/">Metric: Occasional Contributors</a></li> <li><a href="https://chaoss.community/kb/metric-event-locations/">Metric: Event Locations</a></li> <li><a href="https://chaoss.community/kb/metric-licenses-declared/">Metric: Licenses Declared</a></li> <li><a href="https://chaoss.community/kb/metric-bot-activity/">Metric: Bot Activity</a></li> <li><a href="https://chaoss.community/kb/metric-time-to-close/">Metric: Time to Close</a></li> <li><a href="https://chaoss.community/kb/metric-sponsorship/">Metric: Sponsorship</a></li> <li><a href="https://chaoss.community/kb/metric-documentation-discoverability/">Metric: Documentation Discoverability</a></li> <li><a href="https://chaoss.community/kb/metric-change-requests-declined/">Metric: Change Requests Declined</a></li> <li><a href="https://chaoss.community/kb/metric-osi-approved-licenses/">Metric: OSI Approved Licenses</a></li> <li><a href="https://chaoss.community/kb/metric-spdx-document/">Metric: SPDX Document</a></li> <li><a href="https://chaoss.community/kb/metric-number-of-downloads/">Metric: Number of Downloads</a></li> <li><a href="https://chaoss.community/kb/metric-code-of-conduct-at-event/">Metric: Code of Conduct at Event</a></li> <li><a href="https://chaoss.community/kb/metric-event-accessibility%ef%bf%bc/">Metric: Event Accessibility</a></li> <li><a href="https://chaoss.community/kb/metric-time-inclusion-for-virtual-events/">Metric: Time Inclusion for Virtual Events</a></li> <li><a href="https://chaoss.community/kb/metric-board-council-diversity/">Metric: Board/Council Diversity</a></li> <li><a href="https://chaoss.community/kb/metric-inclusive-leadership/">Metric: Inclusive Leadership</a></li> <li><a href="https://chaoss.community/kb/metric-mentorship/">Metric: Mentorship</a></li> <li><a href="https://chaoss.community/kb/metric-documentation-usability/">Metric: Documentation Usability</a></li> <li><a href="https://chaoss.community/kb/metric-documentation-accessibility/">Metric: Documentation Accessibility</a></li> <li><a href="https://chaoss.community/kb/metric-contributor-absence-factor/">Metric: Contributor Absence Factor</a></li> <li><a href="https://chaoss.community/kb/metric-change-request-commits/">Metric: Change Request Commits</a></li> <li><a href="https://chaoss.community/kb/metric-change-request-reviews/">Metric: Change Request Reviews</a></li> <li><a href="https://chaoss.community/kb/metric-event-location-inclusivity/">Metric: Event Location Inclusivity</a></li> <li><a href="https://chaoss.community/kb/metric-psychological-safety/">Metric: Psychological Safety</a></li> <li><a href="https://chaoss.community/kb/metric-organizational-diversity/">Metric: Organizational Diversity</a></li> <li><a href="https://chaoss.community/kb/metric-job-opportunities/">Metric: Job Opportunities</a></li> <li><a href="https://chaoss.community/kb/metric-newcomer-experience/">Metric: Newcomer Experience</a></li> <li><a href="https://chaoss.community/kb/metric-contributor-location/">Metric: Contributor Location</a></li> <li><a href="https://chaoss.community/kb/metric-committers/">Metric: Committers</a></li> <li><a href="https://chaoss.community/kb/metric-technical-fork/">Metric: Technical Fork</a></li> <li><a href="https://chaoss.community/kb/metric-burstiness/">Metric: Burstiness</a></li> <li><a href="https://chaoss.community/kb/metric-code-changes-lines/">Metric: Code Changes Lines</a></li> <li><a href="https://chaoss.community/kb/metric-change-request-acceptance-ratio/">Metric: Change Request Acceptance Ratio</a></li> <li><a href="https://chaoss.community/kb/metric-change-requests-duration/">Metric: Change Requests Duration</a></li> <li><a href="https://chaoss.community/kb/metric-issues-new/">Metric: Issues New</a></li> <li><a href="https://chaoss.community/kb/metric-issues-closed/">Metric: Issues Closed</a></li> <li><a href="https://chaoss.community/kb/metric-issue-resolution-duration/">Metric: Issue Resolution Duration</a></li> <li><a href="https://chaoss.community/kb/metric-issue-age/">Metric: Issue Age</a></li> <li><a href="https://chaoss.community/kb/metric-inactive-contributors/">Metric: Inactive Contributors</a></li> <li><a href="https://chaoss.community/kb/metric-license-coverage/">Metric: License Coverage</a></li> <li><a href="https://chaoss.community/kb/metric-clones/">Metric: Clones</a></li> <li><a href="https://chaoss.community/kb/metric-upstream-code-dependencies/">Metric: Upstream Code Dependencies</a></li> <li><a href="https://chaoss.community/kb/metric-elephant-factor/">Metric: Elephant Factor</a></li> <li><a href="https://chaoss.community/kb/metric-diversity-access-tickets/">Metric: Diversity Access Tickets</a></li> <li><a href="https://chaoss.community/kb/metric-inclusive-experience-at-event/">Metric: Inclusive Experience at Event</a></li> <li><a href="https://chaoss.community/kb/metric-family-friendliness/">Metric: Family Friendliness</a></li> <li><a href="https://chaoss.community/kb/metric-event-demographics/">Metric: Event Demographics</a></li> <li><a href="https://chaoss.community/kb/metric-code-of-conduct-for-a-project/">Metric: Code of Conduct for a Project</a></li> <li><a href="https://chaoss.community/kb/defect-resolution-duration/">Metric: Defect Resolution Duration</a></li> <li><a href="https://chaoss.community/kb/metric-code-changes-commits/">Metric: Code Changes Commits</a></li> <li><a href="https://chaoss.community/kb/metric-release-frequency/">Metric: Release Frequency</a></li> <li><a href="https://chaoss.community/kb/metric-public-health-and-safety/">Metric: Public Health and Safety</a></li> <li><a href="https://chaoss.community/kb/metric-project-burnout/">Metric: Project Burnout</a></li> <li><a href="https://chaoss.community/kb/metric-project-velocity/">Metric: Project Velocity</a></li> <li><a href="https://chaoss.community/kb/metric-project-recommendability/">Metric: Project Recommendability</a></li> <li><a href="https://chaoss.community/kb/metric-organizational-influence/">Metric: Organizational Influence</a></li> <li><a href="https://chaoss.community/kb/metric-organizational-project-skill-demand/">Metric: Organizational Project Skill Demand</a></li> <li><a href="https://chaoss.community/kb/metric-change-request-closure-ratio/">Metric: Change Request Closure Ratio</a></li> <li><a href="https://chaoss.community/kb/metric-project-demographics/">Metric: Project Demographics</a></li> <li><a href="https://chaoss.community/kb/metric-communication-transparency/">Metric: Communication Transparency</a></li> <li><a href="https://chaoss.community/kb/metric-change-request-review-duration/">Metric: Change Request Review Duration</a></li> <li><a href="https://chaoss.community/kb/metric-project-access/">Metric: Project Access</a></li> <li><a href="https://chaoss.community/kb/metric-issues-active/">Metric: Issues Active</a></li> <li><a href="https://chaoss.community/kb/metric-issue-response-time/">Metric: Issue Response Time</a></li> <li><a href="https://chaoss.community/kb/metric-new-contributors-closing-issues/">Metric: New Contributors Closing Issues</a></li> <li><a href="https://chaoss.community/kb/metric-contribution-attribution/">Metric: Contribution Attribution</a></li> <li><a href="https://chaoss.community/kb/metric-labor-investment/">Metric: Labor Investment</a></li> <li><a href="https://chaoss.community/kb/metric-new-contributors/">Metric: New Contributors</a></li> <li><a href="https://chaoss.community/kb/metric-academic-open-source-project-impact/">Metric: Academic Open Source Project Impact</a></li> <li><a href="https://chaoss.community/kb/metric-project-popularity/">Metric: Project Popularity</a></li> <li><a href="https://chaoss.community/kb/metric-contributors/">Metric: Contributors</a></li> <li><a href="https://chaoss.community/kb/metric-types-of-contributions/">Metric: Types of Contributions</a></li> <li><a href="https://chaoss.community/kb/metric-conflict-resolution-and-mediation/">Metric: Conflict Resolution and Mediation</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/philosophical-programmer/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/philosophical-programmer/index.html <h1>Part I: A Philosophical Introduction</h1> <ol> <li> <p>Beyond the Cuckoo Clock</p> <ul> <li> <p>Man is a tool-using creature</p> </li> <li> <p>First tools were extensions of our physical abilities</p> </li> <li> <p>Medieval clock was first extension of our mind</p> </li> <li> <p>Cuckoo clocks are analog computers--take input (swings of pendulum) and turn them into data (time representation)</p> </li> <li> <p>Computers are digital computers, and infinitiely flexible (unike analog computers)</p> </li> </ul> </li> <li> <p>Is There an Aesthetic of Programming?</p> <ul> <li>discussion of algorithms as aesthetics</li> </ul> </li> <li> <p>The Ethical Quotient</p> <ul> <li> <p>ethical challenges: qualities of magnification, precision, alienation, believability, malleability of information, impermeability, and autonomy of operation</p> </li> <li> <p>magnification: computers magnify our thought processes significantly, but have no conscience or controls beyond what we put on them</p> </li> <li> <p>precision: instructions to a computer must be precise</p> </li> <li> <p>alientation: computer isolates programmers and operators from the people affected by their actions/code. "Instead of seein the program as a real instrument affecting the lives of real people, programmers sometimes see it as a game, a challenge to their ingenuity. The alienating quality of the computer permits us to overlook the human consequences of a programming decision or error." This happens to users on the Internet as well.</p> </li> <li> <p>believability: we believe the computer because it presents us with too much information to be able to effectively doubt any of it</p> </li> <li> <p>malleability: data can change without us knowing it</p> </li> <li> <p>impermeability: we cannot observe the computer, only its side effects and results</p> </li> <li> <p>autonomy of operation: the computer is capable of acting without our intervention or supervision</p> </li> </ul> </li> </ol> <h1>Part II: The Structure of the Computer</h1> <ol> <li>Types of Computers</li> <li>The Parts of a Computer</li> <li>The Construction of Memory</li> <li>On a Clear Disk You Can Seek Forever</li> <li>A Brief Interruption</li> <li>Operating Systems</li> </ol> <h1>Part III: Fundamental Tools of Programming</h1> <ol> <li>The Language of the Machine</li> <li>Forms of Data Definition</li> <li>Classes and Types of Statements</li> <li>The Functional Program</li> <li>A Short Commentary</li> <li>Algorithms and Objects</li> </ol> <h1>Part IV: The Programmer's Trade</h1> <ol> <li>The Moth in the Machine</li> <li>The Real World Out There</li> <li>The Limitations of Design</li> <li>Programming as Abstraction and Reflection</li> </ol> https://tedneward.github.io/Research/reading/development/roles/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/roles/index.html <h2>Technical Program Manager (TPM)</h2> <p><a href="https://newsletter.pragmaticengineer.com/">Pragmatic Engineer</a></p> <ul> <li><a href="https://newsletter.pragmaticengineer.com/p/what-tpms-do">"What TPMs Do and What Software Engineers Can Learn From Them"</a></li> <li><a href="https://newsletter.pragmaticengineer.com/p/scaling-engineering-with-the-tpm-role">"Scaling Engineering Organizations with the Technical Program Manager Role"</a> (requires paid access)</li> </ul> https://tedneward.github.io/Research/reading/development/governance/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/governance/index.html <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.infoworld.com/article/4090521/why-software-development-slows-to-a-crawl.html">Why software development slows to a crawl</a>: Software organizations don’t slow down because people stop caring. They slow down because responsibility diffuses across owners, reviewers, and committees, while the work required to push anything forward grows heavier each quarter. The challenge for leaders is to continuously tune governance so that clarity and speed increase together. ... Teams fall into a trap where a once-useful rule becomes a tradition, then a barrier. When someone asks why a particular approval exists, the answer becomes circular: because that’s our process. Good governance adapts to current needs, not historical ones. For every process gate, someone should articulate what specific risk it mitigates and what would happen without it. If the answer is “we’ve always done it this way,” the gate needs reevaluation. The most successful engineering cultures build in mechanisms for process retirement. Any new process comes with an expiration date. After six or 12 months, the rule is automatically retired unless someone actively argues for renewal with data showing its necessity. Every rule must have a documented owner and a one-sentence justification. If the current team cannot state the why, the process should be retired.</li> </ul> https://tedneward.github.io/Research/reading/development/legacy-migration/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/legacy-migration/index.html <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://leaddev.com/technical-direction/measure-first-migrate-later-lessons-from-a-230-velocity-boost">Measure first, migrate later</a></li> </ul> https://tedneward.github.io/Research/reading/development/message-passing/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/message-passing/index.html <p>Message-passing is often at the heart of dynamic programming languages like <a href="/languages/objc.html">Objective-C</a>. For example, the <a href="https://developer.apple.com/documentation/objectivec/objective-c_runtime">ObjC runtime</a> (also described <a href="https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html#//apple_ref/doc/uid/TP40008048">here</a>) is a library for creating instances of objects, obtaining "selectors" to the names of message-receivers (methods) that the object can process, and even do some <a href="metaobjectprotocol.html">MOP</a> operations on the objects.</p> <p>In more widespread distributed systems literature, this is often called "message-oriented middleware" and typically involves messages dispatched over network links to "queues" where they are stored until examined (and possibly consumed) by messaging clients. Those who produce messages are often called "producers"; those who consume them, "consumers".</p> <h2>Reading</h2> https://tedneward.github.io/Research/reading/development/frictionless/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/frictionless/index.html <p><em>(Nicole Forsgren, Abi Noda; Shift Key Press Nov 2025; ISBN 978-1662966378)</em></p> <p><em>(As a general rule, this book is following the social media style of authorship, at a slightly larger scale: Instead of one sentence paragraphs, they're doing one- or two-page chapters. Overall, I'm not entirely impressed so far, particularly since I think a good chunk of the "seven steps" are just a retelling of Kotter's Change Management steps. Actually, as I go through the material, I begin to suspect a good chunk of it was written by an LLM. The case studies and examples probably aren't, but a lot of the prose in between feels very LLM-ish. Stuff in Step 3 feels more genuine and additive.)</em></p> <h1>PART I Understanding DevEx</h1> <h2>Chapter 1 - What is Developer Experience?</h2> <p>Developer experience isn’t just about making developers happy--it’s about removing friction that slows down your entire business.</p> <h2>Chapter 2 - Why Developer Experience Matters</h2> <p>Poor developer experience isn’t just about productivity--it can hide catastrophic business risks in plain sight.</p> <p>One morning in 2012, a developer at Knight Capital pushed a routine update. Nothing seemed off--until the system started hemorrhaging money. In just 45 minutes, a single change triggered $460 million in losses. The culprit? An old feature flag, long retired, accidentally reactivated by a deployment script. But the real problem wasn’t the bug--it was a development environment that made problems impossible to detect and fix quickly, turning a routine mistake into a company-ending disaster. (Dolfing, Henrico. “Case Study 4: The $440 Million Software Error at Knight Capital.” 5 June 2019, <a href="http://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html">www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html</a>).</p> <p>Fast-forward to July 2025: Jason Lemkin was using Replit’s AI coding assistant to build a database for his company SaaStr.AI. He explicitly set a “code freeze”—no changes to live data. But the AI ignored his instructions and deleted his entire production database. (<a href="https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database">https://www.pcmag.com/news/vibe-coding-fiasco-replite-ai-agent-goes-rogue-deletes-company-database</a>)</p> <ul> <li>AI acceleration makes existing friction even more costly.</li> <li>Poor DevEx creates a competitive disadvantage that compounds over time.</li> <li>Developer experience has a positioning problem.</li> <li>Good developer experience isn’t just about making developers happy--it’s about removing friction to accelerate business outcomes.</li> <li>Developer experience acts as a force multiplier.</li> </ul> <h2>Chapter 3 - Friction: The Value Killer</h2> <p>Friction can slow any progress, especially in software, which is why improving DevEx is so important.</p> <p>Friction is anything that slows down or gets in the way of completing work:</p> <ul> <li><em>Onboarding friction.</em> New developers spend days or weeks struggling with incomplete documentation, waiting for system access approvals, and learning unintuitive toolsets before writing a single line of productive code. Even experienced developers face similar friction during laptop refreshes or system reimaging—losing valuable time during active sprints when their contributions are most needed. Remote developers struggle to find answers in documentation because reaching out for help means scheduling a meeting.</li> <li><em>Codebase friction.</em> Insufficient abstractions force developers to touch many places to make simple changes, brittle monolithic fixtures accumulate over time making tests hard to extend, and patterns that once served the code well become obstacles as the system evolves.</li> <li><em>Integration friction.</em> Misconfigured APIs and incompatible toolchains break developer flow, turning what should be seamless connections into frustrating workarounds and manual steps.</li> <li><em>Process friction.</em> Multi-step approval processes, several iterations of review, and bureaucratic requirements often delay actual development work. The cost of this friction often becomes apparent during hackathons when process requirements are relaxed and innovation happens quickly.</li> <li><em>Review friction.</em> Code reviews that should take hours stretch into days due to inefficient PR assignment, unclear code or module ownership (especially in monorepo environments), multiple review cycles caused by unclear requirements, and large and complicated PRs due to developers working in isolation.</li> <li><em>Development friction.</em> Painfully slow build times, test suites that are unreliable and take ages to complete, incomplete test coverage, and flaky tests that fail intermittently without real issues waste developers’ time and break concentration during active coding sessions.</li> <li><em>Deployment friction.</em> Rather than one-click deploys, developers write custom scripts for every code push because automation is incomplete, or spend hours manually gathering data from disparate systems to make deployment decisions.</li> </ul> <p>Eliminating these friction points creates immediate business value.</p> <p>“Productivity” sets off alarm bells--”Experience” opens conversations.</p> <h1>PART II The Three Essential Elements of DevEx</h1> <h2>Chapter 4 - The DevEx Framework</h2> <p><em>Feedback loops, flow state, and cognitive load, each of which is examined further in its own chapter.</em></p> <h2>Chapter 5 - Feedback Loops</h2> <p><em>See OODA.</em></p> <h2>Chapter 6 - Flow State</h2> <p><em>See Csikszentmihalyi.</em></p> <h2>Chapter 7 - Cognitive Load</h2> <p>"Excessive cognitive load directly impedes developers’ ability to deliver value to customers."</p> <p>"The solution is systematic reduction of unnecessary complexity."</p> <p>"AI presents both opportunities and risks for cognitive load."</p> <h2>Chapter 8 - Use the Framework to Identify and Remove Friction</h2> <p>"The three elements of DevEx provide a framework for spotting and fixing friction."</p> <h1>PART III Making the Business Case</h1> <h2>Chapter 9 - Translate Developer Experience into Business Value</h2> <p>"C-suite executives speak the language of business outcomes."</p> <p>"Talk about recovering time: Convert developer hours into dollar value."</p> <p>"Talk about saving money: Quantify cost savings."</p> <p>"Talk about making money: Accelerating revenue through DevEx."</p> <p><em>"Capital One's DevEx transformation" citation here.</em></p> <p>"Talk about proven correlations: Link technical metrics to business outcomes." ... "A specific example of this approach is the developer experience index (DXI), developed by DX. The DXI score is the average of 14 survey questions that ask developers about different aspects of their development process. By correlating this score against self-reported time loss, DX has found that each one-point gain in DXI score translates to saving 13 minutes per week per developer, equivalent to 10 hours annually. (Noda, Abi, and Laura Tacho. “The One Number You Need to Increase Roi per Engineer.” DX, 2025, getdx.com/research/the-one-number-you-need-to-increase-roi-per-engineer/.) This correlation allows organizations to predict financial impact from DevEx improvements."</p> <h2>Chapter 10 - Connect Your Work to What Others Care About</h2> <p>"To gain leadership support, demonstrate how developer experience directly solves their specific business problems."</p> <p>"Align your DevEx work with the problems that keep your CEO and CTO up at night."</p> <p>"Tie it to your company’s biggest priority."</p> <p>"Give senior leaders something to brag about."</p> <p>"Reveal performance gaps."</p> <p>"Always reframe developer experience from a technical nice-to-have into a strategic business enabler."</p> <h1>PART IV Improving DevEx: A 7-Step Process</h1> <h2>STEP 1 Start Your DevEx Journey</h2> <blockquote> <p>Reminder: <a href="/management/leading-change">Kotter's 8 steps</a>:</p> <ol> <li>Establishing a sense of urgency</li> <li>Forming a powerful guiding coalition</li> <li>Creating a vision</li> <li>Communicating the vision</li> <li>Empowering others to act on the vision</li> <li>Planning for and creating short-term wins</li> <li>Consolidating improvements and producing still more change</li> <li>Mantaining new approaches</li> </ol> </blockquote> <h3>Chapter 11 - Interview Developers</h3> <p>"To get good information, you’ll need to interview a diverse set of developers." 12-15. Consider factors such as years of experience, product area, type of development (e.g., legacy, mobile, cloud, data science). Make two planning lists: 1. Demographics that matter. For example, work location (onsite, remote, hybrid) and experience level (junior or senior devs). 2. Work types and organizational areas where devs work.</p> <p>"Let developers know that talking to you is a good use of their time."</p> <p>"Building strong developer experience starts with earning developers’ trust."</p> <p>"Most importantly, honor developers’ reality."</p> <p>"During the interview, let them know you are there to listen and learn."</p> <p>"You’ll know you’ve done enough interviews when you start hearing the same problems repeatedly—this is called saturation. It usually happens at around 12-15 interviews, but you might need fewer if clear patterns emerge earlier (generally no less than 7), or more if you’re consistently getting new information."</p> <h3>Chapter 12 - Synthesize What You Learn</h3> <p>"Start identifying labels while you’re still in interview mode."</p> <p>"Build a flexible framework that evolves with your understanding."</p> <p>"Capture these themes and supporting evidence in a simple document."</p> <p>"Take a second pass through your notes."</p> <h3>Chapter 13 - Visualize Your Developers’ Workflow, Tools, and Friction</h3> <p>"List the steps your developers have to do to write and ship code."</p> <p>"List <em>everything</em>, even if it seems trivial—you never know where friction might be lurking."</p> <p>"Don’t forget specialized developer workflows."</p> <p>"Consider turning your lists into workflow diagrams."</p> <h3>Chapter 14 - Understand Your Stakeholders</h3> <p>"You might be surprised just how many stakeholders you have."</p> <p>Start by listing all potential stakeholder groups. These may include:</p> <ul> <li>Engineering teams: Developers, tech leads, and engineering managers directly affected by improvements.</li> <li>Auxiliary functions: SREs, data scientists, and customer success engineers may not be product engineers, but your investigations and solutions will likely impact their work.</li> <li>Leadership: CTOs, VPs of engineering, and other decision-makers who approve resources.</li> <li>Adjacent functions: Product managers, designers, and QA professionals who interface with development.</li> <li>Support functions: HR professionals interested in retention metrics, Finance team members monitoring costs, Security teams concerned with compliance aspects</li> <li>External groups: Customers or partner teams who may benefit from improved developer productivity</li> </ul> <h3>Chapter 15 - Share What You’ve Learned</h3> <p>"Once you’ve synthesized what you learned, be sure to share your findings."</p> <p>"Once your summaries are shared, be prepared for continued dialogue."</p> <h2>STEP 2 Start Small and Get a Quick Win</h2> <h3>Chapter 16 - Pick the Right Projects</h3> <p>"When kicking off a DevEx improvement initiative, you need visible wins that validate your approach and transform skeptics into advocates."</p> <p>"The RICE framework--Reach, Impact, Confidence, Effort--gives you a simple way to evaluate opportunities without getting bogged down in analysis paralysis. RICE can help you tease out projects that have a higher likelihood of success, targeting those with high reach, high impact, and high confidence, while keeping effort low. For early project selection, think of this as “Quick RICE” or “gut-check RICE”—you’re making rapid assessments to identify promising candidates, not conducting detailed analysis. That deeper dive comes later when you have more data.</p> <ul> <li>Reach: How many developers feel this pain? <ul> <li> <p>Focus on broad reach. Choose improvements that can benefit many developers rather than solving deep problems for a few.</p> </li> <li> <p>Consider frequency. Areas of friction that impact developers frequently (e.g., daily or multiple times a day) are great candidates for early projects.</p> </li> </ul> </li> <li>Impact: Will developers immediately notice the improvement? <ul> <li> <p>Target visible impact. Pick improvements that developers immediately feel, reducing the need to prove value later. Example: Implementing automated dependency checks and standardized PR templates to reduce code review cycles from three days to one day.</p> </li> <li> <p>Focus on problems developers recognize. Target issues that developers already know about and feel personally. While it’s tempting to tackle underlying root causes, developers and their managers want to see and feel progress. These wins build credibility and naturally gain leadership attention.</p> </li> </ul> </li> <li> <p>Confidence: Can you deliver this successfully?</p> <ul> <li> <p>Look for clear milestones where you can show progress. Select initiatives where you can easily identify visible checkpoints that stakeholders can track. Milestones let you demonstrate incremental success, maintain momentum, and identify when something isn’t working before you’ve invested too much time.</p> </li> <li> <p>Choose measurable projects. Select projects where progress and value can be measured through existing or easy-to-capture metrics. This approach eliminates the double burden of both delivering value and proving it exists, allowing your team to focus on solving problems rather than measuring them.</p> </li> </ul> </li> <li> <p>Effort: How quickly can you show results?</p> <ul> <li> <p>Find low-hanging fruit. Look for annoying but fixable issues (some people call these “paper cuts”). Common examples include automating repetitive tasks, fixing flaky tests, improving a difficult UI, or streamlining access to development environments. Depending on your context, some of these may require more work, so evaluate and look for things that can be addressed within weeks (or maybe a few months).</p> </li> <li> <p>Target medium complexity. Choose projects complex enough to deliver meaningful value but not so complex that they risk getting stuck. These “goldilocks” initiatives— such as fixing flaky test suites, improving monitoring dashboards, or finishing data migrations—demonstrate competence while maintaining momentum. Avoid both trivial changes that feel like busywork and ambitious refactors that could take quarters to complete.</p> </li> <li> <p>Consider timelines. Early projects should show results in three months, while later efforts should have visible milestones every six to nine months. For example, avoid choosing an initial project that requires a significant re-architecture, as these typically take a significant amount of time and offer little visible progress until they are complete.</p> </li> </ul> </li> </ul> <p>When evaluating your candidate projects from interview synthesis:</p> <ol> <li>Score each project on a simple scale (High/Medium/Low) for each RICE component.</li> <li>Look for patterns--projects scoring High on reach and impact with Medium or Low effort are often winners.</li> <li>Trust your instincts--if something feels off despite good scores, dig deeper.</li> <li>Consider combinations--sometimes two smaller projects together create more impact than one large one.</li> </ol> <p>"Turn evaluation into clarity."</p> <p>"Your DevEx breakthrough might be a process, not a tool."</p> <h3>Chapter 17 - Share Early Wins to Gain Momentum</h3> <p>"Fail fast and share your learnings."</p> <p>"Celebrate wins along the way."</p> <p>"Bring others along on the journey."</p> <p>"Share your findings thoughtfully."</p> <p>"Keep the conversation going."</p> <h3>Chapter 18 - Avoid Common Pitfalls</h3> <p>Van Buren and Safferstone studied 5,400 leaders implementing new initiatives in companies. Their work reveals common pitfalls that can undermine long-term success. (Van Buren, Mark E., and Todd Safferstone. “The quick wins paradox.” Harvard Business Review 87.1 (2009): 54-61.) Here’s how successful leaders navigate these challenges:</p> <ul> <li> <p>Handling feedback.</p> <ul> <li> <p>Don’t get defensive when questioned. It’s natural to feel protective of work you’ve invested in, but defensiveness shuts down dialogue. Instead, explain your rationale and reasoning—this builds understanding rather than resistance. DevEx teams often assume their experience represents the broader engineering org—a common blind spot.</p> </li> <li> <p>Do treat criticism as valuable input. Engineers are your customers, and their feedback shapes better solutions. Stay curious, seek early feedback, and remember that listening builds trust.</p> </li> </ul> </li> <li> <p>Team dynamics.</p> <ul> <li> <p>Don’t be overconfident when prescribing solutions. Teams may have tried similar approaches and encountered hidden challenges. Focus on identifying and prioritizing problems.</p> </li> <li> <p>Do balance knowledge with genuine curiosity. Let empowered teams explore solutions to the problems you identify, and continue to gather insights from listening tours.</p> </li> </ul> </li> <li> <p>Decision making.</p> <ul> <li> <p>Don’t build solutions in isolation. Even with the best intentions to show quick progress, DevEx teams often fall into the trap of working alone to avoid dealing with different opinions. This is especially true in DevEx initiatives because your customers often have strong opinions about the right sonlution.</p> </li> <li> <p>Do embrace collaboration. Choose initiatives that engage the broader engineering organization. While individual quick fixes may seem more efficient, projects with shared ownership create better solutions and stronger buy-in.</p> </li> </ul> </li> <li> <p>Execution style.</p> <ul> <li> <p>Don’t micromanage engineers—trust empowered teams across the organization to explore solutions. The best initiatives—and products—are those where leaders identify and prioritize problems, and empowered engineering teams explore solutions.</p> </li> <li> <p>Do invest in sharing context. When engineers understand the broader picture, they design better solutions and make informed decisions. This also builds trust in team judgment. Create clear communication channels for updates and decisions. The research shows that the most successful leaders don’t avoid quick wins—they embrace them. The key difference is their focus on building team capability and shared success rather than individual achievements.</p> </li> </ul> </li> </ul> <h2>STEP 3 Use Data to Optimize Your Own DevEx Work</h2> <h3>Chapter 19 - Establish Your Data Foundation</h3> <p>There are a few kinds of data to think about, and they overlap a bit.</p> <ul> <li>Qualitative data is descriptive, and includes things like interview transcripts and open-text responses in surveys.</li> <li>Quantitative data is numerical, specifically in ways that can be calculated, for example a response on a 1-5 survey scale, or statistics from version control system use.</li> </ul> <p>We will also talk about self-report and system data, each of which can be either qualitative or quantitative.</p> <ul> <li>Self-report data comes from people’s perceptions and experiences. It can be qualitative (like interviews or open-text responses) or quantitative (like 1-5 scale survey responses).</li> <li>System data can be observed and measured externally. This is also called objective data. It’s typically quantitative (such as number of commits or turnaround time on a PR) but can also be qualitative (such as error messages).</li> </ul> <table> <thead> <tr> <th>. </th> <th> Qualitative Data </th> <th> Quantitative Data</th> </tr> </thead> <tbody> <tr> <td>Self-report Data </td> <td> Interview transcripts; Open-ended survey responses; Feedback comments </td> <td> Survey ratings; Satisfaction scores; Self-reported productivity metrics</td> </tr> <tr> <td>System Data </td> <td> Error messages; Log entries; Documentation content; User flow </td> <td> Build times; Deployment frequencies; Error rates; System performance metrics</td> </tr> </tbody> </table> <p>"Surveys and system data are strategic partners, each with unique strengths."</p> <p>"The most successful DevEx initiatives use both self-report and system data."</p> <h3>Chapter 20 - Identify and Use the Data You Already Have</h3> <p>"Most organizations have untapped metrics that can provide valuable insights into developer experience."</p> <p>For example:</p> <ul> <li>Development activity. Version control metrics, pull request patterns, code review times, AI assistant adoption and code generation.</li> <li>Build and test performance. CI/CD pipeline metrics, test coverage, build times, failure rates—both for traditional and AI-assisted development.</li> <li>Operations. Deployment frequency, infrastructure costs, monitoring and logging information.</li> <li>Process. Ticket resolution times, incident response data, feature delivery cycles.</li> <li>Feedback. Existing developer surveys, retrospective notes, onboarding feedback.</li> </ul> <p>Find early insights from existing data. Once you find and document your existing data sources, here are some ways to extract valuable insights:</p> <ul> <li> <p>Development history analysis.</p> <ul> <li> <p>Analyze commit patterns: Look for frequency trends, time-of-day patterns, and areas with high churn.</p> </li> <li> <p>Examine pull request metrics: Review time-to-merge, review cycles, and comment patterns.</p> </li> <li> <p>Track feature lifecycles: Identify abandoned projects, reverted changes, and their contexts.</p> </li> <li> <p>Compare AI vs. non-AI development patterns: Review metrics between developers using AI tools and those using traditional approaches. Look for learning curves, productivity changes, and new friction points.</p> </li> </ul> </li> <li> <p>Incident and failure analysis.</p> <ul> <li> <p>Connect code changes to incidents: Map specific commits or deployment patterns to system failures. Look for correlations with AI-assisted coding and AI-enabled toolchains.</p> </li> <li> <p>Study incident reports: Identify recurring issues, response times, and root causes.</p> </li> <li> <p>Review postmortem documentation: Extract lessons learned and improvement recommendations, and note any recurring patterns.</p> </li> </ul> </li> <li> <p>Project documentation mining.</p> <ul> <li> <p>Analyze archived project specs: Understand reasons for project cancellations or pivots.</p> </li> <li> <p>Review technical debt notes: Identify areas where shortcuts were taken and why.</p> </li> <li> <p>Examine architecture decision documents: Learn from past decisions and their outcomes.</p> </li> </ul> </li> <li> <p>Tool and platform usage.</p> <ul> <li> <p>Monitor CI/CD pipeline performance: Track build times, failure rates, and bottlenecks across both traditional and AI-assisted workflows.</p> </li> <li> <p>Review platform support ticket patterns: Identify recurring developer pain points.</p> </li> <li> <p>Analyze tool adoption data: Understand which tools developers actually use vs. available tools.</p> </li> </ul> </li> </ul> <h3>Chapter 21 - Use Surveys for Fast Insights</h3> <p>"You can get actionable insights now through surveys."</p> <p>"A powerful DevEx survey doesn’t need to be long—and it shouldn’t be."</p> <p>"Measure value with the Sean Ellis Test. While active user metrics and satisfaction scores provide valuable signals, occasionally you need a more direct assessment of how essential your solution—or an existing tool—has become to users. The Sean Ellis Test offers a simple and powerful method that startup companies use to determine product-market fit, and it works equally well for internal developer tools. The test centers on a single powerful question: <em>“How would you feel if you could no longer use [your platform/tool]?”</em> With just three possible responses: Very disappointed; Somewhat disappointed ;Not disappointed (it isn’t that useful). Through extensive work with startups, Ellis found that products with strong market fit consistently have at least 40 percent of users answering they would be “very disappointed” if they could no longer use the tool. This threshold has become a reliable benchmark for product success." ... "However, be cautious when interpreting results for mandated tools where users have no alternatives. In these cases, consider adapting the question. For example: <em>“<strong>If you had a choice of developer tools</strong>, how would you feel if you could no longer use [current tool]?”</em> This framing helps distinguish between genuine tool value and organizational dependency. You can also pair the Ellis Test with satisfaction questions to better understand whether high disappointment scores reflect tool love or job necessity."</p> <h3>Chapter 22 - Invest in System Metrics Over Time</h3> <p>When starting out, you might rely heavily on surveys (around 80 percent) with some system data (around 20 percent), giving you quick, broad insights into developer pain points. As you progress in your initiative and you instrument more systems, this balance shifts along an S-curve pattern—system data gradually increases, then ramps up significantly as you build more automated collection capabilities. In mature DevEx programs, system data often comprises the majority of your information simply due to the volume you can collect automatically. Surveys remain crucial throughout your journey and never disappear entirely. Even in mature programs, you’ll continue using survey data for instrumentation decisions, uncovering hidden friction points, and measuring developer satisfaction. The key is that while system data provides precision and continuous monitoring, survey data provides the context and human perspective that systems can’t capture.</p> <p><strong>Beware survey fatigue.</strong><br> We all know this but it’s worth repeating: Bombard your developers with surveys and they’ll stop giving you the insights you actually need. You need to be realistic about what data you can get, what you can’t get, and how often you can get it.<br> <strong>Developers only have so much attention, and work is their priority.</strong> Your developers are solving complex problems all day. When you ask them to stop that work to fill out a survey, you’re making a withdrawal from a limited attention bank. Make too many withdrawals, and the account closes. Here are some tips for survey success:</p> <ul> <li>The 15-minute maximum. If your survey takes longer than 15 minutes to complete, you’ve already lost. Developers will either abandon it halfway through or rush answers just to finish. <ul> <li> <p>The timing sweet spot. Forget monthly surveys. We’ve seen dramatically better results with a three- or six-month cadence. Why? Because that’s roughly how long it takes to implement meaningful changes based on previous feedback, and it’s minimal interruption to developer’s work. Show progress before asking for more input.</p> </li> <li> <p>The importance of consolidation. If your organization already has a developer survey but it’s not giving you what you need, don’t launch a competing survey. That’s positioning yourself against an established player (bad move) and introducing additional work for devs (another bad move). Instead, partner with the existing survey team and position your questions as complementary.</p> </li> <li> <p>The differentiation requirement. Multiple surveys can coexist, but they’re most successful when they have clear, distinct positioning in developers’ minds. A culture survey and a technical workflow survey can both succeed if they’re properly positioned and spaced several weeks or months apart.</p> </li> </ul> </li> </ul> <p><strong>Do a final check.</strong> Before you hit “send” on that survey, ask yourself: “If I were in my developers’ shoes, would I stop what I’m doing to answer these questions?” and “Would I care about the results of this survey? Would I want to share the results with someone else?” If the answers aren’t obvious yeses, you need to reposition your approach. <strong><em>The teams that win at developer feedback aren’t the ones with the most surveys—they’re the ones who respect the context their developers operate in and position their feedback requests accordingly.</em></strong></p> <p>Survey data can be very useful when you don’t have good system data, but don’t fall into the trap of waiting for perfect system data before taking action. Perfect is the enemy of progress. The transition to more system data happens organically as you identify data sources with reasonable quality and coverage across your organization. For instance, if pull requests are a point of friction, you’ll want to analyze metrics like PR cycle time (how long it takes to complete a PR) and PR dwell time (a component of overall cycle time, focusing on how long a PR waits before it is acted on). These objective measures provide precise data about specific friction points.</p> <p>When considering your data, beware of common fallacies that can mislead your DevEx efforts. The <em>streetlight effect</em> occurs when you focus exclusively on areas where you already have strong system data. Don’t limit your attention to what’s easy to measure. Even in mature DevEx improvement programs, surveys continue to provide crucial insights about emerging issues, subjective experiences, and areas that are difficult to instrument. This is why maintaining some survey component remains valuable even as system data grows to dominate your overall metrics portfolio. Equally important is avoiding the <em>snapshot fallacy</em>. Don’t treat a single survey or data collection as permanent truth. DevEx needs evolve as your organization changes, technologies advance, and developers gain experience. Plan for regular pulse surveys and ongoing system data analysis to maintain an accurate, current picture of developer experience.</p> <h3>Chapter 23 - Make Sure You Capture the Right Data</h3> <p>"The most effective data strategies work backward from problems to metrics. Start with the pain points your surveys reveal, then identify what specific data would help you understand and track improvements in those areas."</p> <p>"We suggest you also consider the dimensions of developer experience your metrics represent. Frameworks like SPACE (satisfaction, performance, activity, communication, and efficiency and flow) can help ensure you’re capturing a complete picture rather than just what’s easy to measure."</p> <p>"Don’t forget to collect data to build a strong business case. Many DevEx improvement initiatives stumble because they focus too narrowly on technical metrics like build times and test effectiveness. While these measurements are essential, they often don’t resonate with all stakeholders or clearly demonstrate business value."</p> <p>"As you get more data, you’ll need to be more careful about analyzing and drawing conclusions from it."</p> <p>"Get representative data to understand what’s happening. To build an accurate picture, collect data from across your organization. Your data collection strategy—whether for interviews, surveys, or system logs—directly impacts your conclusions."</p> <p>"Get enough data to understand what’s happening. Your goal is to get enough data to feel confident about insights and next steps, without getting bogged down in academic-level statistical significance. Here are some good rules of thumb for when you’re just getting started.</p> <ul> <li> <p>For surveys, shoot for at least 30-50 responses. More is better, but you’ll start getting good insights at this level. Getting 100+ responses often gives you pretty stable findings for practical purposes. Depending on your organization—for example, how it’s structured, how technology platforms are used, org-wide cultural differences—we suggest getting enough responses (30-50) per business unit or product team.27</p> </li> <li> <p>For interviews, start with 5-8 people. Keep going until you’re not hearing many new things—usually around 12-15 interviews for focused topics. (You may hear this called “reaching saturation.”) If every interview brings totally new information, you’re still learning and probably need more.</p> </li> <li> <p>For system data, aim for comprehensive coverage—ideally data from at least 80 percent of your target systems or population. Make sure your data definitions are consistent and well-documented (e.g., what counts as a “build failure” or “deployment”). Watch out for data quality issues such as missing timestamps, incomplete logs, or inconsistent formatting. If you’re measuring over time, collect at least a few weeks of data to account for typical variations in usage patterns. Be especially careful about seasonality--collecting data during a holiday period or major release cycle might not represent typical patterns."</p> </li> </ul> <p>"Get enough data from enough people to understand what’s happening. For surveys, a response rate above 30 percent is pretty good for most organizational contexts. However, 20-30 percent is fairly typical and usually gives you workable data. If you’re below 15 percent, that’s when you should start wondering if there might be issues with your distribution approach or if the survey itself might be too long or overly complicated."</p> <p>"When analyzing developer feedback, focus on representation over raw numbers. A 30 percent response rate from developers across different teams, roles, and experience levels will tell you more about patterns across the organization than a 70 percent response rate that’s mostly from one group. This principle applies to both survey data and behavioral metrics."</p> <p>"Building high response rates. You can achieve surprisingly high participation rates over time by building trust and demonstrating impact. For example, Amazon’s annual developer survey achieved over 90 percent response rates despite taking an hour to complete—a remarkable achievement that came from consistently showing developers how their feedback drove real changes and transparently reporting results back to them. Similarly, hundreds of companies using the DX platform see response rates over 90 percent, thanks to thoughtful survey design, transparent dashboards, and follow-up action that the companies take on the data. The key insight: <strong><em>Response rates aren’t just about the survey itself, but about the broader culture of feedback and action.</em></strong> When developers see their input driving real changes, they’re willing to invest more time in future surveys."</p> <h3>Chapter 24 - Turn Data Into Actionable Insights</h3> <p>When you’re diving into DevEx data, start with the basics.</p> <p>Performance bottlenecks emerge when your statistical measures don’t align. The gap between median and average in the build time example points to infrastructure inconsistencies or problematic edge cases that are dragging down the development process.</p> <p>Usage patterns show up in frequency counts and trends over time. A trend line of deployment frequencies could reveal that teams deploy less often on Mondays—sparking questions about team workflows, meeting schedules, or even weekend incident recovery patterns.</p> <p>Friction points can become visible through distributions and frequency counts. When you analyze where developers spend the most time or encounter the most errors, you can spot the bottlenecks.</p> <p>Visualizations are your friend--they transform abstract numbers into patterns that people can quickly grasp. A histogram of build times might immediately show you have a “long tail” of problematic builds, while a trend line of deployment frequencies could reveal that teams deploy less often on Mondays or the end of the fiscal year. These visual patterns often spark the right questions: “Why do we see this dip here?” or “What changed when this metric improved?” Even better, visualizations make it easier to share insights with stakeholders who might get lost in raw numbers.</p> <p>You can do much of the analysis yourself, but sometimes it’s best to partner with a data scientist.</p> <p>Privacy and security are important to consider when sharing results and visualizations. ... A good rule is the Rule of 5: Never show data for groups smaller than five developers. This prevents someone from figuring out, say, which specific developer is struggling with a particular API or deployment system. Also, be extra careful with free-text comments in surveys—they can accidentally contain sensitive information.</p> <h2>STEP 4 Decide Strategy and Priority</h2> <p>Every item on your ever-growing list is important to someone, but may not resonate with someone else.</p> <h3>Chapter 25 - Focus Your DevEx Efforts</h3> <p>Use RICE criteria to clarify decisions and create buy-in—now with real data.</p> <p>An important part of prioritizing is saying no. ... RICE helps you say no systematically by forcing you to evaluate the full picture—not just the loudest complaints or the most interesting technical challenges.</p> <h3>Chapter 26 - Use Relevant Criteria to Set Priority for Developer Experience</h3> <p>RICE can help you tease out projects that have a higher likelihood of success, targeting those with high reach, high impact, and high confidence, while keeping effort low.</p> <p>RICE = (Reach x Impact x Confidence) / Effort</p> <p>Reach measures how many developers are affected and how often. This is your foundation for understanding true impact scale. Possible questions to ask:</p> <ul> <li>How many developers does this affect?</li> <li>What’s the frequency of impact? (Daily friction usually trumps quarterly pain.)</li> <li>Does this affect high-leverage developers disproportionately?</li> </ul> <p>Use consistent units across all calculations. For example, don’t mix “daily impacts” for some items with “number of developers affected” for others. This ensures a fair comparison across different types of improvements.</p> <p>Impact assesses how significantly this change will improve developers’ daily experience. Questions to ask:</p> <ul> <li>Will developers immediately feel this improvement?</li> <li>How significantly will this change their daily experience?</li> <li>Does this remove a major constraint or just reduce minor friction? When scoring impact, use a simple 1-3 scale: <ul> <li>1 = Low impact: Minor improvement to workflow.</li> <li>2 = Moderate impact: Noticeable productivity gain.</li> <li>3 = High impact: Major bottleneck elimination or significant time savings.</li> </ul> </li> </ul> <p>Confidence reflects how certain you are that the initiative will succeed as planned, and is typically estimated as a percentage (e.g., 80% confident). Consider:</p> <ul> <li>Technical feasibility and known solutions.</li> <li>Team expertise and capacity.</li> <li>Dependencies and potential blockers.</li> <li>Past experience with similar initiatives.</li> </ul> <p>Effort measures the total investment needed, typically in person-months. Be sure to include:</p> <ul> <li>Development time.</li> <li>Testing and deployment.</li> <li>Documentation and training.</li> <li>Ongoing maintenance considerations.</li> </ul> <p>Look for business impact and strategic alignment. Start by asking, “Is it easy for me to talk about this challenge in terms of metrics that business leaders care about?” Equally important is considering, “How does this initiative align with or leverage existing work to maximize business value?”</p> <p>Think about collective vs. local action.</p> <p>Consider timelines, urgency, and organizational readiness. Key questions here include:</p> <ul> <li>“What challenge is best to tackle first because it is the fastest to complete?”</li> <li>“Do we have the right skills internally, or will this require external expertise?”</li> <li>“Can we address this challenge with our current resources and capabilities?”</li> <li>“Which challenges will get worse if we wait, and what problems will that create?”</li> <li>“How does this initiative align with our organization’s natural business rhythm?”</li> </ul> <p>Use constraints and risks. Great questions to ask here include:</p> <ul> <li>“What challenge should we tackle first because it represents a critical bottleneck in our development flow?”</li> <li>“Do we have control over the key dependencies needed for success?”</li> <li>“Can we implement this change with minimal disruption to existing workflows?”</li> </ul> <p>Don’t forget team and organizational factors. Ask yourself, “Which challenge allows us to strengthen our people and organization while solving technical problems?” Look for projects where team members can make meaningful contributions that showcase their expertise while creating opportunities for growth and recognition. The most successful initiatives align with teams’ professional development interests and incentives.</p> <p>Don’t forget other considerations. The examples above aren’t comprehensive–you’ll want to include criteria specific to your organization.</p> <h3>Chapter 27 - See the Bigger Picture: Use a Quick Rubric</h3> <p>Avoid getting lost in the weeds by looking at the big picture.</p> <p>Create your rubric. Start with the criteria that matter most for your organization. Common DevEx prioritization criteria include:</p> <ul> <li>RICE analysis</li> <li>Can be mapped to business value.</li> <li>Requires collective action.</li> <li>Aligns with timelines and rhythm of business.</li> <li>Is a constraint.</li> <li>Aligns with team goals.</li> <li>[Your context-specific questions here.]</li> </ul> <p>Make the most of your analysis. The combination of RICE scoring and strategic rubrics gives you perspective:</p> <ul> <li>Use RICE scores for quantitative justification, ROI calculations, and detailed resource planning.</li> <li>Use the rubric for strategic communication, stakeholder alignment, and big-picture decision making.</li> <li>When they conflict, consider whether your RICE inputs need adjustment or whether strategic factors (such as timeline or team capacity) should override pure numerical optimization.</li> </ul> <h2>STEP 5 Sell Your Strategy</h2> <p>Many DevEx teams underestimate the importance of strategic communications.</p> <h3>Chapter 28 - Leverage Your Stakeholder Analysis for Effective Communication</h3> <p>The relationships you’ve built and insights you’ve gathered during your listening tour will help you craft messages that resonate with each group.</p> <p>Remember your stakeholders are more than just developers.</p> <p>Tailor your approach based on stakeholder characteristics.</p> <p>Refine your stakeholder list with communication-specific details. "We often find it helpful to list key stakeholders and information in a table or a spreadsheet. This is handy for organization as well as identifying any information you may be missing for certain stakeholder groups. Your stakeholders—and the information you decide to collect—will differ. Here, you’ll identify major stakeholders, their motivation (why they care), what support level they need (skeptical, neutral, enthusiastic), your engagement strategy (inform, convince, collaborate), their information needs, their engagement strategy, and their preferred communication channels."</p> <p>Your stakeholders need to know what the problem is and why they should care. Don’t assume everyone sees the DevEx problems faced by your organization as clearly as you do. People come to the table with diverse backgrounds, shaping their views on what’s good, bad, or even possible in your systems. Some stakeholders may also have preconceived solutions—for instance, leadership might assume that adopting AI tools will automatically solve developer productivity challenges, without understanding the limitations of AI or the specific friction points that need addressing first.</p> <p>Use the EMI framework for stakeholder communications. When crafting DevEx communications, apply the EMI framework to ensure messages resonate with each stakeholder group:</p> <ul> <li>E = Educate: Address knowledge gaps specific to their role and perspective.</li> <li>M = Motivate: Connect DevEx improvements to their priorities and pain points.</li> <li>I = Inform: Provide actionable details relevant to their involvement.</li> </ul> <p>Think of your DevEx initiative like a product—and “sell” it to your stakeholders. When communicating about your DevEx improvement initiative, adopt a product mindset to make your message more compelling. This approach helps you:</p> <ul> <li> <p>Focus on solving specific problems rather than just implementing technology.</p> </li> <li> <p>Clearly articulate value in terms stakeholders understand.</p> </li> <li> <p>Create a coherent narrative about your initiative’s purpose and benefits.</p> </li> </ul> <p>For “investors” (leaders who fund your work), you’ll need to be able to briefly outline the business case, competitive landscape, and expected returns. For “customers” (developers and engineering managers), you’ll need to emphasize how your solution addresses their specific pain points. This product-based framing transforms abstract concepts like “improving developer experience” into concrete solutions that stakeholders can easily understand and support. For a more detailed approach to treating your DevEx tooling and automation ecosystem as a product, see the Third Practice: Make Technology Sustainable and Effective.</p> <h3>Chapter 29 - Effective Messaging Takes Customization and Repetition</h3> <p><em>Psych 101.</em></p> <p>Tell people what’s going on—and say it more than once.</p> <p>Give your DevEx initiative an identity.</p> <p>You’ll want more materials to help others confidently spread the word without always needing you.</p> <p>Build capability, not just awareness. "Effective DevEx communication goes beyond informing people—it builds their capability to understand, engage with, and champion your initiatives. Design your materials to move stakeholders through a three-stage progression:</p> <ul> <li>Understand. They grasp the big picture, value, why DevEx matters, and core terminology.</li> <li>Try. They use what they’ve learned to engage with the ideas and actively look for ways to support and enable your work.</li> <li>Activate. They mentor others, build capabilities in their teams, and provide recommendations for sustaining and improving DevEx initiatives.</li> </ul> <h2>STEP 6 Drive Change at (Your) Scale</h2> <h3>Chapter 30 - Adapt Your Approach to Your Scope</h3> <h3>Chapter 31 - Drive Change With Global Scope of Control</h3> <h3>Chapter 32 - Drive Change With Local Scope of Control</h3> <h3>Chapter 33 - Drive Change With Middle Ground Scope of Control</h3> <h2>STEP 7 Evaluate Your Progress and Show Value</h2> <h3>Chapter 34 - Analyze Your Data</h3> <h3>Chapter 35 - Prepare to Communicate Your Results</h3> <h3>Chapter 36 - Share Progress</h3> <h3>Chapter 37 - Navigate Conflicting Metrics</h3> <h3>Chapter 38 - Learn and Improve</h3> <h3>Chapter 39 - Evaluation Is Just One Step in a Virtuous Cycle</h3> <h1>PART V Evolving and Sustaining DevEx</h1> <h2>THE FIRST PRACTICE Resource Your DevEx Initiative</h2> <h3>Chapter 40 - Bootstrap Your Early Efforts</h3> <h3>Chapter 41 - Build a Dedicated Function</h3> <h3>Chapter 42 - Developing Talent for DevEx Roles</h3> <h3>Chapter 43 - Make Developer Experience Everyone’s Job</h3> <h3>Chapter 44 - Secure Budget at Different Organizational Stages</h3> <h3>Chapter 45 - Making DevEx Stick</h3> <h2>THE SECOND PRACTICE Create Structures to Support Change</h2> <h3>Chapter 46 - A Framework For DevEx Change</h3> <h3>Chapter 47 - Support Teams Through Change</h3> <h3>Chapter 48 - Guide Organizations Through Change</h3> <h3>Chapter 49 - Support Yourself Through Challenging Work</h3> <h2>THE THIRD PRACTICE Make Technology Sustainable and Effective</h2> <h3>Chapter 50 - Adopt a Product Mindset</h3> <h3>Chapter 51 - Use Product Management Principles to Build Better Solutions</h3> <h3>Chapter 52 - Use Tooling Standardization to Drive Efficiency</h3> <h3>Chapter 53 - Deal with Technical Debt in DevEx Initiatives</h3> <h3>Chapter 54 - Bring It All Together: The Sustainable Technology Mindset</h3> <h2>THE FOURTH PRACTICE Design Metrics That Evolve and Amplify Impact</h2> <h3>Chapter 55 - Design Metrics for Your Stakeholders</h3> <h3>Chapter 56 - Use Metrics to Solve Your Stakeholders’ Problems</h3> <h3>Chapter 57 - Fast Is Relative to Your Competition</h3> <h3>Chapter 58 - Start Simple and Scale Smartly</h3> <h3>Chapter 59 - How AI Changes (and Doesn’t Change) the Metrics You Need</h3> <h3>Chapter 60 - Maintain Your Metrics Like Any Product</h3> <h1>Final Thoughts</h1> <h3>Chapter 61 - What Are You Waiting For?</h3> https://tedneward.github.io/Research/reading/development/high-scalability/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/high-scalability/index.html <p><a href="http://highscalability.com">Blog</a></p> <h3><a href="http://highscalability.com/blog/2015/11/9/a-360-degree-view-of-the-entire-netflix-stack.html">A 360 Degree View Of The Entire Netflix Stack</a></h3> <ul> <li>Netflix infrastructure and transcoding is on EC2 with master copies of digital films from movie studios being stored on Amazon S3.</li> <li>Open-source software on the backend includes Java, MySQL, Gluster, Apache Tomcat, Hive, Chukwa, Cassandra, and Hadoop.</li> <li>The Netflix Open Connect CDN is provided for larger ISPs that have over 100,000 subscribers. It is a low-power, high-storage density appliance caches Netflix content within the ISPs’ data centers to reduce internet transit costs.</li> </ul> <h3><a href="http://highscalability.com/blog/2015/11/2/how-shopify-scales-to-handle-flash-sales-from-kanye-west-and.html">How Shopify Scales To Handle Flash Sales From Kanye West And The Superbowl</a></h3> <ul> <li>Handles about 300 million unique visitors a month, but biggest challenge is from "flash sales".</li> <li>Requests received by Nginx and forwarded to a cluster of servers running Docker with the Rails app.</li> <li>Data tier includes Memcached, Redis, ElasticSearch, MySQL, Kafka, and Zookeeper.</li> <li>Performed load testing to populate a "Resiliency Matrix," which specifies happens to systems when a service (e.g. MySQL, Redis, etc.) goes down.</li> <li>Scaling databases is hard, especially sharding. Try to cache a lot before you have to do it. But shards to help isolate incidents.</li> <li>Most shortcomings from resilience testing have been fixed with database scaling and automatic failovers.</li> </ul> <h3><a href="http://highscalability.com/blog/2015/7/22/architecting-backend-for-a-social-product.html">Architecting Backend For A Social Product</a></h3> <h4>Data Storage</h4> <ul> <li>Identify which data is the most frequently queried, and which data is not so frequently required (e.g. can be archived for analysis).</li> <li>Data queried at a high frequency should be in a place where it is always available, faster to read and write, and horizontally scalable.</li> <li>Can represent connected or relational data using a graph database like Neo4j, FlockDB, AllegroGraph and InfiniteGraph.</li> <li>To store data with availability and scalability, consider S3, Google Cloud Storage, or Rackspace Cloud Files. But metadata indexing is a separate concern.</li> </ul> <h4>Session Data</h4> <ul> <li>Can store this in a key-value store like Redis or a document-based storage solution like MongoDB.</li> </ul> <h4>Indexing</h4> <ul> <li>To build an index, can write a simple system at point of content ingestion what creates an inverted index. Can graduate to Apache Storm.</li> <li>The indexing system itself could be Solr or ElasticSearch, both of which use Lucene.</li> </ul> <h4>Queuing &amp; Push Notifications</h4> <ul> <li>Events with high fanout can be processed using a queue such as ActiveMQ.</li> </ul> <h4>Caching Strategy</h4> <ul> <li>The server-side application cache should store data in a format that is ready to render and requires no messaging or further action. This may require denormalization when content is ingested.</li> <li>Redis can be treated as an in-memory application cache with sharding, leader-follower node configuration, and is fast for incremental operations to collections.</li> <li>We can cache at the reverse proxy level, such as on Nginx. This requires setting HTTP response headers correctly.</li> <li>We should cache resources aggressively in the app or in the browser, such as by setting the proper HTTP caching headers, or persisting data in SQLite.</li> </ul> <h4>Transport Protocols</h4> <ul> <li>Consider using HTTP/2 or SPDY over HTTP/1.1.</li> </ul> <h4>Components</h4> <ul> <li>The load balancer determines which proxy server the request should be forwarded to based on a decided strategy.</li> <li>The proxy server handles all incoming requests. It is an SSL termination point, and will cache HTTP responses based on the defined policy.</li> <li>The event processor is responsilble for any events with a fan-out, such as notifications.</li> </ul> <h3><a href="http://highscalability.com/blog/2015/3/9/applovin-marketing-to-mobile-consumers-worldwide-by-processi.html">AppLovin: Marketing To Mobile Consumers Worldwide By Processing 30 Billion Requests A Day</a></h3> <h4>Technology Stack</h4> <h5>Data Storage</h5> <ul> <li>MySQL for ad data.</li> <li>Spark for offline processing and deep data analysis.</li> <li>Redis for basic caching.</li> <li>Thrift for all data storage and transfers.</li> </ul> <h5>Core App And Services</h5> <ul> <li>Jenkins for continuous integration and deployment.</li> <li>Zookeeper for distributed locking.</li> <li>HAProxy for high availability.</li> <li>Checkstyle and PMD for static code analysis.</li> <li>Syslog for DC-centralized log server.</li> <li>Hibernate for transaction-based services.</li> </ul> <h5>Servers And Provisioning</h5> <ul> <li>Chef for configuring servers.</li> <li>Load balancing is a combination of software like HAProxy and hardware.</li> </ul> <h4>Monitoring Stack</h4> <h5>Server Monitoring</h5> <ul> <li>Graphite as data format.</li> <li>PagerDuty for issue escalation.</li> </ul> <h4>Architecture Overview</h4> <h5>General Considerations</h5> <ul> <li>Store everything in RAM. If it does not fit, save it to SSD.</li> <li>Automate everything.</li> <li>Zero-copy message passing.</li> </ul> <h5>Message Processing</h5> <ul> <li>Sending a message means writing to disk.</li> <li>Message dispatching system provides isolation and extensibility of the system.</li> </ul> <h5>Ad Serving</h5> <ul> <li>Nginx is really fast.</li> <li>jemalloc gave a 30% speed improvement.</li> <li>Torrents propagate serving data across all servers, yielding 83% network load drop on originating server.</li> <li>mmap is used to share ad serving data across nginx processes.</li> <li>XXHash is the fastest hash function with a low collision rate. 75x faster than SHA-1.</li> </ul> <h4>Data Warehouse</h4> <ul> <li>Aggregation is key to allow fast reports on large amounts of data.</li> <li>7 days of raw logs is usually enough for debug.</li> <li>Backup in S3 for catastrophic failures.</li> <li>Raw data is stored in a Spark cluster.</li> </ul> <h3><a href="http://highscalability.com/blog/2014/11/10/nifty-architecture-tricks-from-wix-building-a-publishing-pla.html">Nifty Architecture Tricks From Wix - Building A Publishing Platform At Scale</a></h3> <p>Platform on which over 54 million web sites have been created, most of which receive under 100 page views per day. So traditional caching does not apply.</p> <p>Novel consequences:</p> <ul> <li>Does not use transactions. all data is immutable and they use a simple eventual consistency strategy.</li> <li>Does not cache. Instead, they pay great attention to optimizing the rendering path so that every page displays in under 100ms.</li> </ul> <h4>Guidelines For How To Build Services</h4> <ul> <li>Services are stateless. To horizontally scale, just add more servers.</li> <li>With the exception of billing/financial transactions, all other services do not use transactions.</li> <li>When designing a new service caching is not part of the architecture. If a service performs poorly, first optimize the code before emplying caching.</li> </ul> <h4>Editor Segment</h4> <ul> <li>MySQL is a great key-value store. Key is based on a hash function of the file so the key is immutable.</li> <li>An "active database" contains the 6% of sites that are still being updated. It is really fast and relatively small in terms of storage.</li> <li>An "archive database" contains the remaining data that is infrequently accessed. It is relatively slow, but contains huge amounts of storage.</li> </ul> <h4>High Availability For Editor Segment</h4> <ul> <li>All data is immutable. Revisions are stored for everything. If a corruption can’t be fixed, revert to version where the data was fine.</li> <li>To protect against unavailability, data s replicated across different geographical locations and multiple cloud services.</li> </ul> <h4>Modeling Data With No Database Transactions</h4> <ul> <li>JSON files are inserted into the database, and finally a manifest is inserted with the identifiers of all resources that were just saved.</li> </ul> <h4>Public Segment</h4> <ul> <li>Public SLA is that response time is &lt; 100ms at peak traffic. Web sites must be fast with no caching.</li> <li>Data is stored in a denormalized format, optimized for read by primary key. Everything that is needed is returned in a single request.</li> <li>The data returned is in JSON format. Rendering is offloaded to the client.</li> </ul> <h4>Lessons Learned</h4> <ul> <li>Identify your critical path and concerns.</li> <li>De-normalize and precalculate data, and thereby reduce network chattter, for performance.</li> <li>Take advantage of client’s CPU power.</li> <li>Go immutable. Immutability has far reaching consequences for an architecture, and is an elegant solution to a lot of problems.</li> <li>What really locks you down is data. Moving lots of data to a different cloud is really hard.</li> </ul> <h3><a href="http://highscalability.com/blog/2014/7/21/stackoverflow-update-560m-pageviews-a-month-25-servers-and-i.html">StackOverflow Update: 560M Pageviews A Month, 25 Servers, And It's All About Performance</a></h3> <ul> <li>25 servers for everything. That’s high availability, load balancing, caching, databases, searching, and utility functions.</li> <li>Still uses Microsoft products.</li> <li>Uses a scale-up strategy. Not using any cloud service, where their SQL Servers loaded with 384 GB of RAM and 2TB of SSD would cost a fortune.</li> <li>Jeff Atwood said "Hardware is cheap, programmers are expensive" and this apparently still holds true.</li> </ul> <h4>Stats</h4> <ul> <li>Peak is 2600-3000 requests/sec on most weekdays.</li> <li>Each web server has 2x 320GB SSDs in a RAID 1, and each ElasticSearch box has 300 GB also using SSDs.</li> <li>Has a 40:60 read-write ratio.</li> <li>11 web servers using IIS, 2 load balancers using HAProxy, 4 active database nodes using MS SQL, 3 application servers implementing the tag engine, 3 machines running ElasticSearch, 2 machines running Redis for distributed cache and messaging.</li> </ul> <h4>Platform</h4> <ul> <li>ElasticSearch, Redis, HAProxy, and MS SQL.</li> </ul> <h4>Servers</h4> <ul> <li>The database server is at 10% usage except during backups. The database servers have 384GB of RAM and the web servers are at 10%-15% CPU usage.</li> <li>Scale-up is still working. Other scale-out sites with a similar numbers tend to run on between 100 and 300 servers.</li> </ul> <h4>SSDs</h4> <ul> <li>ElasticSearch performs much better on SSDs, given SO writes/re-indexes very frequently.</li> </ul> <h4>High Availability</h4> <ul> <li>The main datacenter is in New York and the backup datacenter is in Oregon.</li> <li>Not everything is replicated between data centers (very temporary cache data that's not needed to eat bandwidth by syncing, etc.), but the important resources are.</li> <li>Nginx was used for SSL, but a transition has been made to using HAProxy to terminate SSL.</li> </ul> <h4>Databasing</h4> <ul> <li>Each site has its own database, and the schema for each is the same. This is effectively a form of partitioning and horizontal scaling.</li> <li>All the schema changes are applied to all site databases at the same time.</li> <li>Partitioning is not required. Indexing takes care of everything and the data just is not large enough.</li> <li>Scores are denormalized, so querying is often needed.</li> <li>The Tag Engine is entirely self-contained, and runs on three servers for redundancy. If all of them fail, the local web servers will load the tag engine in memory and keep on going.</li> </ul> <h4>Coding</h4> <ul> <li>Compilation is very fast.</li> <li>New features are hidden via feature switches.</li> <li>Heavy usage of static classes and methods, for simplicity and better performance.</li> </ul> <h4>Caching</h4> <ul> <li>Network level cache is caching in the browser, CDN, and proxies.</li> <li>Redis works as a distributed in-memory key-value store for all servers that serve the same site.</li> <li>SQL Server Cache caches the entire database in memory.</li> <li>SSD is used as a cache, and is usually only hit when the SQL server cache is warming up.</li> <li>TODO</li> </ul> https://tedneward.github.io/Research/reading/development/linking-loading/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/linking-loading/index.html <ul> <li>Advanced C and C++ Compiling <ul> <li>2014; Milan Stevanovic​</li> <li>"Engineering guide to C/C++ compiling, linking, and binary files structure"</li> <li><a href="http://www.apress.com/9781430266679">http://www.apress.com/9781430266679</a></li> <li><a href="http://link.springer.com/book/10.1007%2F978-1-4302-6668-6">http://link.springer.com/book/10.1007%2F978-1-4302-6668-6</a></li> <li><a href="https://github.com/apress/adv-c-cpp-compiling">https://github.com/apress/adv-c-cpp-compiling</a></li> </ul> </li> <li>Linkers and Loaders <ul> <li>1999; John R. Levine</li> <li><a href="https://www.iecc.com/linker/">https://www.iecc.com/linker/</a></li> </ul> </li> <li>Assemblers and Loaders <ul> <li><a href="https://www.davidsalomon.name/">https://www.davidsalomon.name/</a></li> <li>Written during 1985--1992, the book developed from class notes on computer organization. It includes numerous exercises with answers provided, and review questions following each chapter. The book is bound as a paperback with a colorful cover.</li> <li><a href="https://www.davidsalomon.name/assem.advertis/AssemAd.html">https://www.davidsalomon.name/assem.advertis/AssemAd.html</a> (<a href="http://www.davidsalomon.name/assem.advertis/asl.pdf">PDF</a>) (<a href="./asl.pdf">Local</a>)</li> </ul> </li> <li> <p>Beginner's Guide to Linkers</p> <ul> <li>2010; David Drysdale</li> <li><a href="https://www.lurklurk.org/linkers/linkers.html">https://www.lurklurk.org/linkers/linkers.html</a></li> </ul> </li> <li>Linkers and Loaders <ul> <li>ACM Computing Surveys, Volume 4, Number 3, September 1972</li> <li>Leon Presser, John R. White</li> <li><a href="http://www-inst.eecs.berkeley.edu/~cs162/sp06/hand-outs/p149-presser-linker-loader.pdf">http://www-inst.eecs.berkeley.edu/~cs162/sp06/hand-outs/p149-presser-linker-loader.pdf</a></li> </ul> </li> <li>Binary File Descriptor library (BFD) <ul> <li><a href="https://sourceware.org/binutils/docs/bfd/">https://sourceware.org/binutils/docs/bfd/</a></li> <li><a href="https://publicclu2.blogspot.com/2013/05/binary-file-descriptor-library-bfd.html">https://publicclu2.blogspot.com/2013/05/binary-file-descriptor-library-bfd.html</a></li> </ul> </li> <li>Concurrent Linking with the GNU Gold Linker <ul> <li>2013 Thesis; Sander Mathijs van Veen</li> <li><a href="https://smvv.io/gold.pdf">https://smvv.io/gold.pdf</a></li> </ul> </li> <li>Oracle Solaris 11.1 Linkers and Libraries Guide <ul> <li><a href="https://docs.oracle.com/cd/E26502_01/html/E26507/">https://docs.oracle.com/cd/E26502_01/html/E26507/</a></li> </ul> </li> <li>The Missing Link: Explaining ELF Static Linking, Semantically <ul> <li>OOPSLA 2016</li> <li>Stephen Kell, Dominic P. Mulligan, Peter Sewell</li> <li><a href="http://www.cl.cam.ac.uk/~pes20/rems/papers/oopsla-elf-linking-2016.pdf">http://www.cl.cam.ac.uk/~pes20/rems/papers/oopsla-elf-linking-2016.pdf</a></li> </ul> </li> <li>The Sad State of Symbol Aliases <ul> <li><a href="http://blog.omega-prime.co.uk/2011/07/06/the-sad-state-of-symbol-aliases/">http://blog.omega-prime.co.uk/2011/07/06/the-sad-state-of-symbol-aliases/</a></li> </ul> </li> <li>The Theory, History and Future of System Linkers <ul> <li>HelloGCC Workshop 2013; Luba Tang</li> <li><a href="https://github.com/hellogcc/HelloGCC2013/blob/master/linker_overall.pdf">https://github.com/hellogcc/HelloGCC2013/blob/master/linker_overall.pdf</a></li> </ul> </li> </ul> <h2>Readings: Blogs</h2> <ul> <li>Eli Bendersky - Linkers and loaders <ul> <li><a href="https://eli.thegreenplace.net/tag/linkers-and-loaders">https://eli.thegreenplace.net/tag/linkers-and-loaders</a></li> <li>Load-time relocation of shared libraries <ul> <li><a href="https://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries">https://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries</a></li> </ul> </li> <li>Position Independent Code (PIC) in shared libraries <ul> <li><a href="https://eli.thegreenplace.net/2011/11/03/position-independent-code-pic-in-shared-libraries">https://eli.thegreenplace.net/2011/11/03/position-independent-code-pic-in-shared-libraries</a></li> </ul> </li> <li>Position Independent Code (PIC) in shared libraries on x64 <ul> <li><a href="https://eli.thegreenplace.net/2011/11/11/position-independent-code-pic-in-shared-libraries-on-x64">https://eli.thegreenplace.net/2011/11/11/position-independent-code-pic-in-shared-libraries-on-x64</a></li> </ul> </li> <li>How statically linked programs run on Linux <ul> <li><a href="https://eli.thegreenplace.net/2012/08/13/how-statically-linked-programs-run-on-linux">https://eli.thegreenplace.net/2012/08/13/how-statically-linked-programs-run-on-linux</a></li> </ul> </li> <li>Library order in static linking <ul> <li><a href="https://eli.thegreenplace.net/2013/07/09/library-order-in-static-linking">https://eli.thegreenplace.net/2013/07/09/library-order-in-static-linking</a></li> </ul> </li> </ul> </li> <li>Fun with weak symbols <ul> <li>2012; Mike Hommey</li> <li><a href="https://glandium.org/blog/?p=2388">https://glandium.org/blog/?p=2388</a></li> </ul> </li> <li>Ian Lance Taylor <ul> <li>20 part linker essay by Ian Lance Taylor <ul> <li>Notes: Day 40: Linkers are amazing. <ul> <li>2013; Julia Evans</li> <li><a href="http://jvns.ca/blog/2013/12/10/day-40-learning-about-linkers/">http://jvns.ca/blog/2013/12/10/day-40-learning-about-linkers/</a></li> </ul> </li> <li>TOC: <a href="https://checkoway.net/musings/linkers/">https://checkoway.net/musings/linkers/</a></li> <li>TOC: <a href="https://lwn.net/Articles/276782/">https://lwn.net/Articles/276782/</a></li> <li>1. Introduction, personal history, first half of what's-a-linker - <a href="http://www.airs.com/blog/archives/38">http://www.airs.com/blog/archives/38</a></li> <li>2. What's-a-linker: Dynamic linking, linker data types, linker operation - <a href="http://www.airs.com/blog/archives/39">http://www.airs.com/blog/archives/39</a></li> <li>3. Address spaces, Object file formats - <a href="http://www.airs.com/blog/archives/40">http://www.airs.com/blog/archives/40</a></li> <li>4. Shared Libraries - <a href="http://www.airs.com/blog/archives/41">http://www.airs.com/blog/archives/41</a></li> <li>5. More Shared Libraries -- specifically, linker implementation; ELF Symbols - <a href="http://www.airs.com/blog/archives/42">http://www.airs.com/blog/archives/42</a></li> <li>6. Relocations, Position Dependent Shared Libraries - <a href="http://www.airs.com/blog/archives/43">http://www.airs.com/blog/archives/43</a></li> <li>7. Thread Local Storage (TLS) optimization - <a href="http://www.airs.com/blog/archives/44">http://www.airs.com/blog/archives/44</a></li> <li>8. ELF Segments and Sections - <a href="http://www.airs.com/blog/archives/45">http://www.airs.com/blog/archives/45</a></li> <li>9. Symbol Versions, Relaxation optimization - <a href="http://www.airs.com/blog/archives/46">http://www.airs.com/blog/archives/46</a></li> <li>10. Parallel linking - <a href="http://www.airs.com/blog/archives/47">http://www.airs.com/blog/archives/47</a></li> <li>11. Archive format - <a href="http://www.airs.com/blog/archives/48">http://www.airs.com/blog/archives/48</a></li> <li>12. Symbol resolution - <a href="http://www.airs.com/blog/archives/49">http://www.airs.com/blog/archives/49</a></li> <li>13. Symbol resolution from the user's point of view; Static Linking vs. Dynamic Linking - <a href="http://www.airs.com/blog/archives/50">http://www.airs.com/blog/archives/50</a></li> <li>14. Link time optimization, aka Whole Program optimization; Initialization Code - <a href="http://www.airs.com/blog/archives/51">http://www.airs.com/blog/archives/51</a></li> <li>15. COMDAT sections - <a href="http://www.airs.com/blog/archives/52">http://www.airs.com/blog/archives/52</a></li> <li>16. C++ Template Instantiation, Exception Frames - <a href="http://www.airs.com/blog/archives/53">http://www.airs.com/blog/archives/53</a></li> <li>17. Warning Symbols - <a href="http://www.airs.com/blog/archives/54">http://www.airs.com/blog/archives/54</a></li> <li>18. Incremental Linking - <a href="http://www.airs.com/blog/archives/55">http://www.airs.com/blog/archives/55</a></li> <li>19. <code>__start</code> and <code>__stop</code> Symbols, Byte Swapping - <a href="http://www.airs.com/blog/archives/56">http://www.airs.com/blog/archives/56</a></li> <li>20. Last post; Update on gold's status - <a href="http://www.airs.com/blog/archives/57">http://www.airs.com/blog/archives/57</a></li> </ul> </li> <li>Protected Symbols - <a href="https://www.airs.com/blog/archives/307">https://www.airs.com/blog/archives/307</a></li> <li>gold <ul> <li>Gold Workqueues - <a href="https://www.airs.com/blog/archives/78">https://www.airs.com/blog/archives/78</a></li> <li>STT_GNU_IFUNC - <a href="https://www.airs.com/blog/archives/403">https://www.airs.com/blog/archives/403</a></li> <li>A New ELF Linker <ul> <li>GCC Summit 2008</li> <li>Proceedings of the GCC Developers' Summit (2008)</li> <li><a href="https://research.google.com/pubs/pub34417.html">https://research.google.com/pubs/pub34417.html</a></li> <li><a href="https://gcc.gnu.org/wiki/HomePage?action=AttachFile&amp;do=view&amp;target=gcc-2008-proceedings.pdf">https://gcc.gnu.org/wiki/HomePage?action=AttachFile&amp;do=view&amp;target=gcc-2008-proceedings.pdf</a></li> <li><a href="http://airs.com/ian/gold-slides.pdf">http://airs.com/ian/gold-slides.pdf</a></li> </ul> </li> </ul> </li> </ul> </li> <li>Nick Desaulniers <ul> <li>Part 1 – Object Files and Symbols - <a href="https://nickdesaulniers.github.io/blog/2016/08/13/object-files-and-symbols/">https://nickdesaulniers.github.io/blog/2016/08/13/object-files-and-symbols/</a></li> <li>Part 2 – Static and Dynamic Libraries - <a href="https://nickdesaulniers.github.io/blog/2016/11/20/static-and-dynamic-libraries/">https://nickdesaulniers.github.io/blog/2016/11/20/static-and-dynamic-libraries/</a></li> </ul> </li> <li>Stephen Kell <ul> <li>Linking and loading: what's incidental? <ul> <li><a href="http://www.cl.cam.ac.uk/~srk31/blog/research/linking-and-loading-complexity.html">http://www.cl.cam.ac.uk/~srk31/blog/research/linking-and-loading-complexity.html</a></li> </ul> </li> <li>Dynamic linking and security <ul> <li><a href="http://www.cl.cam.ac.uk/~srk31/blog/research/dynamic-linker-security.html">http://www.cl.cam.ac.uk/~srk31/blog/research/dynamic-linker-security.html</a></li> </ul> </li> <li>C libraries and linking <ul> <li><a href="http://www.cl.cam.ac.uk/~srk31/blog/research/c-libraries-compilers-and-linking.html">http://www.cl.cam.ac.uk/~srk31/blog/research/c-libraries-compilers-and-linking.html</a></li> </ul> </li> <li>Link order <ul> <li><a href="http://www.cl.cam.ac.uk/~srk31/blog/devel/link-order.html">http://www.cl.cam.ac.uk/~srk31/blog/devel/link-order.html</a></li> </ul> </li> <li>Weak dynamic symbols <ul> <li><a href="http://www.cl.cam.ac.uk/~srk31/blog/devel/weak-symbols.html">http://www.cl.cam.ac.uk/~srk31/blog/devel/weak-symbols.html</a></li> </ul> </li> </ul> </li> <li>Trevor Pounds <ul> <li>Versioning Symbols for Shared Libraries (glibc) <ul> <li><a href="https://web.archive.org/web/20150906233231/http://www.trevorpounds.com/blog/?p=33">https://web.archive.org/web/20150906233231/http://www.trevorpounds.com/blog/?p=33</a></li> </ul> </li> <li>Linking to Older Versioned Symbols (glibc) <ul> <li><a href="https://web.archive.org/web/20150906233154/http://www.trevorpounds.com/blog/?p=103">https://web.archive.org/web/20150906233154/http://www.trevorpounds.com/blog/?p=103</a></li> </ul> </li> </ul> </li> </ul> <h2>Readings: Linker Scripts</h2> <ul> <li>OSDev Wiki <ul> <li><a href="https://wiki.osdev.org/Category:Linkers">https://wiki.osdev.org/Category:Linkers</a></li> <li>Linker Scripts <ul> <li><a href="http://wiki.osdev.org/Linker_Scripts">http://wiki.osdev.org/Linker_Scripts</a></li> </ul> </li> </ul> </li> <li>LD - Linker Scripts <ul> <li><a href="https://sourceware.org/binutils/docs/ld/Scripts.html">https://sourceware.org/binutils/docs/ld/Scripts.html</a></li> </ul> </li> </ul> <h2>Readings: LTO</h2> <ul> <li>(Ab)using LTO plugin API for system-wide shrinking of dynamic libraries <ul> <li>GNU Tools Cauldron 2018</li> <li>Vladislav Ivanishin</li> <li><a href="https://www.youtube.com/watch?v=BNYGB7dHgkc">https://www.youtube.com/watch?v=BNYGB7dHgkc</a></li> <li><a href="https://gcc.gnu.org/wiki/cauldron2018?action=AttachFile&amp;do=view&amp;target=lto-plugin-api-abuse.pdf">https://gcc.gnu.org/wiki/cauldron2018?action=AttachFile&amp;do=view&amp;target=lto-plugin-api-abuse.pdf</a></li> </ul> </li> <li>GCC Wiki - LinkTimeOptimization <ul> <li><a href="https://gcc.gnu.org/wiki/LinkTimeOptimization">https://gcc.gnu.org/wiki/LinkTimeOptimization</a></li> </ul> </li> <li>Honza Hubička's Blog <ul> <li>Linktime optimization in GCC, part 1 - brief history <ul> <li><a href="http://hubicka.blogspot.com/2014/04/linktime-optimization-in-gcc-1-brief.html">http://hubicka.blogspot.com/2014/04/linktime-optimization-in-gcc-1-brief.html</a></li> </ul> </li> <li>Linktime optimization in GCC, part 2 - Firefox <ul> <li><a href="http://hubicka.blogspot.com/2014/04/linktime-optimization-in-gcc-2-firefox.html">http://hubicka.blogspot.com/2014/04/linktime-optimization-in-gcc-2-firefox.html</a></li> </ul> </li> <li>Linktime optimization in GCC, part 3 - LibreOffice <ul> <li><a href="https://hubicka.blogspot.com/2014/09/linktime-optimization-in-gcc-part-3.html">https://hubicka.blogspot.com/2014/09/linktime-optimization-in-gcc-part-3.html</a></li> </ul> </li> <li>Link time and inter-procedural optimization improvements in GCC 5 <ul> <li><a href="https://hubicka.blogspot.com/2015/04/GCC5-IPA-LTO-news.html">https://hubicka.blogspot.com/2015/04/GCC5-IPA-LTO-news.html</a></li> </ul> </li> <li>Building libreoffice with GCC 6 and LTO <ul> <li><a href="https://hubicka.blogspot.com/2016/03/building-libreoffice-with-gcc-6-and-lto.html">https://hubicka.blogspot.com/2016/03/building-libreoffice-with-gcc-6-and-lto.html</a></li> </ul> </li> <li>GCC 8: link time and interprocedural optimization <ul> <li><a href="https://hubicka.blogspot.com/2018/06/gcc-8-link-time-and-interprocedural.html">https://hubicka.blogspot.com/2018/06/gcc-8-link-time-and-interprocedural.html</a></li> </ul> </li> </ul> </li> <li>LLVM Link Time Optimization: Design and Implementation <ul> <li><a href="https://llvm.org/docs/LinkTimeOptimization.html">https://llvm.org/docs/LinkTimeOptimization.html</a></li> </ul> </li> <li>Optimizing Large Applications <ul> <li>2014 Master's Thesis; Martin Liška</li> <li><a href="https://arxiv.org/abs/1403.6997">https://arxiv.org/abs/1403.6997</a></li> </ul> </li> <li>Optimizing real world applications with GCC Link Time Optimization <ul> <li>2010 GCC Developers' Summit; T. Glek, J. Hubicka</li> <li><a href="https://arxiv.org/abs/1010.2196">https://arxiv.org/abs/1010.2196</a></li> </ul> </li> <li>ThinLTO <ul> <li>ThinLTO: Scalable and Incremental LTO <ul> <li><a href="http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html">http://blog.llvm.org/2016/06/thinlto-scalable-and-incremental-lto.html</a></li> </ul> </li> <li>ThinLTO: Scalable and Incremental Link-Time Optimization <ul> <li>CppCon 2017; Teresa Johnson</li> <li><a href="https://www.youtube.com/watch?v=p9nH2vZ2mNo">https://www.youtube.com/watch?v=p9nH2vZ2mNo</a></li> </ul> </li> <li>ThinLTO: Scalable and incremental LTO <ul> <li>CGO 2017</li> <li>Teresa Johnson, Mehdi Amini, and Xinliang David Li</li> <li><a href="https://research.google/pubs/pub47584/">https://research.google/pubs/pub47584/</a></li> </ul> </li> </ul> </li> </ul> <h2>Readings: OS</h2> <h3>macOS</h3> <ul> <li>Debugging Dyld <ul> <li><a href="https://lowlevelbits.org/debugging-dyld/">https://lowlevelbits.org/debugging-dyld/</a></li> </ul> </li> </ul> <h3>Linux</h3> <ul> <li>Linker limitations on 32-bit architectures <ul> <li>2019; Alexander E. Patrakov</li> <li><a href="https://lwn.net/Articles/797303/">https://lwn.net/Articles/797303/</a></li> </ul> </li> <li>Linux EDR Evasion With Meterpreter and LD_PRELOAD <ul> <li><a href="https://forensicitguy.github.io/posts/linux-edr-evasion-with-ld-preload/">https://forensicitguy.github.io/posts/linux-edr-evasion-with-ld-preload/</a></li> </ul> </li> <li>Linux Internals <ul> <li>Dynamic Linking Wizardry <ul> <li><a href="https://0x00sec.org/t/linux-internals-dynamic-linking-wizardry/1082">https://0x00sec.org/t/linux-internals-dynamic-linking-wizardry/1082</a></li> </ul> </li> <li>The Art Of Symbol Resolution <ul> <li><a href="https://0x00sec.org/t/linux-internals-the-art-of-symbol-resolution/1488">https://0x00sec.org/t/linux-internals-the-art-of-symbol-resolution/1488</a></li> </ul> </li> </ul> </li> </ul> <h3>Solaris</h3> <ul> <li>Surfing With The Linker Aliens: Solaris Linking &amp; ELF Blogs <ul> <li><a href="http://www.linker-aliens.org/">http://www.linker-aliens.org/</a></li> <li>The Link-editors - a source tour <ul> <li><a href="http://www.linker-aliens.org/blogs/rie/entry/the_link_editors_a_source/">http://www.linker-aliens.org/blogs/rie/entry/the_link_editors_a_source/</a></li> </ul> </li> <li>What Are "Tentative" Symbols? <ul> <li><a href="http://www.linker-aliens.org/blogs/ali/entry/what_are_tentative_symbols/">http://www.linker-aliens.org/blogs/ali/entry/what_are_tentative_symbols/</a></li> </ul> </li> </ul> </li> </ul> <h3>Windows</h3> <ul> <li>Raymond Chen - The Old New Thing <ul> <li>Understanding the classical model for linking <ul> <li><a href="https://blogs.msdn.microsoft.com/oldnewthing/tag/linker">https://blogs.msdn.microsoft.com/oldnewthing/tag/linker</a></li> <li>Groundwork: The algorithm <ul> <li><a href="https://devblogs.microsoft.com/oldnewthing/20130107-00/?p=5633">https://devblogs.microsoft.com/oldnewthing/20130107-00/?p=5633</a></li> </ul> </li> <li>Taking symbols along for the ride <ul> <li><a href="https://devblogs.microsoft.com/oldnewthing/20130108-00/?p=5623">https://devblogs.microsoft.com/oldnewthing/20130108-00/?p=5623</a></li> </ul> </li> <li>You can override an LIB with another LIB, and a LIB with an OBJ, but you can’t override an OBJ <ul> <li><a href="https://devblogs.microsoft.com/oldnewthing/20130109-00/?p=5613">https://devblogs.microsoft.com/oldnewthing/20130109-00/?p=5613</a></li> </ul> </li> <li>Sometimes you don’t want a symbol to come along for a ride <ul> <li><a href="https://devblogs.microsoft.com/oldnewthing/20130110-00/?p=5593">https://devblogs.microsoft.com/oldnewthing/20130110-00/?p=5593</a></li> </ul> </li> <li>Understanding errors in classical linking: The delay-load catch-22 <ul> <li><a href="https://devblogs.microsoft.com/oldnewthing/20130111-00/?p=5583">https://devblogs.microsoft.com/oldnewthing/20130111-00/?p=5583</a></li> </ul> </li> </ul> </li> <li>Using linker segments and <code>__declspec(allocate(…))</code> to arrange data in a specific order <ul> <li><a href="https://blogs.msdn.microsoft.com/oldnewthing/20181107-00/?p=100155">https://blogs.msdn.microsoft.com/oldnewthing/20181107-00/?p=100155</a></li> </ul> </li> <li>Gotchas when using linker sections to arrange data, part 1 <ul> <li><a href="https://blogs.msdn.microsoft.com/oldnewthing/20181108-00/?p=100165">https://blogs.msdn.microsoft.com/oldnewthing/20181108-00/?p=100165</a></li> </ul> </li> <li>Gotchas when using linker sections to arrange data, part 2 <ul> <li><a href="https://blogs.msdn.microsoft.com/oldnewthing/20181109-00/?p=100175">https://blogs.msdn.microsoft.com/oldnewthing/20181109-00/?p=100175</a></li> </ul> </li> <li>Why would the incremental linker insert padding between section fragments? <ul> <li><a href="https://blogs.msdn.microsoft.com/oldnewthing/20190114-00/?p=100695">https://blogs.msdn.microsoft.com/oldnewthing/20190114-00/?p=100695</a></li> </ul> </li> </ul> </li> </ul> <h2>Readings: Relocations</h2> <ul> <li>Binary Dodo - Arun <ul> <li>Investigating linking with COMMON symbols in ELF <ul> <li><a href="https://binarydodo.wordpress.com/2016/05/09/investigating-linking-with-common-symbols-in-elf/">https://binarydodo.wordpress.com/2016/05/09/investigating-linking-with-common-symbols-in-elf/</a></li> </ul> </li> <li>Symbol binding types in ELF and their effect on linking of relocatable files <ul> <li><a href="https://binarydodo.wordpress.com/2016/05/12/symbol-binding-types-in-elf-and-their-effect-on-linking-of-relocatable-files/">https://binarydodo.wordpress.com/2016/05/12/symbol-binding-types-in-elf-and-their-effect-on-linking-of-relocatable-files/</a></li> </ul> </li> <li>Symbol resolution during link-editing <ul> <li><a href="https://binarydodo.wordpress.com/2016/07/01/symbol-resolution-during-link-editing/">https://binarydodo.wordpress.com/2016/07/01/symbol-resolution-during-link-editing/</a></li> </ul> </li> </ul> </li> <li>ELF Binaries and Relocation Entries - Stafford Horne <ul> <li><a href="http://stffrdhrn.github.io/hardware/embedded/openrisc/2019/11/29/relocs.html">http://stffrdhrn.github.io/hardware/embedded/openrisc/2019/11/29/relocs.html</a></li> </ul> </li> <li><a href="https://intezer.com/tag/elf/">Executable and Linkable Format 101</a> - Ignacio Sanmillan <ul> <li>Part 3: Relocations - <a href="https://www.intezer.com/executable-and-linkable-format-101-part-3-relocations/">https://www.intezer.com/executable-and-linkable-format-101-part-3-relocations/</a></li> </ul> </li> <li>GOT and PLT <ul> <li><a href="https://systemoverlord.com/2017/03/19/got-and-plt-for-pwning.html">https://systemoverlord.com/2017/03/19/got-and-plt-for-pwning.html</a></li> </ul> </li> <li>Ian Lance Taylor <ul> <li>Linkers 2. What's-a-linker: Dynamic linking, linker data types, linker operation <ul> <li><a href="http://www.airs.com/blog/archives/39">http://www.airs.com/blog/archives/39</a></li> </ul> </li> <li>Linkers 5. More Shared Libraries -- specifically, linker implementation; ELF Symbols <ul> <li><a href="http://www.airs.com/blog/archives/42">http://www.airs.com/blog/archives/42</a></li> </ul> </li> <li>Linkers 6. Relocations, Position Dependent Shared Libraries <ul> <li><a href="http://www.airs.com/blog/archives/43">http://www.airs.com/blog/archives/43</a></li> </ul> </li> <li>Linkers 9. Symbol Versions, Relaxation optimization <ul> <li><a href="http://www.airs.com/blog/archives/46">http://www.airs.com/blog/archives/46</a></li> </ul> </li> <li>Linker combreloc <ul> <li><a href="https://www.airs.com/blog/archives/186">https://www.airs.com/blog/archives/186</a></li> </ul> </li> <li>Linker relro <ul> <li><a href="https://www.airs.com/blog/archives/189">https://www.airs.com/blog/archives/189</a></li> </ul> </li> </ul> </li> <li>Introduction to the ELF Format - Keith Makan <ul> <li>Part VI(1): The Symbol Table and Relocations - <a href="https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi.html">https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi.html</a></li> <li>Part VI(2): The Symbol Table and Relocations - <a href="https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi_18.html">https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi_18.html</a></li> <li>Part VI(3): More Relocation tricks - r_addend execution - <a href="https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi-more.html">https://blog.k3170makan.com/2018/10/introduction-to-elf-format-part-vi-more.html</a></li> </ul> </li> <li>Linkers and loaders - Eli Bendersky - <a href="http://eli.thegreenplace.net/tag/linkers-and-loaders">http://eli.thegreenplace.net/tag/linkers-and-loaders</a> <ul> <li>Load-time relocation of shared libraries <ul> <li><a href="https://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries">https://eli.thegreenplace.net/2011/08/25/load-time-relocation-of-shared-libraries</a></li> </ul> </li> </ul> </li> <li>Making our own executable packer - Amos Wenger <ul> <li>ELF relocations - <a href="https://fasterthanli.me/blog/2020/elf-relocations/">https://fasterthanli.me/blog/2020/elf-relocations/</a></li> <li>More ELF relocations - <a href="https://fasterthanli.me/blog/2020/more-elf-relocations/">https://fasterthanli.me/blog/2020/more-elf-relocations/</a></li> </ul> </li> <li>Moving code around - Thiago Macieira <ul> <li><a href="http://blog.qt.io/blog/2010/12/04/moving-code-around/">http://blog.qt.io/blog/2010/12/04/moving-code-around/</a></li> <li><a href="http://blog.qt.io/blog/2010/12/05/moving-code-around-more-easily/">http://blog.qt.io/blog/2010/12/05/moving-code-around-more-easily/</a></li> </ul> </li> <li>Oracle Solaris 11.1 Linker and Libraries Guide <ul> <li>Relocation Processing <ul> <li><a href="https://docs.oracle.com/cd/E26502_01/html/E26507/chapter3-29.html">https://docs.oracle.com/cd/E26502_01/html/E26507/chapter3-29.html</a></li> </ul> </li> <li>Relocation Sections <ul> <li><a href="https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-54839.html">https://docs.oracle.com/cd/E23824_01/html/819-0690/chapter6-54839.html</a></li> </ul> </li> </ul> </li> <li>Relocations in ELF Toolchains - Palmer Dabbelt <ul> <li><a href="https://www.sifive.com/blog/2017/08/21/all-aboard-part-2-relocations/">https://www.sifive.com/blog/2017/08/21/all-aboard-part-2-relocations/</a></li> </ul> </li> <li>Relocations: fantastic symbols, but where to find them? - Siddhesh Poyarekar <ul> <li><a href="https://siddhesh.in/posts/relocations-fantastic-symbols-but-where-to-find-them.html">https://siddhesh.in/posts/relocations-fantastic-symbols-but-where-to-find-them.html</a></li> </ul> </li> <li>Resolving ELF Relocation Name / Symbols - Chris Rohlf <ul> <li><a href="https://em386.blogspot.com/2006/10/resolving-elf-relocation-name-symbols.html">https://em386.blogspot.com/2006/10/resolving-elf-relocation-name-symbols.html</a></li> </ul> </li> </ul> <h2>Readings: Security</h2> <ul> <li>An Evil Copy: How the Loader Betrays You <ul> <li>Network and Distributed System Security Symposium (NDSS) 2017</li> <li>Xinyang Ge, Mathias Payer, Trent Jaeger</li> <li><a href="https://nebelwelt.net/publications/files/17NDSS.pdf">https://nebelwelt.net/publications/files/17NDSS.pdf</a></li> <li><a href="https://www.microsoft.com/en-us/research/publication/evil-copy-loader-betrays/">https://www.microsoft.com/en-us/research/publication/evil-copy-loader-betrays/</a></li> </ul> </li> <li>Breaking the links: Exploiting the linker <ul> <li>2011; Tim Brown</li> <li><a href="https://packetstormsecurity.com/files/102814/Breaking-The-Links-Exploiting-The-Linker.html">https://packetstormsecurity.com/files/102814/Breaking-The-Links-Exploiting-The-Linker.html</a></li> <li>paper: <a href="https://dl.packetstormsecurity.net/papers/general/BTL.pdf">https://dl.packetstormsecurity.net/papers/general/BTL.pdf</a></li> <li>slides: <a href="https://labs.portcullis.co.uk/download/BTLCC.pdf">https://labs.portcullis.co.uk/download/BTLCC.pdf</a></li> </ul> </li> <li>Dynamic Loader Oriented Programming on Linux <ul> <li>Reversing and Offensive-oriented Trends Symposium (ROOTS) 2017</li> <li>Julian Kirsch, Bruno Bierbaumer, Thomas Kittel, and Claudia Eckert</li> <li><a href="https://kirschju.re/projects/static/kirsch-roots-2017-paper.pdf">https://kirschju.re/projects/static/kirsch-roots-2017-paper.pdf</a></li> <li><a href="https://github.com/kirschju/wiedergaenger">https://github.com/kirschju/wiedergaenger</a></li> </ul> </li> <li>Types for the Chain of Trust: No (loader) write left behind <ul> <li>2018 PhD dissertation; Rebecca .bx Shapiro</li> <li><a href="http://typedregions.com/">http://typedregions.com/</a></li> <li>Dissertation: <a href="http://typedregions.com/main.pdf">http://typedregions.com/main.pdf</a></li> <li>Defense slides: <a href="http://typedregions.com/defense-slides.pdf">http://typedregions.com/defense-slides.pdf</a></li> </ul> </li> </ul> <h2>Readings: Shared Libraries</h2> <ul> <li>Fun with weak dynamic linking <ul> <li><a href="https://glandium.org/blog/?p=2764">https://glandium.org/blog/?p=2764</a></li> </ul> </li> <li>How To Write Shared Libraries - Ulrich Drepper <ul> <li><a href="https://www.akkadia.org/drepper/dsohowto.pdf">https://www.akkadia.org/drepper/dsohowto.pdf</a></li> </ul> </li> <li>Inlining — shared libraries are special <ul> <li><a href="https://kristerw.blogspot.com/2016/11/inlining-shared-libraries-are-special.html">https://kristerw.blogspot.com/2016/11/inlining-shared-libraries-are-special.html</a></li> </ul> </li> <li>PLT and GOT - the key to code sharing and dynamic libraries <ul> <li><a href="https://www.technovelty.org/linux/plt-and-got-the-key-to-code-sharing-and-dynamic-libraries.html">https://www.technovelty.org/linux/plt-and-got-the-key-to-code-sharing-and-dynamic-libraries.html</a></li> </ul> </li> <li>Shared Libraries: Understanding Dynamic Loading <ul> <li><a href="http://amir.rachum.com/blog/2016/09/17/shared-libraries/">http://amir.rachum.com/blog/2016/09/17/shared-libraries/</a></li> </ul> </li> <li>Software Multiplexing: Share Your Libraries and Statically Link Them Too <ul> <li>SPLASH 2018 OOPSLA</li> <li>Will Dietz, Vikram Adve</li> <li><a href="https://2018.splashcon.org/event/splash-2018-oopsla-software-multiplexing-share-your-libraries-and-statically-link-them-too">https://2018.splashcon.org/event/splash-2018-oopsla-software-multiplexing-share-your-libraries-and-statically-link-them-too</a></li> <li><a href="https://publish.illinois.edu/allvm-project/">https://publish.illinois.edu/allvm-project/</a></li> <li><a href="https://github.com/allvm/allvm-tools">https://github.com/allvm/allvm-tools</a></li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>GNU Linker Output Viewer &amp; Editor <ul> <li><a href="https://github.com/JonTheBurger/GLOVE">https://github.com/JonTheBurger/GLOVE</a></li> </ul> </li> <li>ld-limiter: Limit number of parallel link jobs <ul> <li><a href="https://github.com/yugr/ld-limiter">https://github.com/yugr/ld-limiter</a></li> </ul> </li> <li>libdlbind: Dynamic creation and update of ELF files, or: an allocator for JIT compilers <ul> <li>an extension to System V-style dynamic loaders to allow dynamic allocation of objects and dynamic binding of symbols</li> <li><a href="https://github.com/stephenrkell/libdlbind">https://github.com/stephenrkell/libdlbind</a></li> </ul> </li> <li>linkermapviz <ul> <li>Interactive visualization of GNU ld’s linker map with a tree map.</li> <li><a href="https://github.com/PromyLOPh/linkermapviz">https://github.com/PromyLOPh/linkermapviz</a></li> </ul> </li> <li>linksem: Semantic model for aspects of ELF static linking and DWARF debug information <ul> <li><a href="https://github.com/rems-project/linksem">https://github.com/rems-project/linksem</a></li> </ul> </li> <li>ShlibVisibilityChecker: Tool for locating internal symbols unnecessarily exported from shared libraries <ul> <li><a href="https://github.com/yugr/ShlibVisibilityChecker">https://github.com/yugr/ShlibVisibilityChecker</a></li> </ul> </li> </ul> <h2>Software: Linkers</h2> <ul> <li>ld - the GNU linker <ul> <li><a href="https://sourceware.org/binutils/docs/ld/">https://sourceware.org/binutils/docs/ld/</a></li> </ul> </li> <li>LLD - The LLVM Linker <ul> <li><a href="https://lld.llvm.org/">https://lld.llvm.org/</a></li> </ul> </li> </ul> <h2>Software: OS: macOS</h2> <ul> <li>ld64: Instructions on how to build the ld64 linker on macOS <ul> <li><a href="https://github.com/dmaclach/ld64">https://github.com/dmaclach/ld64</a></li> </ul> </li> <li>[lld] Initial commit for new Mach-O backend <ul> <li><a href="https://reviews.llvm.org/D75382">https://reviews.llvm.org/D75382</a></li> </ul> </li> <li>macdylibbundler: Utility to ease bundling libraries into executables for OSX <ul> <li><a href="https://github.com/auriamg/macdylibbundler">https://github.com/auriamg/macdylibbundler</a></li> </ul> </li> <li>zld: A faster version of Apple's linker <ul> <li><a href="https://github.com/michaeleisel/zld">https://github.com/michaeleisel/zld</a></li> </ul> </li> </ul> <h2>Software: OS: Linux</h2> <ul> <li>dlinject.py: Inject a shared library (i.e. arbitrary code) into a live linux process, without ptrace <ul> <li><a href="https://github.com/DavidBuchanan314/dlinject">https://github.com/DavidBuchanan314/dlinject</a></li> </ul> </li> <li>Faulty.lib: Dynamic linker for compressed libraries, with on-demand decompression (ELF Linux systems) <ul> <li><a href="https://github.com/glandium/faulty.lib">https://github.com/glandium/faulty.lib</a></li> </ul> </li> <li>Implib.so: POSIX equivalent of Windows DLL import libraries <ul> <li><a href="https://github.com/yugr/Implib.so">https://github.com/yugr/Implib.so</a></li> </ul> </li> <li>libosuction: A tool for stripping dynamic libraries of unneeded symbols <ul> <li><a href="https://github.com/ispras/libosuction">https://github.com/ispras/libosuction</a></li> <li>System-Wide Elimination of Unreferenced Code and Data in Dynamically Linked Programs <ul> <li>Ivannikov ISPRAS Open Conference (ISPRAS) 2017</li> <li>V. Ivanishin, E. Kudryashov, A. Monakov, D. Melnik, J. Lee</li> <li><a href="https://doi.org/10.1109/ISPRAS.2017.00007">https://doi.org/10.1109/ISPRAS.2017.00007</a></li> </ul> </li> </ul> </li> <li>libpreloadvaccine: Whitelisting LD_PRELOAD libraries using LD_AUDIT <ul> <li><a href="https://github.com/ForensicITGuy/libpreloadvaccine">https://github.com/ForensicITGuy/libpreloadvaccine</a></li> <li>Whitelisting LD_PRELOAD for Fun and No Profit <ul> <li>Shmoocon 2020; Tony Lambert</li> <li><a href="https://www.youtube.com/watch?v=nM9Y2Sky6S0">https://www.youtube.com/watch?v=nM9Y2Sky6S0</a></li> <li><a href="https://speakerdeck.com/forensicitguy/whitelisting-ld-preload-for-fun-and-no-profit">https://speakerdeck.com/forensicitguy/whitelisting-ld-preload-for-fun-and-no-profit</a></li> </ul> </li> </ul> </li> <li>libprocesshider: Hide a process under Linux using the ld preloader <ul> <li><a href="https://github.com/gianlucaborello/libprocesshider">https://github.com/gianlucaborello/libprocesshider</a></li> <li><a href="https://sysdig.com/blog/hiding-linux-processes-for-fun-and-profit/">https://sysdig.com/blog/hiding-linux-processes-for-fun-and-profit/</a></li> </ul> </li> </ul> <hr> <h1>Talks</h1> <h2>Talks: 2019</h2> <ul> <li>Dynamic Linking On GNU/Linux <ul> <li>emBO++ 2019; Florian Sowade</li> <li><a href="https://www.youtube.com/watch?v=G2r5JhVDddw">https://www.youtube.com/watch?v=G2r5JhVDddw</a></li> <li><a href="https://www.slideshare.net/FlorianSowade/dynamic-linking-on-gnulinux/">https://www.slideshare.net/FlorianSowade/dynamic-linking-on-gnulinux/</a></li> </ul> </li> <li>Pardon the Interposition—Modifying and Improving Software Behavior with Interposers <ul> <li>USENIX LISA 2019; Danny Chen</li> <li><a href="https://www.usenix.org/conference/lisa19/presentation/chen">https://www.usenix.org/conference/lisa19/presentation/chen</a></li> </ul> </li> <li>The ACM-NVIDIA Compiler summer school lectures (2019) <ul> <li><a href="https://nptel.ac.in/courses/128/106/128106009/">https://nptel.ac.in/courses/128/106/128106009/</a></li> <li><a href="https://www.youtube.com/playlist?list=PLPeEbErKGwN15rF-BnlngobapgIRZbAK4">https://www.youtube.com/playlist?list=PLPeEbErKGwN15rF-BnlngobapgIRZbAK4</a></li> <li>Program Execution Environment <ul> <li>part 1: <a href="https://www.youtube.com/watch?v=Sf4JnUcJxSQ">https://www.youtube.com/watch?v=Sf4JnUcJxSQ</a></li> <li>part 2: <a href="https://www.youtube.com/watch?v=e3DnPTUunZk">https://www.youtube.com/watch?v=e3DnPTUunZk</a></li> <li>part 3: <a href="https://www.youtube.com/watch?v=h__sj8IfONg">https://www.youtube.com/watch?v=h__sj8IfONg</a></li> <li>part 4: <a href="https://www.youtube.com/watch?v=yzYXrwxu6YM">https://www.youtube.com/watch?v=yzYXrwxu6YM</a></li> <li>part 5: <a href="https://www.youtube.com/watch?v=4_LK7nOSlbM">https://www.youtube.com/watch?v=4_LK7nOSlbM</a></li> <li>part 6: <a href="https://www.youtube.com/watch?v=CzAwzTkbMDw">https://www.youtube.com/watch?v=CzAwzTkbMDw</a></li> <li>part 7: <a href="https://www.youtube.com/watch?v=IWU5Av6YpEw">https://www.youtube.com/watch?v=IWU5Av6YpEw</a></li> <li>part 8: <a href="https://www.youtube.com/watch?v=Wd4NaS0_7Tc">https://www.youtube.com/watch?v=Wd4NaS0_7Tc</a></li> </ul> </li> </ul> </li> <li>What makes LLD so fast? <ul> <li>FOSDEM 2019; Peter Smith</li> <li><a href="https://fosdem.org/2019/schedule/event/llvm_lld/">https://fosdem.org/2019/schedule/event/llvm_lld/</a></li> <li><a href="https://www.youtube.com/watch?v=CeHhveHHzII">https://www.youtube.com/watch?v=CeHhveHHzII</a></li> </ul> </li> </ul> <h2>Talks: 2018</h2> <ul> <li>Behind the Scenes of the Xcode Build Process <ul> <li>WWDC 2018; Jake Petroules, Jürgen Ributzka, Devin Coughlin, Louis Gerbarg</li> <li><a href="https://developer.apple.com/videos/play/wwdc2018/415/">https://developer.apple.com/videos/play/wwdc2018/415/</a></li> </ul> </li> <li>The Bits Between the Bits: How We Get to main() <ul> <li>CppCon 2018; Matt Godbolt</li> <li><a href="https://www.youtube.com/watch?v=dOfucXtyEsU">https://www.youtube.com/watch?v=dOfucXtyEsU</a></li> <li>Slides: <a href="https://mattgodbolt.github.io/cppcon-bits-between-bits/">https://mattgodbolt.github.io/cppcon-bits-between-bits/</a></li> </ul> </li> </ul> <h2>Talks: 2017</h2> <ul> <li>App Startup Time: Past, Present, and Future <ul> <li>WWDC 2017; Louis Gerbarg</li> <li><a href="https://developer.apple.com/videos/play/wwdc2017/413/">https://developer.apple.com/videos/play/wwdc2017/413/</a></li> <li>"Learn about the dyld dynamic linker used on Apple platforms, how it's changed over the years, and where it's headed next. Find out how improved tooling makes it easier to optimize your app's launch time, and see how new changes coming in dyld will bring even further launch time improvements."</li> </ul> </li> <li>LLD from a user's perspective <ul> <li>FOSDEM 2017; Peter Smith</li> <li><a href="https://archive.fosdem.org/2017/schedule/event/lld/">https://archive.fosdem.org/2017/schedule/event/lld/</a></li> <li><a href="https://archive.fosdem.org/2017/schedule/event/lld/attachments/slides/1446/export/events/attachments/lld/slides/1446/FosdemLLD2017.pdf">https://archive.fosdem.org/2017/schedule/event/lld/attachments/slides/1446/export/events/attachments/lld/slides/1446/FosdemLLD2017.pdf</a></li> </ul> </li> <li>My Little Object File: How Linkers Implement C++ <ul> <li>CppCon 2017; Michael Spencer</li> <li><a href="https://youtu.be/a5L66zguFe4">https://youtu.be/a5L66zguFe4</a></li> </ul> </li> <li>The Missing Link: The Curious symbiosis between C++ and the Linker <ul> <li>C++ Meetup Sydney hosted by IMC; December 6, 2017</li> <li>Dave Gittins</li> <li><a href="https://www.youtube.com/watch?v=twNoIhGKIoo">https://www.youtube.com/watch?v=twNoIhGKIoo</a></li> </ul> </li> </ul> <h2>Talks: 2016</h2> <ul> <li>New LLD linker for ELF: A high performance linker from the LLVM project <ul> <li>2016 EuroLLVM Developers' Meeting; R. Ueyama</li> <li><a href="https://www.youtube.com/watch?v=CYCRqjVa6l4">https://www.youtube.com/watch?v=CYCRqjVa6l4</a></li> <li><a href="https://llvm.org/devmtg/2016-03/Presentations/EuroLLVM%202016-%20New%20LLD%20linker%20for%20ELF.pdf">https://llvm.org/devmtg/2016-03/Presentations/EuroLLVM%202016-%20New%20LLD%20linker%20for%20ELF.pdf</a></li> </ul> </li> <li>Optimizing App Startup Time: Linkers, loaders, and you <ul> <li>WWDC 2016; Nick Kledzik, Louis Gerbarg</li> <li><a href="https://developer.apple.com/videos/play/wwdc2016/406/">https://developer.apple.com/videos/play/wwdc2016/406/</a></li> <li>"Launching an App is a complicated and subtle process and the ramifications on launch times of different App design patterns are often non-obvious. Come learn what happens in the time between when an App begins launching and when the main() function gets control and how that time relates to the code and structure of your App. Learn about the inner details of the dynamic loader, dyld, and best practices for structuring your code to perform at its best from the very start."</li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/nakedobjects/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/nakedobjects/index.html <p><a href="http://isis.apache.org/versions/2.0.0-M1/guides/ugfun/resources/core-concepts/Pawson-Naked-Objects-thesis.pdf">Richard Pawson's Thesis</a> was the backbone of this. Later, after creating the original Naked Objects framework (in Java Swing), he wrote <a href="http://www.nakedobjects.org/book/">a book</a> (now unfortunately woefully out of date in terms of implmentation) on it.</p> <p>See also: <a href="/presentation/nakedobjects">Naked Objects Framework</a> and <a href="/presentation/isis">Apache Isis</a>, the two direct successors to Pawson's work for the CLR and JVM, respectively.</p> https://tedneward.github.io/Research/reading/development/fuzzing/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/fuzzing/index.html <h1>General</h1> <ul> <li>Awesome Fuzzing <ul> <li><a href="https://github.com/secfigo/Awesome-Fuzzing">https://github.com/secfigo/Awesome-Fuzzing</a></li> </ul> </li> <li>Google Fuzzing Forum <ul> <li>Tutorials, examples, discussions, research proposals, and other resources related to fuzzing</li> <li><a href="https://github.com/google/fuzzing">https://github.com/google/fuzzing</a></li> </ul> </li> <li><a href="https://www.reddit.com/r/fuzzing/">https://www.reddit.com/r/fuzzing/</a></li> <li>The Fuzzing Project <ul> <li><a href="https://fuzzing-project.org/">https://fuzzing-project.org/</a></li> </ul> </li> </ul> <hr> <h1>Readings</h1> <h2>Readings: Books</h2> <ul> <li><a href="https://www.fuzzingbook.org/">Generating Software Tests: Breaking Software for Fun and Profit</a></li> </ul> <h2>Readings: General</h2> <ul> <li>Evaluating Fuzz Testing <ul> <li>ACM Conference on Computer and Communications Security (CCS) 2018</li> <li>George T. Klees, Andrew Ruef, Benjamin Cooper, Shiyi Wei, Michael Hicks</li> <li><a href="http://www.cs.umd.edu/~mwh/papers/klees2018fuzzeval.html">http://www.cs.umd.edu/~mwh/papers/klees2018fuzzeval.html</a></li> <li><a href="https://arxiv.org/abs/1808.09700">https://arxiv.org/abs/1808.09700</a></li> <li><a href="https://www.youtube.com/watch?v=ID8XtoMn43I">https://www.youtube.com/watch?v=ID8XtoMn43I</a></li> <li>Evaluating Empirical Evaluations (for Fuzz Testing) <ul> <li><a href="http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/">http://www.pl-enthusiast.net/2018/08/23/evaluating-empirical-evaluations-for-fuzz-testing/</a></li> </ul> </li> <li>How to Spot Good Fuzzing Research <ul> <li><a href="https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuzzing-research/">https://blog.trailofbits.com/2018/10/05/how-to-spot-good-fuzzing-research/</a></li> </ul> </li> <li>Fuzzing and how to evaluate it <ul> <li>ISSISP 2018; Michael Hicks</li> <li><a href="https://cs.anu.edu.au/cybersec/issisp2018/assets/slides/hicks-fuzz-testing-eval.pdf">https://cs.anu.edu.au/cybersec/issisp2018/assets/slides/hicks-fuzz-testing-eval.pdf</a></li> </ul> </li> </ul> </li> <li><a href="https://github.com/wcventure/FuzzingPaper">FuzzingPaper: Recent Papers Related To Fuzzing</a></li> <li>Fuzzing: a survey <ul> <li>Cybersecurity 2018</li> <li>Jun Li, Bodong Zhao and Chao Zhang</li> <li><a href="https://cybersecurity.springeropen.com/articles/10.1186/s42400-018-0002-y">https://cybersecurity.springeropen.com/articles/10.1186/s42400-018-0002-y</a></li> </ul> </li> <li>Fuzzing: Hack, Art, and Science <ul> <li>Communications of the ACM (CACM) 63(2) 2020</li> <li>Patrice Godefroid</li> <li><a href="https://doi.org/10.1145/3363824">https://doi.org/10.1145/3363824</a></li> <li><a href="https://cacm.acm.org/magazines/2020/2/242350-fuzzing">https://cacm.acm.org/magazines/2020/2/242350-fuzzing</a></li> <li><a href="https://patricegodefroid.github.io/public_psfiles/Fuzzing-101-CACM2020.pdf">https://patricegodefroid.github.io/public_psfiles/Fuzzing-101-CACM2020.pdf</a></li> </ul> </li> <li>The Art, Science, and Engineering of Fuzzing: A Survey <ul> <li>IEEE Transactions on Software Engineering (2019)</li> <li>Valentin J.M. Manes, HyungSeok Han, Choongwoo Han, Sang Kil Cha, Manuel Egele, Edward J. Schwartz, Maverick Woo</li> <li><a href="https://arxiv.org/abs/1812.00140">https://arxiv.org/abs/1812.00140</a></li> <li><a href="https://www.jiliac.com/publication/survey/">https://www.jiliac.com/publication/survey/</a></li> <li>Genealogy Database of Fuzzers <ul> <li><a href="https://github.com/SoftSec-KAIST/Fuzzing-Survey">https://github.com/SoftSec-KAIST/Fuzzing-Survey</a></li> <li><a href="https://fuzzing-survey.org/">https://fuzzing-survey.org/</a></li> </ul> </li> </ul> </li> <li>The Fuzzing Hype-Train: How Random Testing Triggers Thousands of Crashes <ul> <li>IEEE Security &amp; Privacy, 17(1) 2019</li> <li>Mathias Payer</li> <li><a href="https://ieeexplore.ieee.org/document/8674043">https://ieeexplore.ieee.org/document/8674043</a></li> </ul> </li> </ul> <h2>Readings: Practice</h2> <ul> <li>Billions and billions of constraints: Whitebox fuzz testing in production <ul> <li>International Conference on Software Engineering ICSE 2013</li> <li>E. Bounimova, P. Godefroid, D. Molnar</li> <li><a href="https://www.microsoft.com/en-us/research/publication/billions-and-billions-of-constraints-whitebox-fuzz-testing-in-production/">https://www.microsoft.com/en-us/research/publication/billions-and-billions-of-constraints-whitebox-fuzz-testing-in-production/</a></li> <li><a href="https://patricegodefroid.github.io/public_psfiles/icse2013.pdf">https://patricegodefroid.github.io/public_psfiles/icse2013.pdf</a></li> <li><a href="https://courses.cs.washington.edu/courses/cse484/14au/slides/whitebox-fuzzing.pdf">https://courses.cs.washington.edu/courses/cse484/14au/slides/whitebox-fuzzing.pdf</a></li> <li>Interview with Patrice Godefroid - <a href="https://www.coursera.org/learn/software-security/lecture/SItpe/interview-with-patrice-godefroid">https://www.coursera.org/learn/software-security/lecture/SItpe/interview-with-patrice-godefroid</a></li> </ul> </li> <li>Building an ARM-based Fuzzing Cluster <ul> <li><a href="https://baerli.github.io/arm-fuzzingcluster/">https://baerli.github.io/arm-fuzzingcluster/</a></li> </ul> </li> <li>Circumventing Fuzzing Roadblocks with Compiler Transformations <ul> <li><a href="https://lafintel.wordpress.com/2016/08/15/circumventing-fuzzing-roadblocks-with-compiler-transformations/">https://lafintel.wordpress.com/2016/08/15/circumventing-fuzzing-roadblocks-with-compiler-transformations/</a></li> <li>LAF LLVM Passes - <a href="https://gitlab.com/laf-intel/laf-llvm-pass">https://gitlab.com/laf-intel/laf-llvm-pass</a></li> </ul> </li> <li>Fuzzers love assertions <ul> <li>Jesse Ruderman</li> <li><a href="http://www.squarefree.com/2014/02/03/fuzzers-love-assertions/">http://www.squarefree.com/2014/02/03/fuzzers-love-assertions/</a></li> </ul> </li> <li>Fuzzing with AFL workshop <ul> <li>SteelCon 2017, BSides London and Bristol 2019</li> <li>Michael Macnair</li> <li>Exercises to learn how to fuzz with American Fuzzy Lop</li> <li><a href="https://github.com/mykter/afl-training">https://github.com/mykter/afl-training</a></li> </ul> </li> <li>Fuzzing workflows: a fuzz job from start to finish <ul> <li><a href="https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-fuzz-job-from-start-to-finish/">https://foxglovesecurity.com/2016/03/15/fuzzing-workflows-a-fuzz-job-from-start-to-finish/</a></li> </ul> </li> <li>John Regehr <ul> <li>Who Fuzzes the Fuzzer? - <a href="https://blog.regehr.org/archives/501">https://blog.regehr.org/archives/501</a></li> <li>Better Random Testing by Leaving Features Out - <a href="https://blog.regehr.org/archives/591">https://blog.regehr.org/archives/591</a></li> <li>Swarm Testing <ul> <li>ISSTA 2012; Alex Groce, Chaoqiang Zhang, Eric Eide, Yang Chen, John Regehr</li> <li><a href="http://www.cs.utah.edu/~regehr/papers/swarm12.pdf">http://www.cs.utah.edu/~regehr/papers/swarm12.pdf</a></li> <li>Draft Paper about Better Fuzzing - <a href="https://blog.regehr.org/archives/595">https://blog.regehr.org/archives/595</a></li> </ul> </li> <li>Tricking a Whitebox Testcase Generator - <a href="https://blog.regehr.org/archives/672">https://blog.regehr.org/archives/672</a></li> <li>How to Fuzz an ADT Implementation - <a href="https://blog.regehr.org/archives/896">https://blog.regehr.org/archives/896</a></li> <li>Fuzzers Need Taming - <a href="https://blog.regehr.org/archives/925">https://blog.regehr.org/archives/925</a></li> <li>Levels of Fuzzing - <a href="https://blog.regehr.org/archives/1039">https://blog.regehr.org/archives/1039</a></li> <li>A New Compiler Fuzzing Paper - <a href="https://blog.regehr.org/archives/1061">https://blog.regehr.org/archives/1061</a></li> <li>What afl-fuzz Is Bad At - <a href="https://blog.regehr.org/archives/1238">https://blog.regehr.org/archives/1238</a></li> <li>API Fuzzing vs. File Fuzzing: A Cautionary Tale - <a href="https://blog.regehr.org/archives/1269">https://blog.regehr.org/archives/1269</a></li> <li>Reducers are Fuzzers - <a href="https://blog.regehr.org/archives/1284">https://blog.regehr.org/archives/1284</a></li> <li>Write Fuzzable Code - <a href="https://blog.regehr.org/archives/1687">https://blog.regehr.org/archives/1687</a></li> <li>Helping Generative Fuzzers Avoid Looking Only Where the Light is Good, Part 1 - <a href="https://blog.regehr.org/archives/1700">https://blog.regehr.org/archives/1700</a></li> </ul> </li> <li>One Weird Trick for Finding More Crashes: crasher recycling <ul> <li><a href="https://insights.sei.cmu.edu/cert/2013/09/one-weird-trick-for-finding-more-crashes-1.html">https://insights.sei.cmu.edu/cert/2013/09/one-weird-trick-for-finding-more-crashes-1.html</a></li> </ul> </li> <li>Statistical Evaluation of a Fuzzing Dictionary <ul> <li><a href="https://bshastry.github.io/2018/10/01/Evaluating-Dictionary-For-Fuzzing.html">https://bshastry.github.io/2018/10/01/Evaluating-Dictionary-For-Fuzzing.html</a></li> <li>Static Program Analysis as a Fuzzing Aid <ul> <li><a href="https://link.springer.com/chapter/10.1007/978-3-319-66332-6_2">https://link.springer.com/chapter/10.1007/978-3-319-66332-6_2</a></li> </ul> </li> <li>Orthrus: A tool to manage, conduct, and assess dictionary-based fuzz testing <ul> <li><a href="https://github.com/test-pipeline/Orthrus">https://github.com/test-pipeline/Orthrus</a></li> </ul> </li> </ul> </li> <li>The Art of Fuzzing – Slides and Demos <ul> <li>2017; René Freingruber</li> <li>AFL &amp; WinAFL, Taint Analysis, Reversing Tricks for Fuzzing, in-memory fuzzing, DynamoRio</li> <li><a href="https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-slides-and-demos/">https://sec-consult.com/en/blog/2017/11/the-art-of-fuzzing-slides-and-demos/</a></li> <li><a href="https://sec-consult.com/wp-content/uploads/files/vulnlab/the_art_of_fuzzing_slides.pdf">https://sec-consult.com/wp-content/uploads/files/vulnlab/the_art_of_fuzzing_slides.pdf</a></li> </ul> </li> <li>What You Corrupt Is Not What You Crash: Challenges in Fuzzing Embedded Devices <ul> <li>NDSS 2018</li> <li>M. Muench, J. Stijohann, F. Kargl, A. Francillon, Davide Balzarotti</li> <li><a href="http://s3.eurecom.fr/docs/ndss18_muench.pdf">http://s3.eurecom.fr/docs/ndss18_muench.pdf</a></li> <li><a href="https://github.com/avatartwo/ndss18_wycinwyc">https://github.com/avatartwo/ndss18_wycinwyc</a></li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>Angora: a mutation-based fuzzer <ul> <li>increase branch coverage by solving path constraints without symbolic execution</li> <li><a href="https://github.com/AngoraFuzzer/Angora">https://github.com/AngoraFuzzer/Angora</a></li> <li>Angora: Efficient Fuzzing by Principled Search <ul> <li>Security and Privacy (S&amp;P) 2018</li> <li>Peng Chen, Hao Chen</li> <li><a href="https://arxiv.org/abs/1803.01307">https://arxiv.org/abs/1803.01307</a></li> </ul> </li> </ul> </li> <li>ClusterFuzz: a scalable fuzzing infrastructure which finds security and stability issues in software <ul> <li><a href="https://github.com/google/clusterfuzz">https://github.com/google/clusterfuzz</a></li> <li><a href="https://opensource.googleblog.com/2019/02/open-sourcing-clusterfuzz.html">https://opensource.googleblog.com/2019/02/open-sourcing-clusterfuzz.html</a></li> </ul> </li> <li>DeepState: A unit test-like interface for fuzzing and symbolic execution <ul> <li>DeepState is a framework that provides C and C++ developers with a common interface to various symbolic execution and fuzzing engines. Users can write one test harness using a Google Test-like API, then execute it using multiple backends without having to learn the complexities of the underlying engines. It supports writing unit tests and API sequence tests, as well as automatic test generation.</li> <li><a href="https://github.com/trailofbits/deepstate">https://github.com/trailofbits/deepstate</a></li> <li>DeepState: Symbolic Unit Testing for C and C++ <ul> <li>NDSS 18 Workshop on Binary Analysis Research</li> <li>Peter Goodman and Alex Groce</li> <li><a href="https://www.cefns.nau.edu/~adg326/bar18.pdf">https://www.cefns.nau.edu/~adg326/bar18.pdf</a></li> </ul> </li> <li>Fuzzing an API with DeepState <ul> <li>Part 1: <a href="https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/">https://blog.trailofbits.com/2019/01/22/fuzzing-an-api-with-deepstate-part-1/</a></li> <li>Part 2: <a href="https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/">https://blog.trailofbits.com/2019/01/23/fuzzing-an-api-with-deepstate-part-2/</a></li> </ul> </li> </ul> </li> <li>FuzzFactory: Domain-Specific Fuzzing with Waypoints <ul> <li><a href="https://github.com/rohanpadhye/FuzzFactory">https://github.com/rohanpadhye/FuzzFactory</a></li> <li>SPLASH 2019 OOPSLA <ul> <li>Rohan Padhye, Caroline Lemieux, Koushik Sen, Laurent Simon, Hayawardh Vijayakumar</li> <li><a href="https://2019.splashcon.org/details/splash-2019-oopsla/57/FuzzFactory-Domain-Specific-Fuzzing-with-Waypoints">https://2019.splashcon.org/details/splash-2019-oopsla/57/FuzzFactory-Domain-Specific-Fuzzing-with-Waypoints</a></li> <li><a href="https://people.eecs.berkeley.edu/~ksen/papers/fuzzfactory.pdf">https://people.eecs.berkeley.edu/~ksen/papers/fuzzfactory.pdf</a></li> <li><a href="https://people.eecs.berkeley.edu/~rohanpadhye/files/fuzzfactory-oopsla19.pdf">https://people.eecs.berkeley.edu/~rohanpadhye/files/fuzzfactory-oopsla19.pdf</a></li> </ul> </li> </ul> </li> <li>Grammarinator: ANTLRv4 grammar-based test generator <ul> <li><a href="https://github.com/renatahodovan/grammarinator">https://github.com/renatahodovan/grammarinator</a></li> </ul> </li> <li>Honggfuzz <ul> <li>Security oriented fuzzer with powerful analysis options. Supports evolutionary, feedback-driven fuzzing based on code coverage (software- and hardware-based)</li> <li><a href="https://honggfuzz.dev/">https://honggfuzz.dev/</a></li> <li><a href="https://github.com/google/honggfuzz">https://github.com/google/honggfuzz</a></li> <li>Feedback-driven fuzzing - <a href="https://github.com/google/honggfuzz/blob/master/docs/FeedbackDrivenFuzzing.md">https://github.com/google/honggfuzz/blob/master/docs/FeedbackDrivenFuzzing.md</a></li> <li>Fuzzing TCP servers - <a href="http://blog.swiecki.net/2018/01/fuzzing-tcp-servers.html">http://blog.swiecki.net/2018/01/fuzzing-tcp-servers.html</a></li> <li>Internals of Hongfuzz - Intel PT - <a href="https://tunnelshade.in/blog/2018/09/hongfuzz-intel-pt-instrumentation/">https://tunnelshade.in/blog/2018/09/hongfuzz-intel-pt-instrumentation/</a></li> </ul> </li> <li>JFS (JIT Fuzzing Solver) <ul> <li>Constraint solver based on coverage-guided fuzzing</li> <li><a href="https://github.com/delcypher/jfs">https://github.com/delcypher/jfs</a></li> <li>Just Fuzz It: Solving Floating-Point Constraints using Coverage-Guided Fuzzing <ul> <li>ESEC/FSE 2019</li> <li>Daniel Liew, Cristian Cadar, Alastair F. Donaldson, and J. Ryan Stinnett.</li> <li><a href="https://srg.doc.ic.ac.uk/files/papers/jfs-esecfse-19.pdf">https://srg.doc.ic.ac.uk/files/papers/jfs-esecfse-19.pdf</a></li> </ul> </li> </ul> </li> <li>MoonLight: Fuzzing Corpus Design and Construction <ul> <li><a href="https://gitlab.anu.edu.au/lunar/moonlight">https://gitlab.anu.edu.au/lunar/moonlight</a></li> </ul> </li> <li>Nautilus: Fishing for Deep Bugs with Grammars <ul> <li><a href="https://github.com/RUB-SysSec/nautilus">https://github.com/RUB-SysSec/nautilus</a></li> <li>Network and Distributed System Security Symposium (NDSS) 2019 <ul> <li>Cornelius Aschermann, Tommaso Frassetto, Thorsten Holz, Patrick Jauernig, Ahmad-Reza Sadeghi, Daniel Teuchert</li> <li><a href="https://www.syssec.ruhr-uni-bochum.de/research/publications/nautilus/">https://www.syssec.ruhr-uni-bochum.de/research/publications/nautilus/</a></li> </ul> </li> </ul> </li> <li>Orthrus <ul> <li><a href="https://github.com/test-pipeline/orthrus">https://github.com/test-pipeline/orthrus</a></li> <li>Orthrus is a tool for managing, conducting, and assessing dictionary-based security (fuzz) testing for autotools projects. At the moment, it supports Clang/LLVM instrumentation and the AFL ecosystem (afl-fuzz, afl-utils, afl-cov). The ultimate aim is for Orthrus to be a generic wrapper around state-of-the-art fuzz and instrumentation tools on the one hand, and disparate build systems on the other.</li> <li>Static Program Analysis as a Fuzzing Aid <ul> <li>RAID 2017</li> <li>Bhargava Shastry, Markus Leutner, Tobias Fiebig, Kashyap Thimmaraju, Fabian Yamaguchi, Konrad Rieck, Stefan Schmid, Jean-Pierre Seifert, Anja Feldmann</li> <li><a href="https://aperture-labs.org/pdf/raid2017.pdf">https://aperture-labs.org/pdf/raid2017.pdf</a></li> <li><a href="http://users.sec.t-labs.tu-berlin.de/~bshastry/raid17_slidedeck.pdf">http://users.sec.t-labs.tu-berlin.de/~bshastry/raid17_slidedeck.pdf</a></li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-319-66332-6_2">https://link.springer.com/chapter/10.1007/978-3-319-66332-6_2</a></li> </ul> </li> <li>Static Exploration of Taint-Style Vulnerabilities Found by Fuzzing <ul> <li>USENIX WOOT 2017</li> <li>Bhargava Shastry, Federico Maggi, Fabian Yamaguchi, Konrad Rieck, Jean-Pierre Seifert</li> <li><a href="https://arxiv.org/abs/1706.00206">https://arxiv.org/abs/1706.00206</a></li> </ul> </li> </ul> </li> <li>QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing <ul> <li>USENIX Security 2018</li> <li>Insu Yun, Sangho Lee, Meng Xu, Yeongjin Jang, Taesoo Kim</li> <li><a href="https://github.com/sslab-gatech/qsym/">https://github.com/sslab-gatech/qsym/</a></li> <li><a href="https://www.usenix.org/conference/usenixsecurity18/presentation/yun">https://www.usenix.org/conference/usenixsecurity18/presentation/yun</a></li> </ul> </li> <li>Radamsa <ul> <li>a general-purpose fuzzer</li> <li><a href="https://gitlab.com/akihe/radamsa">https://gitlab.com/akihe/radamsa</a></li> </ul> </li> <li>RamFuzz: Combining Unit Tests, Fuzzing, and AI <ul> <li>A fuzzer for individual method parameters</li> <li>RamFuzz is a fuzzer for individual method parameters in unit tests. A unit test can use RamFuzz to generate random parameter values for methods under test. The values are logged, and the log can be replayed to repeat the exact same test scenario. But RamFuzz also allows mutation of the replay, where some parts of the log are replayed while others are replaced by newly generated values. The new run is also logged, yielding a mutated test scenario and allowing the classic fuzzing evolution process of progressively mutating the input until a bug is triggered.</li> <li><a href="https://github.com/dekimir/RamFuzz">https://github.com/dekimir/RamFuzz</a></li> <li>RamFuzz: A Framework for C++ Test Generation via Deep Learning - <a href="https://github.com/dekimir/RamFuzz/blob/master/sci/ramfuzz.md">https://github.com/dekimir/RamFuzz/blob/master/sci/ramfuzz.md</a></li> </ul> </li> <li>RapidFuzz <ul> <li>An experiment of hacking around RapidCheck to combine RapidCheck's property-based testing with libFuzzer.</li> <li>It was very much influenced by Dan Luu's post which suggested exactly this combination: <a href="http://danluu.com/testing/">http://danluu.com/testing/</a></li> <li><a href="https://github.com/unapiedra/rapidfuzz">https://github.com/unapiedra/rapidfuzz</a></li> </ul> </li> <li>TriforceAFL: AFL/QEMU fuzzing with full-system emulation. <ul> <li><a href="https://github.com/nccgroup/triforceafl">https://github.com/nccgroup/triforceafl</a></li> <li><a href="https://github.com/timnewsham/TriforceAFL">https://github.com/timnewsham/TriforceAFL</a></li> <li><a href="https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2016/june/project-triforce-run-afl-on-everything/">https://www.nccgroup.trust/uk/about-us/newsroom-and-events/blogs/2016/june/project-triforce-run-afl-on-everything/</a></li> </ul> </li> <li>zzuf: a transparent application input fuzzer <ul> <li><a href="https://github.com/samhocevar/zzuf">https://github.com/samhocevar/zzuf</a></li> <li><a href="http://caca.zoy.org/wiki/zzuf">http://caca.zoy.org/wiki/zzuf</a></li> </ul> </li> </ul> <h2>Software: AFL</h2> <ul> <li>american fuzzy lop (AFL) <ul> <li><a href="http://lcamtuf.coredump.cx/afl/">http://lcamtuf.coredump.cx/afl/</a></li> <li>Technical "whitepaper" for afl-fuzz <ul> <li><a href="http://lcamtuf.coredump.cx/afl/technical_details.txt">http://lcamtuf.coredump.cx/afl/technical_details.txt</a></li> </ul> </li> <li>Automatically inferring file syntax with afl-analyze <ul> <li><a href="https://lcamtuf.blogspot.com/2016/02/say-hello-to-afl-analyze.html">https://lcamtuf.blogspot.com/2016/02/say-hello-to-afl-analyze.html</a></li> </ul> </li> </ul> </li> <li>AFL++ <ul> <li>afl++ is afl 2.56b with community patches, AFLfast power schedules, qemu 3.1 upgrade + laf-intel support, MOpt mutators, InsTrim instrumentation, unicorn_mode, Redqueen and a lot more!</li> <li><a href="https://aflplus.plus/">https://aflplus.plus/</a></li> <li><a href="https://github.com/AFLplusplus/AFLplusplus">https://github.com/AFLplusplus/AFLplusplus</a></li> </ul> </li> <li>AFL-based-fuzzers-overview <ul> <li><a href="https://github.com/google/fuzzing/blob/master/docs/afl-based-fuzzers-overview.md">https://github.com/google/fuzzing/blob/master/docs/afl-based-fuzzers-overview.md</a></li> </ul> </li> <li>afl-utils <ul> <li>Utilities for automated crash sample processing/analysis, easy afl-fuzz job management and corpus optimization</li> <li><a href="https://gitlab.com/rc0r/afl-utils">https://gitlab.com/rc0r/afl-utils</a></li> </ul> </li> <li>afl-cov - AFL Fuzzing Code Coverage <ul> <li><a href="https://github.com/mrash/afl-cov">https://github.com/mrash/afl-cov</a></li> </ul> </li> <li>afl-cov: AFL fuzzing coverage CFG visualization <ul> <li><a href="https://github.com/axt/afl-cov">https://github.com/axt/afl-cov</a></li> </ul> </li> <li>afl-fuzz on different file systems <ul> <li><a href="https://barro.github.io/2018/06/afl-fuzz-on-different-file-systems/">https://barro.github.io/2018/06/afl-fuzz-on-different-file-systems/</a></li> </ul> </li> <li>AFL-Mutation-Chain <ul> <li>Recovers an approximation of the mutation chain that led to a particular seed in an AFL queue. Outputs the chain in either JSON or Graphviz DOT format.</li> <li><a href="https://github.com/adrianherrera/afl-mutation-chain">https://github.com/adrianherrera/afl-mutation-chain</a></li> </ul> </li> <li>AFLGo: Directed Greybox Fuzzing <ul> <li><a href="https://github.com/aflgo/aflgo">https://github.com/aflgo/aflgo</a></li> <li>Directed Greybox Fuzzing <ul> <li>Conference on Computer and Communications Security (CCS) 2017</li> <li>Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, Abhik Roychoudhury</li> <li><a href="http://www.comp.nus.edu.sg/~abhik/pdf/CCS17.pdf">http://www.comp.nus.edu.sg/~abhik/pdf/CCS17.pdf</a></li> </ul> </li> </ul> </li> <li>Awesome-AFL <ul> <li>A curated list of different AFL forks and AFL inspired fuzzers with detailed equivalent academic papers with AFL-fuzzing tutorials</li> <li><a href="https://github.com/Microsvuln/Awesome-AFL">https://github.com/Microsvuln/Awesome-AFL</a></li> </ul> </li> <li>Driller: augmenting AFL with symbolic execution! <ul> <li><a href="https://github.com/shellphish/driller">https://github.com/shellphish/driller</a></li> <li>Driller: Augmenting fuzzing through selective symbolic execution <ul> <li>NDSS 2016</li> <li>N. Stephens, J. Grosen, C. Salls, A. Dutcher, R. Wang, J. Corbetta, Y. Shoshitaishvili, C. Kruegel, G. Vigna</li> <li><a href="https://www.cs.ucsb.edu/~vigna/publications/2016_NDSS_Driller.pdf">https://www.cs.ucsb.edu/~vigna/publications/2016_NDSS_Driller.pdf</a></li> </ul> </li> </ul> </li> <li>Internals of AFL fuzzer - Compile Time Instrumentation <ul> <li><a href="https://tunnelshade.in/blog/2018/01/afl-internals-compile-time-instrumentation/">https://tunnelshade.in/blog/2018/01/afl-internals-compile-time-instrumentation/</a></li> </ul> </li> <li>WinAFL: A fork of AFL for fuzzing Windows binaries <ul> <li><a href="https://github.com/googleprojectzero/winafl">https://github.com/googleprojectzero/winafl</a></li> </ul> </li> <li>Zoo AFL: AFL utilities and modifications <ul> <li><a href="https://habr.com/en/company/dsec/blog/449134/">https://habr.com/en/company/dsec/blog/449134/</a></li> </ul> </li> </ul> <h2>Software: libFuzzer</h2> <ul> <li>libFuzzer – a library for coverage-guided fuzz testing. <ul> <li><a href="http://llvm.org/docs/LibFuzzer.html">http://llvm.org/docs/LibFuzzer.html</a></li> <li><a href="http://tutorial.libFuzzer.info">http://tutorial.libFuzzer.info</a></li> </ul> </li> <li>Deconstructing LibProtobuf/Mutator Fuzzing <ul> <li><a href="https://bshastry.github.io/2019/01/18/Deconstructing-LPM.html">https://bshastry.github.io/2019/01/18/Deconstructing-LPM.html</a></li> </ul> </li> <li>Efficient Fuzzing Guide <ul> <li><a href="https://chromium.googlesource.com/chromium/src/testing/libfuzzer/+/HEAD/efficient_fuzzing.md">https://chromium.googlesource.com/chromium/src/testing/libfuzzer/+/HEAD/efficient_fuzzing.md</a></li> </ul> </li> <li>Fuzzing arbitrary functions in ELF binaries using LIEF and LibFuzzer <ul> <li><a href="https://blahcat.github.io/2018/03/11/fuzzing-arbitrary-functions-in-elf-binaries/">https://blahcat.github.io/2018/03/11/fuzzing-arbitrary-functions-in-elf-binaries/</a></li> </ul> </li> <li>Introduction to using libFuzzer with llvm-toolset <ul> <li><a href="https://developers.redhat.com/blog/2019/03/05/introduction-to-using-libfuzzer-with-llvm-toolset/">https://developers.redhat.com/blog/2019/03/05/introduction-to-using-libfuzzer-with-llvm-toolset/</a></li> </ul> </li> <li>libfuzzer-workshop <ul> <li>Materials of "Modern fuzzing of C/C++ Projects" workshop</li> <li><a href="https://github.com/Dor1s/libfuzzer-workshop">https://github.com/Dor1s/libfuzzer-workshop</a></li> </ul> </li> <li>libfuzzerfication <ul> <li>LibFuzzerfication project uses libFuzzer for fuzzing popular applications and libraries.</li> <li><a href="https://github.com/ouspg/libfuzzerfication">https://github.com/ouspg/libfuzzerfication</a></li> </ul> </li> <li>libprotobuf-mutator <ul> <li>a library to randomly mutate protobuffers; can be used together with guided fuzzing engines, such as libFuzzer</li> <li><a href="https://github.com/google/libprotobuf-mutator">https://github.com/google/libprotobuf-mutator</a></li> </ul> </li> </ul> <h2>Software: Benchmarking</h2> <ul> <li>fuzzer-test-suite: Set of tests for fuzzing engines <ul> <li><a href="https://github.com/google/fuzzer-test-suite">https://github.com/google/fuzzer-test-suite</a></li> </ul> </li> <li>FuzzBench: Fuzzer Benchmarking as a Service <ul> <li><a href="https://github.com/google/FuzzBench">https://github.com/google/FuzzBench</a></li> <li><a href="https://security.googleblog.com/2020/03/fuzzbench-fuzzer-benchmarking-as-service.html">https://security.googleblog.com/2020/03/fuzzbench-fuzzer-benchmarking-as-service.html</a></li> <li><a href="https://opensource.googleblog.com/2020/03/fuzzbench-fuzzer-benchmarking-as-service.html">https://opensource.googleblog.com/2020/03/fuzzbench-fuzzer-benchmarking-as-service.html</a></li> </ul> </li> </ul> <h2>Software: OS: Linux</h2> <ul> <li>difuze: Fuzzer for Linux Kernel Drivers <ul> <li><a href="https://github.com/ucsb-seclab/difuze">https://github.com/ucsb-seclab/difuze</a></li> <li>DIFUZE: Interface Aware Fuzzing for Kernel Drivers - CCS 2017</li> <li><a href="https://acmccs.github.io/papers/p2123-corinaA.pdf">https://acmccs.github.io/papers/p2123-corinaA.pdf</a></li> <li><a href="http://seclist.us/difuze-fuzzer-for-linux-kernel-drivers.html">http://seclist.us/difuze-fuzzer-for-linux-kernel-drivers.html</a></li> </ul> </li> <li>syzkaller: an unsupervised, coverage-guided kernel fuzzer <ul> <li><a href="https://github.com/google/syzkaller">https://github.com/google/syzkaller</a></li> <li>Coverage-guided kernel fuzzing with syzkaller - <a href="https://lwn.net/Articles/677764/">https://lwn.net/Articles/677764/</a></li> </ul> </li> </ul> <h2>Software: OS: Windows</h2> <ul> <li>Sienna Locomotive: A user-friendly fuzzing and crash triage tool for Windows <ul> <li><a href="https://github.com/trailofbits/sienna-locomotive">https://github.com/trailofbits/sienna-locomotive</a></li> <li>User-Friendly Fuzzing with Sienna Locomotive <ul> <li><a href="https://blog.trailofbits.com/2019/04/08/user-friendly-fuzzing-with-sienna-locomotive/">https://blog.trailofbits.com/2019/04/08/user-friendly-fuzzing-with-sienna-locomotive/</a></li> </ul> </li> </ul> </li> </ul> <h2>Software: Performance</h2> <p>Fuzzing applied to software performance.</p> <ul> <li>PerfFuzz: Automatically Generate Pathological Inputs for C/C++ programs <ul> <li>ISSTA 2018</li> <li>Caroline Lemieux, Rohan Padhye, Koushik Sen, and Dawn Song</li> <li><a href="https://github.com/carolemieux/perffuzz">https://github.com/carolemieux/perffuzz</a></li> <li><a href="https://people.eecs.berkeley.edu/~rohanpadhye/files/PerfFuzz-issta18.pdf">https://people.eecs.berkeley.edu/~rohanpadhye/files/PerfFuzz-issta18.pdf</a></li> </ul> </li> <li>perf fuzzer: Targeted fuzzing of the perf_event_open() system call <ul> <li>Technical Report, University of Maine, Tech. Rep., 2015</li> <li>V. M. Weaver and D. Jones</li> <li><a href="http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/2015_perf_fuzzer_tr.pdf">http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/2015_perf_fuzzer_tr.pdf</a></li> <li>perf_fuzzer perf_event syscall fuzzer <ul> <li><a href="http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/">http://web.eece.maine.edu/~vweaver/projects/perf_events/fuzzer/</a></li> </ul> </li> <li>perf_event Validation Tests <ul> <li><a href="http://web.eece.maine.edu/~vweaver/projects/perf_events/validation/">http://web.eece.maine.edu/~vweaver/projects/perf_events/validation/</a></li> <li><a href="https://github.com/deater/perf_event_tests">https://github.com/deater/perf_event_tests</a></li> </ul> </li> </ul> </li> <li>SlowFuzz: Automated Domain-Independent Detection of Algorithmic Complexity Vulnerabilities <ul> <li>A spin on libFuzzer so as to favor inputs incurring a slowdown. The key modifications consist of changing the fitness function, to favor inputs that excercise more basic block edges, as well as introducing probabilities in the selection of mutations to be performed, so as to preserve "locality" of the created inputs.</li> <li><a href="https://github.com/nettrino/slowfuzz">https://github.com/nettrino/slowfuzz</a></li> <li>ACM CCS 2017 <ul> <li>Theofilos Petsios, Jason Zhao, Angelos D. Keromytis, Suman Jana</li> <li><a href="https://arxiv.org/abs/1708.08437">https://arxiv.org/abs/1708.08437</a></li> <li><a href="https://www.youtube.com/watch?v=ZuWWs64UTts">https://www.youtube.com/watch?v=ZuWWs64UTts</a></li> </ul> </li> </ul> </li> </ul> <hr> <h1>Talks</h1> <h2>Talks: 2020</h2> <ul> <li>Lightning in a Bottle: 25 Years of Fuzzing <ul> <li>FuzzCon 2020; Richard Johnson</li> <li><a href="https://docs.google.com/presentation/d/1L3ZmbCxdkZZCCrZ1gBI5Kj32rAnNNho3rZVzQXjnCvs">https://docs.google.com/presentation/d/1L3ZmbCxdkZZCCrZ1gBI5Kj32rAnNNho3rZVzQXjnCvs</a></li> </ul> </li> </ul> <h2>Talks: 2019</h2> <ul> <li>C++ Sanitizers and Fuzzing for the Windows Platform Using New Compilers, Visual Studio, and Azure <ul> <li>CppCon 2019: Jim Radigan</li> <li><a href="https://www.youtube.com/watch?v=0EsqxGgYOQU">https://www.youtube.com/watch?v=0EsqxGgYOQU</a></li> <li><a href="https://github.com/CppCon/CppCon2019/tree/master/Presentations/address_sanitizers__cloud_at_microsoft">https://github.com/CppCon/CppCon2019/tree/master/Presentations/address_sanitizers__cloud_at_microsoft</a></li> </ul> </li> <li>Fuzzing for developers <ul> <li>StockholmCpp::0x15 - Pi Day 2019; Paul Dreik</li> <li><a href="https://www.youtube.com/watch?v=e_Oc9SkCo5s">https://www.youtube.com/watch?v=e_Oc9SkCo5s</a></li> </ul> </li> <li>Going Beyond Coverage-Guided Fuzzing with Structured Fuzzing <ul> <li>Black Hat 2019; Jonathan Metzman</li> <li><a href="https://www.youtube.com/watch?v=S8JvzWDnjc0">https://www.youtube.com/watch?v=S8JvzWDnjc0</a></li> <li><a href="https://www.blackhat.com/us-19/briefings/schedule/#going-beyond-coverage-guided-fuzzing-with-structured-fuzzing-16110">https://www.blackhat.com/us-19/briefings/schedule/#going-beyond-coverage-guided-fuzzing-with-structured-fuzzing-16110</a></li> </ul> </li> <li>Make your programs more reliable with Fuzzing <ul> <li>ACCU 2019; Marshall Clow</li> <li><a href="https://www.youtube.com/watch?v=x0FQkAPokfE">https://www.youtube.com/watch?v=x0FQkAPokfE</a></li> </ul> </li> <li>Modern Source Fuzzing <ul> <li>OffensiveCon19; Ned Williamson</li> <li><a href="https://www.youtube.com/watch?v=xzG0pLM4Q64">https://www.youtube.com/watch?v=xzG0pLM4Q64</a></li> </ul> </li> <li>Testing Legacy Code - Fuzzing for Better Input Data <ul> <li>Meeting C++ 2019; Tina Ulbrich, Niel Waldren</li> <li><a href="https://meetingcpp.com/mcpp/slides/2019/Testing%20Legacy%20Code%20-%20Fuzzing%20for%20Better%20Input%20Data.pdf">https://meetingcpp.com/mcpp/slides/2019/Testing%20Legacy%20Code%20-%20Fuzzing%20for%20Better%20Input%20Data.pdf</a></li> </ul> </li> <li>What the Fuzz <ul> <li>Black Hat Europe 2019; Cornelius Aschermann and Sergej Schumilo</li> <li><a href="https://www.youtube.com/watch?v=Wy7qY5ms3qY">https://www.youtube.com/watch?v=Wy7qY5ms3qY</a></li> <li><a href="https://www.blackhat.com/eu-19/briefings/schedule/#what-the-fuzz-18031">https://www.blackhat.com/eu-19/briefings/schedule/#what-the-fuzz-18031</a></li> </ul> </li> </ul> <h2>Talks: 2018</h2> <ul> <li>Adventures in Fuzzing <ul> <li>NYU Talk 2018; Brandon Falk</li> <li><a href="https://www.youtube.com/watch?v=SngK4W4tVc0">https://www.youtube.com/watch?v=SngK4W4tVc0</a></li> <li><a href="https://github.com/gamozolabs/adventures_in_fuzzing">https://github.com/gamozolabs/adventures_in_fuzzing</a></li> </ul> </li> <li>Finding security vulnerabilities with modern fuzzing techniques <ul> <li>RuhrSec 2018; Rene Freingruber</li> <li><a href="https://www.youtube.com/watch?v=KFmeHz_vxfo">https://www.youtube.com/watch?v=KFmeHz_vxfo</a></li> </ul> </li> <li>Fuzzing Corpus Optimization - Moonwalking with Moonbeams <ul> <li>CSides Canberra 2018; Shane Magrath</li> <li><a href="https://github.com/DSTCyber/presentations/tree/master/2018-fuzzing-corpus-distillation">https://github.com/DSTCyber/presentations/tree/master/2018-fuzzing-corpus-distillation</a></li> <li><a href="https://gitlab.anu.edu.au/lunar/moonlight">https://gitlab.anu.edu.au/lunar/moonlight</a></li> </ul> </li> <li>Fuzzing with AFL <ul> <li>NDC TechTown 2018; Erlend Oftedal</li> <li><a href="https://www.youtube.com/watch?v=DFQT1YxvpDo">https://www.youtube.com/watch?v=DFQT1YxvpDo</a></li> <li><a href="https://vimeo.com/292689978">https://vimeo.com/292689978</a></li> </ul> </li> <li>Making Your Library More Reliable with Fuzzing <ul> <li>C++Now 2018; Marshall Clow</li> <li><a href="https://www.youtube.com/watch?v=LlLJRHToyUk">https://www.youtube.com/watch?v=LlLJRHToyUk</a></li> <li><a href="https://github.com/boostcon/cppnow_presentations_2018/blob/master/05-10-2018_thursday/making_your_library_more_reliable_with_fuzzing__marshall_clow__cppnow_05182018.pdf">https://github.com/boostcon/cppnow_presentations_2018/blob/master/05-10-2018_thursday/making_your_library_more_reliable_with_fuzzing__marshall_clow__cppnow_05182018.pdf</a></li> </ul> </li> <li>Seems Exploitable: Exposing Hidden Exploitable Behaviors Using Extended Differential Fuzzing <ul> <li>HITB2018AMS; Fernando Arnaboldi</li> <li><a href="https://conference.hitb.org/hitbsecconf2018ams/sessions/seems-exploitable-exposing-hidden-exploitable-behaviors-using-extended-differential-fuzzing/">https://conference.hitb.org/hitbsecconf2018ams/sessions/seems-exploitable-exposing-hidden-exploitable-behaviors-using-extended-differential-fuzzing/</a></li> </ul> </li> <li>Structure aware fuzzing <ul> <li>Meeting C++ 2018; Réka Kovács</li> <li><a href="https://www.youtube.com/watch?v=wTWNmOSKfD4">https://www.youtube.com/watch?v=wTWNmOSKfD4</a></li> <li><a href="https://meetingcpp.com/mcpp/slides/2018/Structured%20fuzzing.pdf">https://meetingcpp.com/mcpp/slides/2018/Structured%20fuzzing.pdf</a></li> </ul> </li> <li>Want more stable kernel? Fuzz it! <ul> <li>DevConf.cz 2018; Hangbin Liu</li> <li><a href="https://www.youtube.com/watch?v=XlPQpVlC37A">https://www.youtube.com/watch?v=XlPQpVlC37A</a></li> </ul> </li> </ul> <h2>Talks: 2017</h2> <ul> <li>Between Testing and Formal Verification <ul> <li>SecAppDev, March 2017; Jan Tobias Mühlberg</li> <li><a href="https://handouts.secappdev.org/handouts/2017/Jan%20Tobias%20Muehlberg/201705-secappdev-muehlberg.pdf">https://handouts.secappdev.org/handouts/2017/Jan%20Tobias%20Muehlberg/201705-secappdev-muehlberg.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=tjkLPErCFW0">https://www.youtube.com/watch?v=tjkLPErCFW0</a></li> <li><a href="https://github.com/verifast/verifast">https://github.com/verifast/verifast</a></li> </ul> </li> <li>Finding Security Vulnerabilities by Fuzzing and Dynamic Code Analysis <ul> <li>Verification Futures 2017; Richard Storer</li> <li><a href="http://www.testandverification.com/wp-content/uploads/2017/Verification_Futures/Richard_Storer_MathEmbedded.pdf">http://www.testandverification.com/wp-content/uploads/2017/Verification_Futures/Richard_Storer_MathEmbedded.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=vCLLA7C5Afg">https://www.youtube.com/watch?v=vCLLA7C5Afg</a></li> </ul> </li> <li>Fuzz or lose: why and how to make fuzzing a standard practice for C++ <ul> <li>CppCon 2017; Kostya Serebryany</li> <li><a href="https://www.youtube.com/watch?v=k-Cv8Q3zWNQ">https://www.youtube.com/watch?v=k-Cv8Q3zWNQ</a></li> </ul> </li> <li>Fuzz Testing <ul> <li>C++ Weekly 85; Jason Turner</li> <li><a href="https://www.youtube.com/watch?v=gO0KBoqkOoU">https://www.youtube.com/watch?v=gO0KBoqkOoU</a></li> </ul> </li> <li>Fuzzing with AFL <ul> <li>Circle City Con 2017; Adam DC949</li> <li><a href="http://www.irongeek.com/i.php?page=videos/circlecitycon2017/106-fuzzing-with-afl-adam-dc949">http://www.irongeek.com/i.php?page=videos/circlecitycon2017/106-fuzzing-with-afl-adam-dc949</a></li> </ul> </li> <li>Modern Fuzzing of Media-Processing Projects <ul> <li>FOSDEM 2017; Max Moroz</li> <li><a href="https://archive.fosdem.org/2017/schedule/event/om_fuzzing/">https://archive.fosdem.org/2017/schedule/event/om_fuzzing/</a></li> <li><a href="https://www.youtube.com/watch?v=QQs82BdoA9c">https://www.youtube.com/watch?v=QQs82BdoA9c</a></li> </ul> </li> </ul> <h2>Talks: 2016</h2> <ul> <li>Effective File Format Fuzzing – Thoughts, Techniques and Results <ul> <li>Black Hat Europe 2016; Mateusz Jurczyk</li> <li><a href="https://www.youtube.com/watch?v=qTTwqFRD1H8">https://www.youtube.com/watch?v=qTTwqFRD1H8</a></li> <li><a href="https://www.blackhat.com/eu-16/briefings/schedule/#effective-file-format-fuzzing--thoughts-techniques-and-results-4879">https://www.blackhat.com/eu-16/briefings/schedule/#effective-file-format-fuzzing--thoughts-techniques-and-results-4879</a></li> <li><a href="https://j00ru.vexillium.org/talks/blackhat-eu-effective-file-format-fuzzing-thoughts-techniques-and-results/">https://j00ru.vexillium.org/talks/blackhat-eu-effective-file-format-fuzzing-thoughts-techniques-and-results/</a></li> <li><a href="https://www.blackhat.com/docs/eu-16/materials/eu-16-Jurczyk-Effective-File-Format-Fuzzing-Thoughts-Techniques-And-Results.pdf">https://www.blackhat.com/docs/eu-16/materials/eu-16-Jurczyk-Effective-File-Format-Fuzzing-Thoughts-Techniques-And-Results.pdf</a></li> </ul> </li> <li>Fuzz Smarter Not Harder: An afl fuzz Primer <ul> <li>BSides San Francisco 2016; Craig Young</li> <li><a href="https://www.youtube.com/watch?v=29RbO5bftwo">https://www.youtube.com/watch?v=29RbO5bftwo</a></li> </ul> </li> <li>The Smart Fuzzer Revolution <ul> <li>BSides Lisbon 2016; Dan Guido</li> <li><a href="https://www.youtube.com/watch?v=g1E2Ce5cBhI">https://www.youtube.com/watch?v=g1E2Ce5cBhI</a></li> </ul> </li> </ul> <h2>Talks: 2015</h2> <ul> <li>Automated Software Testing for the 21st Century <ul> <li>TCE 2015; Patrice Godefroid</li> <li><a href="https://patricegodefroid.github.io/public_psfiles/talk-tce2015.pdf">https://patricegodefroid.github.io/public_psfiles/talk-tce2015.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=HbIrWdjbX8Y">https://www.youtube.com/watch?v=HbIrWdjbX8Y</a></li> </ul> </li> </ul> <h2>Talks: 2007</h2> <ul> <li>Revolutionizing the Field of Grey-box Attack Surface Testing with Evolutionary Fuzzing <ul> <li>DEFCON 2007; Jared DeMott, Richard Enbody, William Punch</li> <li><a href="http://www.blackhat.com/presentations/bh-usa-07/DeMott_Enbody_and_Punch/Presentation/bh-usa-07-demott_enbody_and_punch.pdf">http://www.blackhat.com/presentations/bh-usa-07/DeMott_Enbody_and_Punch/Presentation/bh-usa-07-demott_enbody_and_punch.pdf</a></li> <li><a href="https://www.blackhat.com/presentations/bh-usa-07/DeMott_Enbody_and_Punch/Whitepaper/bh-usa-07-demott_enbody_and_punch-WP.pdf">https://www.blackhat.com/presentations/bh-usa-07/DeMott_Enbody_and_Punch/Whitepaper/bh-usa-07-demott_enbody_and_punch-WP.pdf</a></li> <li><a href="https://www.vdalabs.com/tools/EFS.pdf">https://www.vdalabs.com/tools/EFS.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=g7uFJnbvjNg">https://www.youtube.com/watch?v=g7uFJnbvjNg</a></li> <li><a href="https://www.youtube.com/watch?v=G7sX7RhJ_2w">https://www.youtube.com/watch?v=G7sX7RhJ_2w</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/history-pieces/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/history-pieces/index.html <p>Databases: A Year in Review <a href="https://www.cs.cmu.edu/~pavlo/blog/2025/01/2024-databases-retrospective.html">2024</a> <a href="https://www.cs.cmu.edu/~pavlo/blog/2024/01/2023-databases-retrospective.html">2023</a> <a href="https://www.cs.cmu.edu/~pavlo/blog/2022/12/2022-databases-retrospective.html">2022</a> <a href="https://www.cs.cmu.edu/~pavlo/blog/2021/12/2021-databases-retrospective.html">2021</a></p> <p>State of WebAssembly <a href="https://platform.uno/blog/state-of-webassembly-2024-2025/">2024 and 2025</a> <a href="https://platform.uno/blog/state-of-webassembly-2023-2024/">2023 and 2024</a> <a href="https://platform.uno/blog/the-state-of-webassembly-2022-and-2023/">2022 and 2023</a> <a href="https://platform.uno/blog/the-state-of-webassembly-2021-and-2022/">2021 and 2022</a></p> https://tedneward.github.io/Research/reading/development/low-code-no-code/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/low-code-no-code/index.html <h1>What is Low-Code?</h1> <p>Low-code development platforms focus on ease-of-use and a visual app building approach so business professionals can utilize them for creating their own solutions. This means that those closest to business problems can be empowered to quickly turn ideas into action, with point-and-click simplicity.</p> <h3>What are the benefits?</h3> <p>In today’s fast pace world where every business is a digital business, application development is becoming a core competency of every company. Businesses need to stay ahead of increased competition, keep pace with customer demands and find new ways to grow. Low-code is an approach to addressing these challenges.</p> <p><strong>Maximize the effectiveness</strong> of every employee by delivering real-time, actionable insights across systems in in a highly personalized manner</p> <p><strong>Improve speed and agility</strong> by making it easy for everyone to continuously innovate in an application ecosystem on a single platform</p> <p><strong>Reduce IT complexity &amp; costs</strong> by eliminating traditional IT development burdens, bolstering security and standardizing governance on one platform rather than across many applications</p> <h3>What is the difference between low-code and no-code?</h3> <p>Low-code is a well-defined category of application development platforms designed to make application developers faster and more productive. No-code is more like a feature of low-code where visual techniques are used to develop parts, or the whole, of an application.</p> <p>Gartner views “no-code” application platforms as part of the Low Code Application Platform (LCAP) market. Gartner calls “no-code” a marketing and positioning statement, implying that the platform requires text entry only for formulas or simple expressions, with all other aspects of development being enabled by visual modeling or configuration. Gartner’s LCAP market definition includes such no-code platforms.</p> <p>Whether you call it low-code or no-code, the most important factor is what skill set is needed to build applications with the platform. Forrester divides the market of low-code platforms into two parts: those designed for “business developers” and those designed for “professional IT developers.”</p> <h3>Why do businesses need to use low-code to stay ahead?</h3> <p><strong>Improved agility:</strong> Because they’re visual in nature, creating apps with low-code allows for a more agile, effective process.</p> <p><strong>Decreased costs:</strong> Building apps and programs in less time saves companies a lot of money. It reduces the need for more developers—saving money on labor and employment. And, because of its ability to make nearly every task more efficient, entire organizations become more productive—increased productivity = more money.</p> <p><strong>Better customer experience:</strong> Low-code innovation is fast and effective, allowing companies to keep pace with the changing landscape of customer wants and need. Happier customers means loyal customers, and loyal customers create profit for life.</p> <p><strong>Tech talent gap:</strong> As the tech talent gap widens, low-code allows businesses to begin closing the gap without spending crazy amounts of money.</p> <p><strong>Decreased development time:</strong> On average, users can minimize the development period of apps by 10 times (compared to traditional methods), bringing products to market faster.</p> <p><strong>Automation:</strong> Like everything else in business and life, automation keeps things simple and ensures most hiccups are avoided. Low-code has operational features that can accelerate the development of the products and empower business experts who know the silos of data and process start to build-up.</p> <h1>Manifesto</h1> <p>Technology, and the role it plays in the business landscape, is an ever-evolving juggernaut. Core systems and business challenges are progressively complex. The pace of that evolution increases exponentially — what is fast right now will be glacial in hindsight.</p> <p>To keep up, you must build and deploy innovative, timely solutions. We believe that building solutions with low-code makes that possible. We believe software should be built well. We believe it should deliver business value. We believe it should be delivered at speed.</p> <p>To make this happen at scale we adhere to these Application Development Pillars:</p> <h3>Focus on business impact</h3> <p>Create alignment, achieve clarity, succeed quickly.</p> <h3>No brain power goes to waste</h3> <p>Unleash all the makers from across the enterprise.</p> <h3>Do everything with an agile attitude</h3> <p>Empower small teams, build for the cloud, deploy swiftly and often.</p> <h3>Assemble from existing business capabilities</h3> <p>Utilize established assets, don’t default to building from scratch.</p> <h3>Connect everything and everybody</h3> <p>APIs, integrations, new ways to access data — be open and accessible.</p> <p>We know low-code is capable of building valuable solutions at speed, but as with any tool, it needs to be used correctly to get the best results.</p> <h1>9 Principles</h1> <p>We defined the following 9 Principles of Low-Code Application Developmentto empower makers from the enterprise to the dorm room to build software that makes a difference. The less we push ourselves into this, the greater value it will have for all our audiences — prospects, customers, the media, potential employees.</p> <h3>Model-Driven Development</h3> <p>Transform ideas into applications that deliver business value through abstraction, automation, and openness.</p> <h3>Collaboration</h3> <p>Leverage a shared visual language to support the interchange of knowledge and ideas between business domain experts and developers.</p> <h3>Agility</h3> <p>Manage the full enterprise application development lifecycle with agile workstreams to eliminate bottlenecks, support iterative delivery and achieve shortest time-to-value.</p> <h3>The Cloud</h3> <p>Cloud enables the ease and speed of application deployment that customers demand.</p> <h3>Openness</h3> <p>Anything can be integrated with an agnostic enterprise application development platform – this removes limitations on what can be built.</p> <h3>Multi-User Development</h3> <p>Multiple developers should be able to work on an application at the same time. The platform must support and synchronize their work streams.</p> <h3>Experimentation &amp; Innovation</h3> <p>Development tools need to be affordable and nimble so innovators everywhere can experiment, explore, and create.</p> <h3>Governance &amp; Control</h3> <p>Robust governance and control processes and protocols are essential.</p> <h3>Community</h3> <p>A platform without a community is no platform at all.</p> <p><em>(Source: <a href="https://www.mendix.com/low-code-guide/the-low-code-manifesto/">Medix</a>)</em> | <em>(Source: <a href="https://www.creatio.com/page/low-code">Creatio</a>)</em></p> <h1>Reasons to go low-code</h1> <ul> <li><strong>Legacy systems are outdated</strong>: These systems can no longer support the needs of businesses to quickly innovate and transform to stay competitive</li> <li><strong>Traditional platforms and point solutions have limits</strong>: These solutions are limiting businesses to the processes they support and don't reach the processes that make them unique</li> <li><strong>Shortage of software developers</strong>: There will never be enough developers to support the pace of innovation required to stay ahead</li> <li><strong>Maintain a competitive advantage</strong>: Stay ahead of increased competition by driving rapid innovation on what makes you competitively unique</li> <li><strong>Customer satisfaction</strong>: Keep pace with customer demands by continually perfecting the processes that deliver exceptional customer experiences</li> <li><strong>Innovation at the edge</strong>: Enable new ways to grow by enabling completely new operating models that serve customers in new and better ways</li> <li><strong>Win the talent war</strong>: Attract employees with a culture of continuous improvement and opportunities to learn and apply new technical skills</li> </ul> <p><em>(Source: <a href="https://www.quickbase.com/business-application-platform/what-is-low-code">Quickbase</a>)</em></p> <h1><a href="https://www.quickbase.com/blog/a-brief-history-of-low-code-development-platforms">A Brief History of Low-Code Development Platforms</a></h1> <p>1982: James Martin published <em>Application Development Without Programmers</em>. “The number of programmers available per computer is shrinking so fast that most computers in the future must be put to work at least in part without programmers." Led to 4GLs, CASE tools, and RAD tools.</p> <p><strong>Why These Failed:</strong></p> <ol> <li><strong>They promised more than they could deliver.</strong> IT managers felt burned by unrealistic expectations sold to them by consultants and vendors. While 4GL and visual programming technologies offered a glimpse of a better world for IT and the business, the tools themselves simply could not live up to the hype. Building applications that would scale was particularly difficult.</li> <li><strong>The tools did not support best practices.</strong> Version control, testing, deployment, documentation, and other development best practices did not exist for most tools, and had to be performed manually.</li> <li><strong>They amplified security risks.</strong> Empowering non-technical people to build these original stand-alone software apps (even with the assistance of new tools) exposed the organization to several risks — chief among them that most non-technical builders did not possess the skillset to create and deploy applications with appropriate security and governance.</li> <li><strong>The internet swallowed everything.</strong> By the mid 2000s, a significant portion of software development was already focused on web applications, as more business sought to enable better worker productivity by delivering business applications via the cloud rather than on traditional server environments. This offset some of the need for traditional IT solutions for everyday problems.</li> </ol> <p><strong>Why would these new platforms succeed where others have failed?</strong></p> <p>The answer is in the platform. Rather than offering an interface that simply obscures the actual code generating an application, the new generation of low-code platformsare self-contained (yet extensible) platforms that enable people to build within an environment that’s already hospitable to all the unseen components of the application. In fact, most modern low-code platforms are delivered via the web, meaning users don’t have to worry about any updates at all.</p> <p>The cloud platform approach also empowers these tools to provide far more security and reliability than ever — making it much easier for organizations to deploy with confidence that they have the right controls in place to meet their security and compliance standards. If the platform itself offers high-level security and compliance controls, the path to deploying platform applicationssecurely is much shorter.</p> <p>Finally, the user base for these platforms has matured significantly in the last decade with the world’s most successful companies utilizing low-code platforms to power their unique processes.This has given rise to best practices, a thriving ecosystem of partners and low-code builders, and a better understanding overall of the capabilities of each platform.</p> <h1><a href="https://www.quickbase.com/blog/the-future-of-low-code-platforms">The Future of Low-Code Platforms</a></h1> <p><strong>No-code beats low-code in the long run</strong><br> Recently Forrester named Quick Base a Leader in its New Wave: Low-code Development Platforms for Business Developers. Forrester also issued a separate report focused on Low-code Development Platforms for AD&amp;D Pros. We think Forrester’s decision to define two separate markets — one aimed at non-technical business developers (sometimes called “no-code platforms”) and one aimed at professional developers working in IT — calls attention to fundamental differences in the ways low-code and no-code platforms can be deployed.</p> <p>Whereas low-code platforms for professional developers are designed to help professional developers build solutions faster (and therefore serve more business requests), no-code platforms are designed to offload much or all of the responsibility of developing and delivering apps to the business developers themselves.</p> https://tedneward.github.io/Research/reading/development/nlp-for-semantic-search/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/nlp-for-semantic-search/index.html <p><a href="https://www.pinecone.io/learn/nlp/">Website</a></p> <p>Book in progress.</p> https://tedneward.github.io/Research/reading/development/fuzzy-logic/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/fuzzy-logic/index.html <p><a href="https://www.spiceworks.com/tech/devops/articles/fuzzy-logic/amp/">What is Fuzzy Logic? Definition, Working, Pros and Cons</a></p> https://tedneward.github.io/Research/reading/development/index/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/index/index.html <p><a href="https://cses.fi/book/book.pdf">Competitive Programmer's Handbook</a> (Not sure where else this belongs.)</p> <p><a href="https://github.com/ossu/computer-science">Computer Science Education</a></p> <p><a href="https://www.reforge.com/blog/technical-decision-making">Why Engineers Should Invest in Decision Making Skills Early</a>: "... to make more successful technical calls and advance careers, engineers actually need to develop better strategic decision-making skills — not just technical execution skills. In fact, an over-reliance on technical execution skills early in a career leads to untapped impact and stalled growth opportunities down the road."</p> <p><a href="https://terrycrowley.medium.com/education-of-a-programmer-aaecf2d35312">"Education of a Programmer"</a>: "When I left Microsoft in October 2016 after almost 21 years there and almost 35 years in the industry, I took some time to reflect on what I had learned over all those years. This is a lightly edited version of that post."</p> <p><a href="https://alexewerlof.medium.com/my-guiding-principles-after-20-years-of-programming-a087dc55596c">"My guiding principles after 20 years of programming"</a>:<br> 1. Don’t fight the tools: libraries, language, platform, etc. Use as much native constructs as possible. Don’t bend the technology, but don’t bend the problem either. Pick the right tool for the job or you’ll have to find the right job for the tool you got.<br> 2. You don’t write the code for the machines, you write it for your colleagues and your future self (unless it’s a throw away project or you’re writing assembly). Write it for the junior ones as a reference.<br> 3. Any significant and rewarding piece of software is the result of collaboration. Communicate effectively and collaborate openly. Trust others and earn their trust. Respect people more than code. Lead by example. Convert your followers to leaders.<br> 4. Divide and conquer. Write isolated modules with separate concerns which are loosely coupled. Test each part separately and together. Keep the tests close to reality but test the edge cases too.<br> 5. Deprecate yourself. Don’t be the go-to person for the code. Optimize it for people to find their way fixing bugs and adding features to the code. Free yourself to move on to the next project/company. Don’t own the code or you’ll never grow beyond that.<br> 6. Security comes in layers: each layer needs to be assessed individually but also in relation to the whole. Risk is a business decision and has direct relation to vulnerability and probability. Each product/organization has a different risk appetite (the risk they are willing to take for a bigger win). Often these 3 concerns fight with each other: UX, Security, Performance.<br> 7. Realize that every code has a life cycle and will die. Sometimes it dies in its infancy before seeing the light of production. Be OK with letting go. Know the difference between 4 categories of features and where to put your time and energy:<br> * Core: like an engine in a car. The product is meaningless without it.<br> * Necessary: like a car’s spare wheel. It’s rarely used but when needed, its function decides the success of the system.<br> * Added value: like a car’s cup-holder. It’s nice to have but the product is perfectly usable without it.<br> * Unique Selling Point: the main reason people should buy your product instead of your rivals. For example, your car is the best off-road vehicle.<br> 8. Don’t attach your identity to your code. Don’t attach anyone’s identity to their code. Realize that people are separate from the artifacts they produce. Don’t take code criticism personally but be very careful when criticizing others’ code.<br> 9. Tech debt is like fast food. Occasionally it’s acceptable but if you get used to it, it’ll kill the product faster than you think (and in a painful way).<br> 10. When making decisions about the solution all things equal, go for this priority: <strong>Security &gt; Reliability &gt; Usability (Accessibility &amp; UX) &gt; Maintainability &gt; Simplicity (Developer experience/DX) &gt; Brevity (code length) &gt; Finance &gt; Performance</strong> But don’t follow that blindly because it is dependent on the nature of the product. Like any career, the more experience you earn, the more you can find the right balance for each given situation. For example, when designing a game engine, performance has the highest priority, but when creating a banking app, security is the most important factor.<br> 11. Bugs’ genitals are called copy &amp; paste. That’s how they reproduce. Always read what you copy, always audit what you import. Bugs take shelter in complexity. “Magic” is fine in my dependency but not in my code.<br> 12. Don’t only write code for the happy scenario. Write good errors that answer why it happened, how it was detected and what can be done to resolve it. Validate all system input (including user input): fail early but recover from errors whenever possible. Assume the user hold a gun: put enough effort into your errors to convince them to shoot something other than your head!<br> 13. Don’t use dependencies unless the cost of importing, maintaining, dealing with their edge cases/bugs and refactoring when they don’t satisfy the needs is significantly less than the code that you own.<br> 14. Stay clear from hype-driven development. But learn all you can. Always have pet projects.<br> 15. Get out of your comfort zone. Learn every day. Teach what you learn. If you’re the master, you’re not learning. Expose yourself to other languages, technologies, culture and stay curious.<br> 16. Good code doesn’t need documentation, great code is well documented so that anyone who hasn’t been part of the evolution, trial &amp; error process and requirements that led to the current status can be productive with it. An undocumented feature is a non-existing feature. A non-existing feature shouldn’t have code.<br> 17. Avoid overriding, inheritance and implicit smartness as much as possible. Write pure functions. They are easier to test and reason about. Any function that’s not pure should be a class. Any code construct that has a different function, should have a different name.<br> 18. Never start coding (making a solution) unless you fully understand the problem. It’s very normal to spend more time listening and reading than typing code. Understand the domain before starting to code. A problem is like a maze. You need to progressively go through the code-test-improve cycle and explore the problem space till you reach the end.<br> 19. Don’t solve a problem that doesn’t exist. Don’t do speculative programming. Only make the code extensible if it is a validated assumption that it’ll be extended. Chances are by the time it gets extended, the problem definition looks different from when you wrote the code. Don’t overengineer: focus on solving the problem at hand and an effective solution implemented in an efficient manner.<br> 20. Software is more fun when it’s made together. Build a sustainable community. Listen. Inspire. Learn. Share.</p> <p>"Junior engineer: Take this tightly defined feature &amp; build it<br> Mid-level engineer: Take this vaguely defined feature &amp; build it<br> Senior engineer: Take this known problem &amp; figure out how to solve it<br> Staff engineer: Take this goal &amp; find the problems we should be solving." --<a href="https://twitter.com/ZainRzv/status/1502550200396750851">@ZainRzv</a></p> <hr> <p><a href="https://kellanem.com/notes/new-tech">"Questions for a new technology."</a>:</p> <ol> <li> <p>What problem are we trying to solve? (Tech should never be introduced as an end to itself)</p> </li> <li> <p>How could we solve the problem with our current tech stack? (If the answer is we can’t, then we probably haven’t thought about the problem deeply enough)</p> </li> <li> <p>Are we clear on what new costs we are taking on with the new technology? (monitoring, training, cognitive load, etc)</p> </li> <li> <p>What about our current stack makes solving this problem in a cost-effective manner (in terms of money, people or time) difficult?</p> </li> <li> <p>If this new tech is a replacement for something we currently do, are we committed to moving everything to this new technology in the future? Or are we proliferating multiple solutions to the same problem? (aka “Will this solution kill and eat the solution that it replaces?”)</p> </li> <li> <p>Who do we know and trust who uses this tech? Have we talked to them about it? What did they say about it? What don’t they like about it? (if they don’t hate it, they haven’t used it in depth yet)</p> </li> <li> <p>What’s a low risk way to get started?</p> </li> <li> <p>Have you gotten a mixed discipline group of senior folks together and thrashed out each of the above points? Where is that documented?</p> </li> </ol> https://tedneward.github.io/Research/reading/development/memory-management/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/memory-management/index.html <p>Primary reading: <a href="https://gchandbook.org/"><em>Garbage Collection Handbook</em></a>, <a href="https://www.cs.kent.ac.uk/people/staff/rej/gcbook/"><em>Garbage Collection</em></a> | <a href="https://www.iecc.com/gclist/GC-faq.html">GC FAQ</a>, <a href="https://www.iecc.com/gclist/GC-algorithms.html">GC Techniques and algorithms</a>, <a href="https://www.iecc.com/gclist/GC-lang.html">GC Language Interfaces</a>, and <a href="https://www.iecc.com/gclist/GC-harder.html">"more difficult topics"</a> | <a href="http://www.memorymanagement.org/">Ravenbrook Memory Management Reference</a></p> <h2>Concepts/terminology</h2> <h3>Memory</h3> <ul> <li><a href="https://people.freebsd.org/~lstewart/articles/cpumemory.pdf">What Every Programmer Should Know About Memory</a> (<a href="/reading/development/cpumemory.pdf">PDF</a>): "As CPU cores become both faster and more numerous, the limiting factor for most programs is now, and will be for some time, memory access. Hardware designers have come up with ever more sophisticated memory handling and acceleration techniques–such as CPU caches–but these cannot work optimally without some help from the programmer. Unfortunately, neither the structure nor the cost of using the memory subsystem of a computer or the caches on CPUs is well understood by most programmers. This paper explains the structure of memory subsystems in use on modern commodity hardware, illustrating why CPU caches were developed, how they work, and what programs should do to achieve optimal performance by utilizing them."</li> </ul> <h3>Memory Allocation schemes (high-level)</h3> <ul> <li> <p><strong>Static</strong>: The compiler, at the time of compilation (or the program's startup, either approach can work) has knowledge of the desired allocations, and creates space for them out of somewhere in the process' available space. These locations are fixed throughout the lifetime of the program, and cannot be expanded or shrunk--they are literally static throughout the entirety of the application's life. This is how C/C++ handle <code>static</code>-modified variable declarations, for example. Most C/C++ globals fall into this category.</p> </li> <li> <p><strong>Stack</strong>: The stack is a region of RAM that gets created on every thread that your application is running on. It works in a LIFO (Last In, First Out) manner, meaning that as soon as you allocate – or “push” – memory on to the stack, that chunk of memory will be the first to be deallocated – or “popped.” Every time a function declares a new variable, it is “pushed” onto the stack, and after that variable falls out of scope (such as when the function closes), that variable will be deallocated from the stack automatically. Once a stack variable is freed, that region of memory becomes available for other stack variables.</p> <p>Due to the pushing and popping nature of the stack, memory management is very logical and is able to be handled completely by the CPU; this makes it very quick, especially since each byte in the stack tends to be reused very frequently which means it tends to be mapped to the processor’s cache. However, there are some cons to this form of strict management. The size of the stack is a fixed value, and allocating more onto the stack than it can hold will result in a stack overflow. The size of the stack is decided when the thread is created, and each variable has a maximum size that it can occupy based on its data type; this prevents certain variables such as integers from ever growing beyond a certain value, and forces more complex data types such as arrays to specify their size prior to runtime since the stack won’t let them be resized. Variables allocated on the stack also are always local in nature because they are always next in line to be popped (unless more variables are pushed prior to the popping of earlier variables).</p> </li> <li> <p><strong>Heap</strong>: The heap is a memory set aside for runtime/dynamic memory allocation. Allocation is handled at runtime, and therefore is up to the program to determine structure, means, etc. Once you allocate space, that space can be accessed at any point in time not only throughout just the thread, but throughout the application’s entire life. Deallocation must occur at some point, if the space is to be reused. Most operating systems will, once an application ends, reclaim that space. (Usually by tearing down the process entirely--we're long past the days where programs are obtaining exclusive access to physical addresses.)</p> <p>Interaction with the heap is usually through some form of reference; in languages that support direct access, these are ‘pointers,’ which are variables whose values are the address of another variable, such as a memory location. By creating a pointer, you ‘point’ at a memory location on the heap, which is what signifies the initial location of your variable and tells the program where to access the value. Due to the dynamic nature of the heap, it is completely unmanaged by the CPU aside from initial allocation and heap resizing; in non-garbage collected languages such as C and C++, this requires you as the developer to manage memory and to manually free memory locations when they are no longer needed. Failing to do so can create memory leaks and cause memory to become fragmented, which will cause reads from the heap to take longer and makes it difficult to continuously allocate more memory onto the heap.</p> </li> </ul> <h3>Heap management (aka garbage collection) strategies</h3> <p><a href="https://en.wikipedia.org/wiki/Tracing_garbage_collection">Wikipedia: Tracing garbage collection</a></p> <p>Automated memory management (GC) almost always refers to heap management; I've never heard it applied to any other (static or stack) form of memory allocation and reclamation. Several strategies are possible (and numerous hybrid approaches):</p> <ul> <li> <p><strong>Allocate without reclamation</strong>: Technically, this is a legal automatic memory management technique, though obviously it is optimized more for performance of allocation and reclamation (i.e., zero time spent reclamation) than longevity. The JVM has one such allocator, called the Epsilon GC, and is typically used for specific purposes (such as benchmarking or diagnostics).</p> </li> <li> <p><strong>Compile-time</strong>: Compile-time garbage collection is a form of static analysis allowing memory to be reused and reclaimed based on invariants known during compilation.</p> <p>This form of garbage collection has been studied in the Mercury programming language, and it saw greater usage with the introduction of LLVM's automatic reference counter (ARC) into Apple's ecosystem (iOS and OS X) in 2011.</p> </li> <li> <p><strong>Reference counting</strong>: Each allocated object has a count associated with it indicating how many references are currently pointing to it. When the reference count drops to zero, the object is eligible for reclamation. Reference counts can either be managed automatically (as in, the language/runtime manage it without programmer intervention required) or manually (programmers must ensure they call some kind of <code>release</code> method or function to indicate a finished state of use). Reference counting is highly vulnerable to mutually-referencing objects (cyclic object graphs) as a source of memory leaks.</p> <p>Historical notes:</p> <ul> <li> <p>Microsoft COM was the first "mainstream" platform to really embrace reference counting as a part of its formal semantics (the <code>IUnknown</code> interface had three methods, two of which--<code>AddRef</code> and <code>Release</code>--managed the reference count of the allocated component), with very mixed results. When COM went distributed (DCOM), reference counts jumped in severity, since now the garbage collection was distributed, and a missed or dropped call could keep an object alive forever.</p> </li> <li> <p>CORBA used reference counting as well, with much the same result.</p> </li> <li> <p>Java RMI chose to use a different form of distributed garbage collection, using a "heartbeat" to keep an object alive, and if no such heartbeat came through every so often, the object was assumed to be unreachable and therefore eligible for reclamation.</p> </li> <li> <p>Objective-C used reference counting, then later moved to "automatic reference counts" (ARC) which was then pushed under the covers by the Swift language.</p> </li> </ul> <p>Some additional ideas/concepts around ref-counting:</p> <ul> <li> <p><strong>Deferred reference counting</strong>: In deferred reference counts, local updates to the reference count (that is, those arising from assignments to and from local variables) are "deferred", as long as it is known that the reference count will remain positive when it should be positive. For example, the code to swap the (reference-counted) values contained in two variables a and b naively performs the following operations:</p> <pre><code> incrc(a); tmp := a; incrc(b); decrc(a); a := b; incrc(tmp); decrc(b); b := tmp decrc(tmp); (tmp goes out of scope) </code></pre> <p>After all the dust has cleared, after six reference count operations, the reference counts of the objects referred to by a and b (now by b and a) are unchanged. It is more efficient to recognize this and leave the reference counts unchanged.</p> </li> <li> <p><strong>Lazy Freeing</strong>: Note that naive reference counting can take arbitrarily long to return from a decrement, because freeing one object may recursively trigger the freeing of other objects whose counts drop to zero. Instead, an object whose reference count falls to zero may be placed on a queue/list/stack of objects that are no longer in use, but not yet processed onto the free list. Alternatively, when an object is allocated off the free list, the pointers to objects within it may be scanned. This is also known as "Weizenbauming", at least in some circles.</p> </li> <li> <p><strong>One-bit reference counting</strong>: William Stoye's trick in his combinator machine. Reference counts are frequently one, and noting this in the pointer, rather than the object, can save space, cut memory references, and allow the easy recycling of vast amounts of garbage. In a combinator machine, the actual primitives can be parameterized by the reference counts of their operands to recycle memory in place. Note, however, that a combinator machine generates vast amounts of garbage to begin with; this technique is unlikely to work as well in general use.</p> </li> </ul> </li> <li> <p><strong>Smart pointers</strong>: Most smart pointers are small "wrappers" around native pointers, carrying a reference count that tries to be as automatic as possible. Most popular in C++ implementations, though some languages (Rust) look to incorporate smart-pointer-type semantics directly into the language. These generally try to then utilize stack-based concepts to help automagically manage reference counts for allocated objects, but circular reference counts are still possible (and probable). In languages like Java or C#, one might imagine that all object references are smart, in that they are known to the platform. (Note: the CLR specifically has object reference pointers--what the CLI Spec calls "managed references"--as well as "native pointers", which are raw locations in memory, without the additional CLR-backed support, for interop with native code.)</p> </li> <li> <p><strong>Reachability</strong>: When moving beyond pointer/reference-based schemes, most automated memory systems need to know which objects are eligible for reclamation, and which aren't--in essence, which are still under the possibility of being used by code. (An automatic memory system should <strong><em>never</em></strong> deallocate an object that is in use.) This analysis is usually known as "reachability" analysis, that is, finding which objects are "reachable" by user code, and therefore unsafe to reclaim. Many systems have multiple possibilities of reachability:</p> <ul> <li><em>reachable</em> (in use)</li> <li><em>softly reachable</em>: Softly-reachable objects are eligible for reclamation, but are not reclaimed except and until memory pressure is too great (in other words, the runtime will fail if space is not found). Useful for caching behavior.</li> <li><em>weakly reachable</em>: An object is weakly reachable when the garbage collector finds no strong or soft references, but at least one path to the object with a weak reference. Weakly reachable objects are finalized some time after their weak references have been cleared. The only real difference between a soft reference and a weak reference is that the garbage collector uses algorithms to decide whether or not to reclaim a softly reachable object, but always reclaims a weakly reachable object. Weak references work well in applications that need to, for example, associate extra data with an unchangeable object, such as a thread the application did not create. Systems with weak references usually have some form of notification system when the referent is cleared, such as Java's <code>ReferenceQueue</code> mechanic. If you make a weak reference to the thread with a reference queue, your program can be notified when the thread is no longer strongly reachable. Upon receiving this notification, the program can perform any required cleanup of the associated data object. This makes them useful for object-pooling kinds of behavior.</li> <li><em>finalizer-reachable</em> or <em>"f-reachable"</em>: an object queued by the runtime for reclamation but still requires its finalizer to run; thus it is reachable only by the finalizer thread/mechanism within the runtime.</li> <li><em>phantom-reachable</em>: An object is phantomly reachable when the garbage collector finds no strong, soft, or weak references, but at least one path to the object with a phantom reference. Phantomly reachable objects are objects that have been finalized, but not reclaimed. These are the hardest to use, but offer a mechanism by which to do cleanup after a finalizer has been executed, essentially providing another way to do resource cleanup.</li> <li><em>unreachable</em> (not in use): There is no path from any executing code to this object, so there is no way this object could be used, making it safe to eliminate.</li> </ul> <p>Reading: <a href="http://pawlan.com/monica/articles/refobjs/">Monica Pawlan's original article</a> | <a href="https://www.kdgregory.com/index.php?page=java.refobj">Java reference types</a></p> <p>Note: This is using the JVM terminology; the CLR does not have soft or phantom references, and I've never heard of Python or Ruby having anything beyond weak references, either. Any GC language has reachable/unreachable, and any language which has some kind of concept of "destructor" or "finalizer" has to have f-reachable (or it goes to great lengths to invoke those destructors/finalizers the moment the last reference is dropped, which usually implies either some very sophisticated compiler analysis or a reference count buried under the surface, as in Swift/Obj-C's automatic reference counting (ARC).)</p> </li> <li> <p><a href="https://www.memorymanagement.org/glossary/m.html#term-mark-sweep"><strong>Mark-Sweep</strong></a>: A garbage collection pass occurs in two phases: a "mark" phase, to identify the difference between collectible objects and non-collectible objects, and a "sweep" phase, in which the collectible objects are finalized and collected. <strong>Pros</strong>: simple, <strong>Cons</strong>: tends to lead to major fragmentation of the heap; the mark phase may require all activity in the garbage collector to cease (so as to avoid problems of marking an object collectible if concurrently it's becoming reachable again--but I think this isn't possible for a lot of systems; see "concurrent mark-and-sweep" below); as the heap gets larger, taking the time to crawl the heap becomes exponentially greater.</p> <p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/4/4a/Animation_of_the_Naive_Mark_and_Sweep_Garbage_Collector_Algorithm.gif/330px-Animation_of_the_Naive_Mark_and_Sweep_Garbage_Collector_Algorithm.gif" alt=""></p> <p>Some variations on mark-sweep:</p> <ul> <li> <p><strong>Tricolor marking</strong>: Because of these performance problems, most modern tracing garbage collectors implement some variant of the tri-color marking abstraction, but simple collectors (such as the mark-and-sweep collector) often do not make this abstraction explicit. Tri-color marking works as described below.</p> <p>Three sets are created – white, black and gray:</p> <p><img src="https://upload.wikimedia.org/wikipedia/commons/thumb/1/1d/Animation_of_tri-color_garbage_collection.gif/330px-Animation_of_tri-color_garbage_collection.gif" alt=""></p> <ul> <li>The white set, or condemned set, is the set of objects that are candidates for having their memory recycled.</li> <li>The black set is the set of objects that can be shown to have no outgoing references to objects in the white set, and to be reachable from the roots. Objects in the black set are not candidates for collection.</li> <li>The gray set contains all objects reachable from the roots but yet to be scanned for references to "white" objects. Since they are known to be reachable from the roots, they cannot be garbage-collected and will end up in the black set after being scanned.</li> </ul> <p>In many algorithms, initially the black set starts as empty, the gray set is the set of objects which are directly referenced from roots and the white set includes all other objects. Every object in memory is at all times in exactly one of the three sets. The algorithm proceeds as following:</p> <ol> <li>Pick an object from the gray set and move it to the black set.</li> <li>Move each white object it references to the gray set. This ensures that neither this object nor any object it references can be garbage-collected.</li> <li>Repeat the last two steps until the gray set is empty.</li> </ol> <p>When the gray set is empty, the scan is complete; the black objects are reachable from the roots, while the white objects are not and can be garbage-collected.</p> <p>Since all objects not immediately reachable from the roots are added to the white set, and objects can only move from white to gray and from gray to black, the algorithm preserves an important invariant – no black objects reference white objects. This ensures that the white objects can be freed once the gray set is empty. This is called the tri-color invariant. Some variations on the algorithm do not preserve this invariant but use a modified form for which all the important properties hold.</p> <p>The tri-color method has an important advantage – it can be performed "on-the-fly", without halting the system for significant periods of time. This is accomplished by marking objects as they are allocated and during mutation, maintaining the various sets. By monitoring the size of the sets, the system can perform garbage collection periodically, rather than as needed. Also, the need to touch the entire working set on each cycle is avoided.</p> </li> <li> <p><strong>Mark-and-don't-sweep</strong>: This might also be regarded as "lazy sweep" or "incremental sweep". Whenever memory is needed, the allocator scans objects until it discovers one that is marked "unused" and is also large enough for the request. All objects scanned, whether large enough or not, are marked "used". When the scan pointer reaches the end of memory, the sense of the mark bits is reversed (that is, if the current assignment is 0==used and 1==unused, then the new assignment is 0==unused, 1==used). Temporarily, all objects on the heap are now marked "unused". Next, a mark phase is run, which tags all reachable objects as "used" so that they will not be reallocated later.</p> </li> <li> <p><strong>Snapshot mark-and-sweep</strong>: Snapshot mark-and-sweep uses the observation that the set of unreachable objects does not shrink. It is possible (using, for instance, copy-on-write page mapping) to quickly make a copy of the address space, process it concurrently to determine what is garbage, and send that information back to the running process. More garbage may have been generated in the interim, but that is ok, because it will be found in the next collection.</p> </li> <li> <p><strong>Concurrent mark-and-sweep</strong>:</p> <p>Articles:</p> <ul> <li><a href="http://doc.cat-v.org/inferno/concurrent_gc/">"Very Concurrent Mark and Sweep Garbage Collection without Fine-Grain Synchronization"</a> (<a href="http://doc.cat-v.org/inferno/concurrent_gc/concurrent_gc.pdf">pdf</a>): We describe a new incremental algorithm for the concurrent reclamation of a program’s allocated, yet unreachable, data. Our algorithm is a variant of mark-&amp;-sweep collection that—unlike prior designs—runs mutator, marker, and sweeper threads concurrently without explicit fine-grain synchronization on shared-memory multiprocessors. A global, but infrequent, synchronization coordinates the per-object coloring marks used by the three threads; ne-grain synchronization is achieved without locking via the basic memory consistency guarantees commonly provided by multiprocessor hardware. We have implemented two versions of this algorithm (called VCGC): in the Inferno operating system and in the SML/NJ ML compiler. Measurements, compared to a sequential generational collector, indicate that VCGC can substantially reduce worst-case pause latencies as well as reduce overall memory usage. We remark that the degrees of freedom on the rates of marking and sweeping enable exploration of a range of resource tradeoffs, but makes "optimal” tuning for even a small set of applications difficult.</li> </ul> </li> </ul> </li> <li> <p><a href="https://www.memorymanagement.org/glossary/m.html#term-mark-compact"><strong>Mark-Sweep-Compact</strong></a>: A mark-sweep algorithm that also takes the time after the sweep to rearrange the objects in memory so as to put available spaces next to one another, reducing fragmentation and making it easier to satisfy future requests. Takes longer, and requires objects in memory to be movable (either by adding a layer of indirection on access, or by allowing pointers to be "fixed up" later to point to the right location).</p> </li> <li> <p><a href="https://www.memorymanagement.org/glossary/c.html#term-copying-garbage-collection"><strong>Copying</strong></a>:</p> </li> <li> <p><strong>Generational</strong>: A form of copying collector. Based on the observation that most objects have short lifetimes, it is useful to restrict garbage collection to the most recently allocated objects. A generational collector maintains several <code>generations'' of objects. Newly created objects are all put in the</code>youngest'' generation, and when the space allocated for that generation is full, the collector will use the root set (and any pointers from older generations to the youngest generation -- see below) to reclaim dead objects from the youngest generation only, leaving the <code>older'' generations untouched. Objects that survive after several (perhaps just one) collection of the youngest generation are</code>promoted'' to the next older generation, and when that generation becomes full, it and all the generations younger than it are collected.</p> <p>The difficulty with generational collectors is the identification of pointers from older generations into younger ones. An observation is that such references are uncommon in most programs: new objects typically point to older objects; this is clearly true in pure functional languages where assignment is prohibited, but is common for many programs and languages. Nonetheless, such references do exist, and a collector must handle them. When a pointer to a younger generation object is written into an old object, the pointer (or just the old object) is recorded so it can be scanned when the younger generation is collected.</p> </li> <li> <p><strong>Conservative Collection</strong>: Conservative garbage collection makes use of the observation that if you are not relocating (copying) objects, then you need not be quite so certain about exactly what is a pointer. It suffices to scan the root set (and objects) for any pointer-like bit patterns, and treat those pointer-like bit patterns as actual pointers while marking. The result is that the "true" reachable objects are all found, along with a few others. This works surprisingly well in practice (if a few additional tricks are added) and often permits the use of garbage collection with oblivious source code and compilers.<br> Conservative collection is very important because it permits the use of garbage collection with programs that were not written with garbage collection in mind. That is, it simpifies the use of garbage collection with existing (non-garbage-collected) libraries of code.</p> </li> </ul> <p>Ironically, many of these strategies could be used in custom allocation/deallocation schemes in C/C++, but typically aren't due to the complexity of memory management; specifically, C++'s inability to allow for the "movement" of objects as part of the process. This inability to move objects around in memory (that is, specifically, to track where the object ends up and "backport" the new location to the already-existing pointers pointing to it) leads to excessive fragmentation over time, and that in turn can lead to inability to satisfy an allocation request--no one "hole" in the memory space is large enough to accommodate the request, despite there being enough <em>total</em> memory to do so.</p> <h3>Additional concepts</h3> <ul> <li><strong>Finalizers</strong>: Blocks of code to be run to assist with object reclamation efforts, for those situations in which just releasing the memory occupied is not sufficient to release all allocated resources (files, connections, locks, etc). These need to be run prior to the object's deallocation, since the variable state inside the object is often necessary as part of the resource-deallocation process, but this is usually running on a thread owned by the runtime, which means this is a scenario in which user code is being run on a runtime-owned thread, which raises all sorts of negative possibilities.</li> </ul> <h3>Glossary/Jargon</h3> <ul> <li><strong>root set</strong>: The data that is immediately available to a program, without following any pointers. Typically this would include local variables from the activation stack, values in machine registers, and global, static, or module variables.</li> <li><strong>reachable data</strong>: Data that is accessible by following pointers (references) from the root set. Reachability is a conservative approximation of liveness and is used by most garbage collectors.</li> <li><strong>live data</strong>: Data that is reachable, and that the program will actually make use of in the future. Garbage collectors typically cannot tell the difference between live and reachable data, but programmers and compilers sometimes can.</li> <li><strong>garbage</strong>: Data that is not reachable.</li> <li><strong>semantic garbage</strong>: Data that is reachable, but not live.</li> <li><strong>precise</strong>: A garbage collector that can determine unambiguously whether a given value is a pointer or not. This can be seen as a requirement on the run-time the collector operates within that it is possible to identify whether some datum is a pointer.</li> <li><strong>conservative</strong>: A garbage collector which can scan data without needing to determine with certainty whether an object is a pointer.</li> <li><strong>semi-conservative</strong>, <strong>mostly-precise</strong>: A collector which permits a mixture of precisely and conservatively identified values. For example, values in registers and on the stack might be treated conservatively, but objects on the heap would be fully described and precisely scanned.</li> <li><strong>incremental</strong>: An incremental collector interleaves collection activity with the actual work of the program. The interleaving is usually orderly; that is, the collector and mutator do not simultaneously access and modify data. If the collector and mutator are viewed as separate threads, then they are scheduled as coroutines. However, the sum of the incremental phases takes longer to complete than one batch garbage collection pass, so these garbage collectors may yield lower total throughput.</li> <li><strong>concurrent</strong>: In a concurrent collector, the collector and mutator may simultaneously access and modify data. The interleaving is disorderly, and preemption may occur at any time. However, the sum of the concurrent phases takes longer to complete than one batch garbage collection pass, so these garbage collectors may yield lower total throughput.</li> <li><strong>stop-the-world</strong>, <strong>stop-and-collect</strong>: Simple stop-the-world garbage collectors completely halt execution of the program to run a collection cycle, thus guaranteeing that new objects are not allocated and objects do not suddenly become unreachable while the collector is running. This has the disadvantage that the program can perform no useful work while a collection cycle is running (sometimes called the "embarrassing pause"). Stop-the-world garbage collection is therefore mainly suitable for non-interactive programs. Its advantage is that it is both simpler to implement and faster than incremental garbage collection.</li> <li><strong>real-time</strong>: A real-time garbage collector (yes, it is possible) guarantees that the garbage collection costs associated with any single operation are bounded by a small time constant. The cost of this guarantee is typically reduced throughput, higher memory overheads, and custom code generation. When hard real-time scheduling is not required, an incremental or concurrent collector is probably a better choice.</li> <li><strong>forwarding-pointer</strong>: In a collector which moves objects, a forwarding pointer is a reference installed by the garbage collector from an old location to the new one. Some systems (notably the Lisp machines) have hardware support for forwarding pointers, which allow both the old and new address to be used interchangably without explicit checks for forwarding pointers.</li> <li><strong>flip</strong>: In a copying collector, when an arena has had all active objects removed from it, and is eligible for refilling, the sense of the arenas is changed -- that is, an "old" arena is now a "new" arena, and some "new" arena becomes "old". This is referred to as a flip.</li> <li><strong>weak reference</strong>, <strong>weak pointer</strong>: A pointer to an object which does not prevent the object from being reclaimed. If the only pointers to an object are from weak references, the object may disappear, in which case the reference is replaced with some distinguished value, typically a language's equivalent of a NULL pointer. Weak references are often used for implementing caches.</li> <li><strong>write barrier</strong>, <strong>read barrier</strong>: A write barrier is a mechanism for executing some memory management code when a write to some object takes place (that object is then "behind the write barrier", or - informally - "write barrier-ed", or - sloppily - "write-protected"). It can take the form of inlined code (if memory management is integral to the compiler), or a memory-protection fault which is handled by the memory management code. There are also "read barriers", the nature of which is obvious.<br> The roles a write barrier can play in GC are a little trickier to explain to a novice, but I'll give it a stab. <ol> <li> <p>Consider a simple generational stop-and-collect collector, such as the one which SML/NJ used to have. "Generational" means that data is partitioned into old and new. This partition is useful to the GC for two reasons: (a) because data tends to die young, collecting just new data will probably free a lot of space, and (b) because pointers tend to point from new objects to old objects, and not vice versa, it is cheap to find all the pointers to new objects.</p> <p>Property (b) is only true if you can tell when a pointer to a new object has been written into an old object. Otherwise you have to scan all the old objects to find pointers to new objects, which loses one of the main advantages of generational GC. So you put the old data behind a write barrier, and record those writes. When you come to GC the new data, you know the only pointers from old to new are those which you have recorded.</p> </li> <li> <p>Consider a tracing GC (see note below) which is incremental or concurrent, i.e. the user's program (the 'mutator') can run before the GC is complete. Now there is an invariant: black objects do not point to white objects. If the mutator writes a white pointer into a black object, this invariant is broken and the GC can fail. There are two basic solutions: prevent the mutator from seeing white objects ("read barriers") or prevent the mutator from writing white pointers into black objects ("write barriers"). The write barrier solution puts the black objects behind a write barrier. When a white-on-black write takes place there are various fixes: incrementally grey the white object, regrey the black object, &amp;c.</p> <p>(note) For a tracing collector (marking or copying), one conceptually colours the data white (not yet seen by the collector), black (alive and scanned by the collector) and grey (alive but not yet scanned by the collector). The collector proceeds by scanning grey objects for pointers to white objects. The white objects found are turned grey, and the grey objects scanned are turned black. When there are no more grey objects, the collection is complete and all the white objects can be recycled.</p> </li> </ol> <p>The overhead of write barriers is more likely to be noticeable in an imperative-style program which frequently writes pointers into existing data structures than in a functional-style program which constructs data only once and never changes them.</p> </li> <li> <p><strong>remembered set</strong>: A list of all the interesting references between two sets of objects maintained separately, so you don't have to find them by scanning. In generational garbage collection, this technique is used for references from an older generation to a younger one. Typically, remembered sets are maintained by write barriers (q.v.).</p> </li> <li><strong>register set partitioning</strong>: Garbage-collected languages often partition the set of registers a priori into those always traced and updated (if necessary) by the collector and those ignored by it. The language always maintains the former in a format understood by the collector; and never uses the latter to hold pointers to collectable objects. More complicated schemes are also possible.</li> <li><strong>big bag of pages</strong>: Big Bag of Pages, or BIBOP, is the technique of using an object's address (page number) to encode information about the object. For example, all objects allocated on a particular page might be the same size, or the same type. This technique can be used to avoided allocating object header bits or header tags.</li> </ul> <h2>Papers/articles/notes</h2> <ul> <li><a href="https://www.memorymanagement.org/">Memory Management Reference (site)</a></li> <li><a href="https://www.cs.cornell.edu/courses/cs4120/2018sp/lectures/39gc/lec39-sp18.pdf">CS 4120/5120 Lecture notes</a> (Andrew Myers)</li> <li><a href="https://spin.atomicobject.com/2014/09/03/visualizing-garbage-collection-algorithms/">Visualizing garbage collection algorithms</a> (<a href="https://github.com/kenfox/gc-viz">Source</a>): Demonstrates NoGC, ref-counting, mark-sweep, mark-compact, and copying.</li> <li><a href="https://www.dmitrysoshnikov.education/p/essentials-of-garbage-collectors">Garbage Collection Algorithms</a></li> </ul> <h3>C/C++</h3> <ul> <li><a href="https://hboehm.info/gc/">Boehm-Weiser collector</a></li> </ul> <h3>.NET</h3> <ul> <li><a href="https://docs.microsoft.com/en-us/dotnet/standard/garbage-collection/weak-references">.NET Weak References</a></li> <li><a href="https://courses.cs.washington.edu/courses/csep521/07wi/prj/rick.pdf">Garbage Collection Algorithms</a>: covers mark-sweep, semi-space, variations, and then gets into open problems. General overview, pretty light, feels like somebody's book report on the subject.</li> </ul> <h3>Java</h3> <ul> <li><a href="https://plumbr.io/java-garbage-collection-handbook">Plumbr Handbook Java GC</a></li> <li><a href="https://www.oracle.com/java/technologies/whitepaper.html">Java HotSpot Architecture</a></li> <li><a href="https://www.overops.com/blog/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/">Garbage collectors: Serial vs parallel vs CMS vs G1</a>; targets Java8, so a little out of date</li> <li><a href="https://www.overops.com/blog/improve-your-application-performance-with-garbage-collection-optimization/">Improve App Performance with These Advanced GC Techniques</a>; a newer (2021) version of the previous article</li> </ul> <h3>Python</h3> <ul> <li><a href="https://stackify.com/python-garbage-collection/">Python GC</a></li> <li><a href="https://devguide.python.org/garbage_collector/">CPython GC</a></li> <li><a href="https://towardsdatascience.com/memory-management-and-garbage-collection-in-python-c1cb51d1612c">Memory Mgmt and GC in Python</a></li> <li><a href="https://www.python.org/dev/peps/pep-0205/">Python Weak References</a></li> </ul> https://tedneward.github.io/Research/reading/development/dynamic-analysis-sanitizers/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/dynamic-analysis-sanitizers/index.html <ul> <li> <p>Sanitizers: AddressSanitizer, ThreadSanitizer, MemorySanitizer</p> <ul> <li><a href="https://github.com/google/sanitizers">https://github.com/google/sanitizers</a></li> <li><a href="https://github.com/google/sanitizers/wiki">https://github.com/google/sanitizers/wiki</a></li> </ul> </li> <li> <p>Compilers</p> <ul> <li> <p>Clang</p> <ul> <li>AddressSanitizer - <a href="https://clang.llvm.org/docs/AddressSanitizer.html">https://clang.llvm.org/docs/AddressSanitizer.html</a></li> <li>DataFlowSanitizer - <a href="https://clang.llvm.org/docs/DataFlowSanitizer.html">https://clang.llvm.org/docs/DataFlowSanitizer.html</a></li> <li>Hardware-assisted AddressSanitizer Design Documentation - <a href="https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html">https://clang.llvm.org/docs/HardwareAssistedAddressSanitizerDesign.html</a></li> <li>LeakSanitizer - <a href="https://clang.llvm.org/docs/LeakSanitizer.html">https://clang.llvm.org/docs/LeakSanitizer.html</a></li> <li>MemorySanitizer - <a href="https://clang.llvm.org/docs/MemorySanitizer.html">https://clang.llvm.org/docs/MemorySanitizer.html</a></li> <li>SanitizerCoverage - <a href="https://clang.llvm.org/docs/SanitizerCoverage.html">https://clang.llvm.org/docs/SanitizerCoverage.html</a></li> <li>SanitizerStats - <a href="https://clang.llvm.org/docs/SanitizerStats.html">https://clang.llvm.org/docs/SanitizerStats.html</a></li> <li>Sanitizer special case list - <a href="https://clang.llvm.org/docs/SanitizerSpecialCaseList.html">https://clang.llvm.org/docs/SanitizerSpecialCaseList.html</a></li> <li>ThreadSanitizer - <a href="https://clang.llvm.org/docs/ThreadSanitizer.html">https://clang.llvm.org/docs/ThreadSanitizer.html</a></li> <li>UndefinedBehaviorSanitizer - <a href="https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html">https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html</a></li> </ul> </li> <li> <p>GCC</p> <ul> <li><a href="https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html">https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html</a></li> <li>Address and Thread Sanitizers in GCC <ul> <li><a href="https://developers.redhat.com/blog/2014/12/02/address-and-thread-sanitizers-gcc/">https://developers.redhat.com/blog/2014/12/02/address-and-thread-sanitizers-gcc/</a></li> </ul> </li> <li>Useful GCC address sanitizer checks not enabled by default <ul> <li><a href="https://kristerw.blogspot.com/2018/06/useful-gcc-address-sanitizer-checks-not.html">https://kristerw.blogspot.com/2018/06/useful-gcc-address-sanitizer-checks-not.html</a></li> </ul> </li> </ul> </li> </ul> </li> <li> <p>Adding Clang Sanitizers to a CMake Build</p> <ul> <li><a href="https://genbattle.bitbucket.io/blog/2018/01/05/Dev-Santa-Claus-Part-1/">https://genbattle.bitbucket.io/blog/2018/01/05/Dev-Santa-Claus-Part-1/</a> <ul> <li><a href="https://old.reddit.com/r/cpp/comments/7qzqlg/dev_santa_claus_pt1_adding_clang_sanitizers_to_a/">https://old.reddit.com/r/cpp/comments/7qzqlg/dev_santa_claus_pt1_adding_clang_sanitizers_to_a/</a></li> </ul> </li> </ul> </li> <li> <p>Creating an LLVM Sanitizer from Hopes and Dreams</p> <ul> <li><a href="https://blog.trailofbits.com/2019/06/25/creating-an-llvm-sanitizer-from-hopes-and-dreams/">https://blog.trailofbits.com/2019/06/25/creating-an-llvm-sanitizer-from-hopes-and-dreams/</a></li> <li>llvm-sanitizer-tutorial and documentation <ul> <li><a href="https://github.com/trailofbits/llvm-sanitizer-tutorial">https://github.com/trailofbits/llvm-sanitizer-tutorial</a></li> </ul> </li> </ul> </li> <li> <p>GWP-ASan: Sampling heap memory error detection in-the-wild</p> <ul> <li>2019; Vlad Tsyrklevich</li> <li><a href="https://sites.google.com/a/chromium.org/dev/Home/chromium-security/articles/gwp-asan">https://sites.google.com/a/chromium.org/dev/Home/chromium-security/articles/gwp-asan</a></li> </ul> </li> <li> <p>Kernel Concurrency Sanitizer (KCSAN)</p> <ul> <li><a href="https://github.com/google/ktsan/wiki/KCSAN">https://github.com/google/ktsan/wiki/KCSAN</a></li> <li>Concurrency bugs should fear the big bad data-race detector <ul> <li>part 1: <a href="https://lwn.net/Articles/816850/">https://lwn.net/Articles/816850/</a></li> <li>part 2: <a href="https://lwn.net/Articles/816854/">https://lwn.net/Articles/816854/</a></li> </ul> </li> </ul> </li> <li> <p>Kernel Thread Sanitizer (KTSAN)</p> <ul> <li><a href="https://github.com/google/ktsan">https://github.com/google/ktsan</a></li> <li><a href="https://github.com/google/ktsan/wiki">https://github.com/google/ktsan/wiki</a></li> </ul> </li> </ul> <h4>Research</h4> <ul> <li>A Preliminary Study on Open-Source Memory Vulnerability Detectors <ul> <li>International Conference on Software Analysis, Evolution, and Reengineering (SANER), ERA track, 2020</li> <li>Yu Nong and Haipeng Cai</li> <li><a href="http://chapering.github.io/pubs/saner20-a.pdf">http://chapering.github.io/pubs/saner20-a.pdf</a></li> <li>AddressSanitizer, CBMC, DrMemory, MemorySanitizer, Valgrind</li> <li>Research questions: <ul> <li>RQ1: How effective are these detectors in terms of precision, recall, and accuracy?</li> <li>RQ2: How efficient are the detectors in terms of their cost for detecting vulnerabilities?</li> <li>RQ3: How do these detectors compare in terms of their detection accuracy?</li> </ul> </li> </ul> </li> <li>AddressSanitizer: A Fast Address Sanity Checker <ul> <li>2012 USENIX Annual Technical Conference (ATC)</li> <li>Konstantin Serebryany, Derek Bruening, Alexander Potapenko, Dmitry Vyukov</li> <li><a href="https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany">https://www.usenix.org/conference/atc12/technical-sessions/presentation/serebryany</a></li> <li><a href="https://research.google/pubs/pub37752/">https://research.google/pubs/pub37752/</a></li> </ul> </li> <li>CastSan: Efficient Detection of Polymorphic C++ Object Type Confusions with LLVM <ul> <li>European Symposium on Research in Computer Security (ESORICS) 2018</li> <li>Muntean P., Wuerl S., Grossklags J., Eckert C.</li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-319-99073-6_1">https://link.springer.com/chapter/10.1007/978-3-319-99073-6_1</a></li> <li><a href="https://www.docdroid.net/INWYBF7/castsan-esorics18.pdf">https://www.docdroid.net/INWYBF7/castsan-esorics18.pdf</a></li> </ul> </li> <li>CUP: Comprehensive User-Space Protection for C/C++ <ul> <li>AsiaCCS 2018</li> <li>Nathan Burow, Derrick McKee, Scott A. Carr, Mathias Payer</li> <li><a href="https://hexhive.github.io/publications/files/18AsiaCCS.pdf">https://hexhive.github.io/publications/files/18AsiaCCS.pdf</a></li> <li><a href="https://github.com/HexHive/CUP">https://github.com/HexHive/CUP</a></li> </ul> </li> <li>DangSan: Scalable Use-after-free Detection <ul> <li>European Conference on Computer Systems (EuroSys) 2017</li> <li>Erik van der Kouwe, Vinod Nigade, Cristiano Giuffrida</li> <li><a href="http://www.cs.vu.nl/~giuffrida/papers/dangsan_eurosys17.pdf">http://www.cs.vu.nl/~giuffrida/papers/dangsan_eurosys17.pdf</a></li> <li><a href="https://github.com/vusec/dangsan">https://github.com/vusec/dangsan</a></li> </ul> </li> <li>EffectiveSan: Type and Memory Error Detection using Dynamically Typed C/C++ <ul> <li>PLDI 2018</li> <li>Gregory J. Duck, Roland H. C. Yap</li> <li><a href="https://arxiv.org/abs/1710.06125">https://arxiv.org/abs/1710.06125</a></li> <li><a href="https://github.com/GJDuck/EffectiveSan">https://github.com/GJDuck/EffectiveSan</a></li> <li><a href="https://pldi18.sigplan.org/event/pldi-2018-papers-effectivesan-type-and-memory-error-detection-using-dynamically-typed-c-c-">https://pldi18.sigplan.org/event/pldi-2018-papers-effectivesan-type-and-memory-error-detection-using-dynamically-typed-c-c-</a></li> </ul> </li> <li>HexType: Efficient Detection of Type Confusion Errors for C++ <ul> <li>ACM Conference on Computer and Communication Security (CCS) 2017</li> <li>Yuseok Jeon, Priyam Biswas, Scott A. Carr, Byoungyoung Lee, and Mathias Payer.</li> <li>Slides (PDF): <a href="http://nebelwelt.net/publications/files/1734c3-presentation.pdf">http://nebelwelt.net/publications/files/1734c3-presentation.pdf</a></li> <li>Paper (PDF): <a href="https://nebelwelt.net/publications/files/17CCS.pdf">https://nebelwelt.net/publications/files/17CCS.pdf</a></li> <li>34C3 (2017) <ul> <li><a href="https://media.ccc.de/v/34c3-8848-type_confusion_discovery_abuse_and_protection">https://media.ccc.de/v/34c3-8848-type_confusion_discovery_abuse_and_protection</a></li> <li><a href="https://www.youtube.com/watch?v=jbglFfkRYQs">https://www.youtube.com/watch?v=jbglFfkRYQs</a></li> </ul> </li> <li>HexType: Efficient Detection of Type Confusion Errors for C++ <ul> <li><a href="https://github.com/HexHive/HexType">https://github.com/HexHive/HexType</a></li> </ul> </li> </ul> </li> <li>HexVASAN: Venerable Variadic Vulnerabilities Vanquished <ul> <li>Usenix Security Symposium 2017</li> <li>Priyam Biswas, Alessandro Di Federico, Scott A. Carr, Prabhu Rajasekaran, Stijn Volckaert, Yeoul Na, Michael Franz, and Mathias Payer.</li> <li><a href="https://nebelwelt.net/publications/files/17SEC.pdf">https://nebelwelt.net/publications/files/17SEC.pdf</a></li> <li><a href="https://nebelwelt.net/publications/files/17SEC-presentation.pdf">https://nebelwelt.net/publications/files/17SEC-presentation.pdf</a></li> <li><a href="https://github.com/HexHive/HexVASAN">https://github.com/HexHive/HexVASAN</a></li> </ul> </li> <li>MemorySanitizer: fast detector of uninitialized memory use in C++ <ul> <li>Code Generation and Optimization (CGO) 2015</li> <li>Evgeniy Stepanov and Konstantin Serebryany</li> <li><a href="https://research.google.com/pubs/archive/43308.pdf">https://research.google.com/pubs/archive/43308.pdf</a></li> </ul> </li> <li>PartiSan: Fast and Flexible Sanitization via Run-time Partitioning <ul> <li>Research in Attacks, Intrusions and Defenses (RAID) 2018</li> <li>Julian Lettner, Dokyung Song, Taemin Park, Stijn Volckaert, Per Larsen, Michael Franz</li> <li><a href="https://arxiv.org/abs/1711.08108">https://arxiv.org/abs/1711.08108</a></li> </ul> </li> <li>TypeSanitizer: Practical Type Confusion Detection <ul> <li>Computer and Communications Security (CCS) 2016</li> <li>Istvan Haller, Yuseok Jeon, Hui Peng, Mathias Payer, Herbert Bos, Cristiano Giuffrida, Erik van der Kouwe</li> <li><a href="https://nebelwelt.net/publications/files/16CCS2.pdf">https://nebelwelt.net/publications/files/16CCS2.pdf</a></li> <li>TypeSan checks casts in C++ code - code released for CCS 2016 <ul> <li><a href="https://github.com/vusec/typesan">https://github.com/vusec/typesan</a></li> </ul> </li> </ul> </li> </ul> <h3>Projects</h3> <ul> <li>QASan: QEMU-AddressSanitizer <ul> <li><a href="https://github.com/andreafioraldi/qasan">https://github.com/andreafioraldi/qasan</a></li> <li>Sanitized Emulation with QASan <ul> <li><a href="https://andreafioraldi.github.io/articles/2019/12/20/sanitized-emulation-with-qasan.html">https://andreafioraldi.github.io/articles/2019/12/20/sanitized-emulation-with-qasan.html</a></li> </ul> </li> </ul> </li> <li>sanitizers-cmake: CMake modules to help use sanitizers <ul> <li><a href="https://github.com/arsenm/sanitizers-cmake">https://github.com/arsenm/sanitizers-cmake</a></li> </ul> </li> <li>TypeART: LLVM-based type and memory allocation tracking sanitizer <ul> <li><a href="https://github.com/tudasc/TypeART">https://github.com/tudasc/TypeART</a></li> <li>Compiler-aided Type Tracking for Correctness Checking of MPI Applications <ul> <li>International Workshop on Software Correctness for HPC Applications (Correctness) 2018</li> <li>Alexander Hück, Jan-Patrick Lehr, Sebastian Kreutzer, Joachim Protze, Christian Terboven, Christian Bischof, Matthias S. Müller</li> <li><a href="https://doi.org/10.1109/Correctness.2018.00011">https://doi.org/10.1109/Correctness.2018.00011</a></li> <li><a href="https://correctness-workshop.github.io/2018/papers/lehr.pdf">https://correctness-workshop.github.io/2018/papers/lehr.pdf</a></li> </ul> </li> </ul> </li> </ul> <h3>Talks</h3> <ul> <li>AddressSanitizer, ThreadSanitizer and MemorySanitizer -- Dynamic Testing Tools for C++ <ul> <li>GTAC 2013; Kostya Serebryany</li> <li><a href="https://www.youtube.com/watch?v=Q2C2lP8_tNE">https://www.youtube.com/watch?v=Q2C2lP8_tNE</a></li> </ul> </li> <li>Address and Thread Sanitizer in GCC: State of the Onion <ul> <li>GNU Tools Cauldron 2013; Dodji Seketeli</li> <li><a href="https://gcc.gnu.org/wiki/cauldron2013?action=AttachFile&amp;do=view&amp;target=asan.pdf">https://gcc.gnu.org/wiki/cauldron2013?action=AttachFile&amp;do=view&amp;target=asan.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=41_WZk6AePY">https://www.youtube.com/watch?v=41_WZk6AePY</a></li> </ul> </li> <li>C++ Sanitizers <ul> <li>C++ Weekly; Ep. 84 (2017)</li> <li><a href="https://www.youtube.com/watch?v=MB6NPkB4YVs">https://www.youtube.com/watch?v=MB6NPkB4YVs</a></li> </ul> </li> <li>Debugging with LLVM: A quick introduction to LLDB and LLVM sanitizers <ul> <li>FOSDEM 2020</li> <li>Andrzej Warzynski; Graham Hunter</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_with_llvm/">https://fosdem.org/2020/schedule/event/debugging_with_llvm/</a></li> </ul> </li> <li>Finding races and memory errors with LLVM instrumentation <ul> <li>2011 LLVM Developers’ Meeting; Konstantin Serebryany</li> <li><a href="https://www.youtube.com/watch?v=sJqQTUtV6GY">https://www.youtube.com/watch?v=sJqQTUtV6GY</a></li> </ul> </li> <li>Finding races and memory errors with compiler instrumentation: AddressSanitizer, ThreadSanitizer <ul> <li>GNU Tools Cauldron 2012; Konstantin Serebryany, Dmitry Vyukov</li> <li><a href="https://gcc.gnu.org/wiki/cauldron2012?action=AttachFile&amp;do=get&amp;target=kcc.pdf">https://gcc.gnu.org/wiki/cauldron2012?action=AttachFile&amp;do=get&amp;target=kcc.pdf</a></li> <li><a href="https://gcc.gnu.org/wiki/cauldron2012#Finding_races_and_memory_errors_with_GCC_instrumentation_.28AddressSanitizer.29">https://gcc.gnu.org/wiki/cauldron2012#Finding_races_and_memory_errors_with_GCC_instrumentation_.28AddressSanitizer.29</a></li> </ul> </li> <li>MemorySanitizer, ThreadSanitizer: Scalable run-time detection of uninitialized memory reads and data races with LLVM instrumentation <ul> <li>2012 LLVM Developers’ Meeting; Kostya Serebryany</li> <li><a href="https://www.youtube.com/watch?v=HDgttiIvMxA">https://www.youtube.com/watch?v=HDgttiIvMxA</a></li> </ul> </li> <li>News from Sanitizers <ul> <li>GNU Tools Cauldron 2014; Evgeniy Stepanov, Kostya Serebryany</li> <li><a href="https://docs.google.com/presentation/d/1QG2qlzpxlsdDsX4uHhTLWAoUmbzbhedFMoMvviud0Us">https://docs.google.com/presentation/d/1QG2qlzpxlsdDsX4uHhTLWAoUmbzbhedFMoMvviud0Us</a></li> <li><a href="https://www.youtube.com/watch?v=Ek7z_oC0dak">https://www.youtube.com/watch?v=Ek7z_oC0dak</a></li> </ul> </li> <li>The Type Sanitizer: Free Yourself from -fno-strict-aliasing <ul> <li>2017 LLVM Developers’ Meeting; Hal Finkel</li> <li><a href="https://www.youtube.com/watch?v=vAXJeN7k32Y">https://www.youtube.com/watch?v=vAXJeN7k32Y</a></li> </ul> </li> <li>ThreadSanitizer APIs for external libraries <ul> <li>2017 LLVM Developers’ Meeting; Kuba Mracek</li> <li><a href="https://www.youtube.com/watch?v=-J9bMpqfc7A">https://www.youtube.com/watch?v=-J9bMpqfc7A</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/effective-enterprise-java/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/effective-enterprise-java/index.html <p><em>(by Ted Neward, Addison-Wesley)</em></p> <h1>Architecture</h1> <ol> <li> <p>Prefer components as the key element of development,&nbsp;deployment, and reuse</p> </li> <li> <p>Prefer loose coupling across component boundaries</p> </li> <li> <p>Differentiate layers from tiers</p> </li> <li> <p>Keep data and processors close together</p> </li> <li> <p>Remember that identity breeds contention</p> </li> <li> <p>Use hook points to inject optimizations, customizations,&nbsp;or new functionality</p> </li> <li> <p>Be robust in the face of failure</p> </li> <li> <p>Define your performance and scalability goals</p> </li> <li> <p>Restrict EJB to transactional processing</p> </li> <li> <p>Never optimize without profiling first</p> </li> <li> <p>Recognize the cost of vendor neutrality</p> </li> <li> <p>Build in monitoring support</p> </li> <li> <p>Build in administration support</p> </li> <li> <p>Make deployment as simple as possible</p> </li> </ol> <h1>Communication</h1> <ol> <li> <p>Understand all your communications options</p> </li> <li> <p>Consider your lookup carefully</p> </li> <li> <p>Recognize the cost of network access</p> </li> <li> <p>Prefer context-complete communication styles</p> </li> <li> <p>Prefer data-driven communication over behavior-driven&nbsp;communication</p> </li> <li> <p>Avoid waiting for remote service requests to respond</p> </li> <li> <p>Consider partitioning components to avoid excessive load&nbsp;on any one machine</p> </li> <li> <p>Consider using Web Services for open integration</p> </li> </ol> <h1>Data</h1> <ol> <li> <p>Pass data in bulk</p> </li> <li> <p>Consider rolling your own communication proxies</p> </li> <li> <p>Keep it simple</p> </li> <li> <p>Prefer rules engines for complex state evaluation and execution</p> </li> <li> <p>Prefer transactional processing for implicitly nonatomic&nbsp;failure scenarios</p> </li> <li> <p>Differentiate user transactions from system transactions</p> </li> <li> <p>Minimize lock windows</p> </li> <li> <p>Never cede control outside your component while&nbsp;holding locks</p> </li> <li> <p>Understand EJB transactional affinity</p> </li> <li> <p>Prefer local transactions to distributed ones</p> </li> </ol> <h1>Distribution</h1> <ol> <li> <p>Consider using optimistic concurrency for better scalability</p> </li> <li> <p>Consider using pessimistic concurrency for explicit&nbsp;concurrency control</p> </li> <li> <p>Consider lower isolation levels for better transactional&nbsp;throughput</p> </li> <li> <p>Use savepoints to keep partial work in the face of rollback</p> </li> <li> <p>Replicate resources when possible to avoid lock regions</p> </li> <li> <p>Favor the immutable, for it needs no locks</p> </li> <li> <p>Use HttpSession sparingly</p> </li> </ol> <h1>Persistence</h1> <ol> <li> <p>Use objects-first persistence to preserve your domain&nbsp;model</p> </li> <li> <p>Use relational-first persistence to expose the power&nbsp;of the relational model</p> </li> <li> <p>Use procedural-first persistence to create an&nbsp;encapsulation layer</p> </li> <li> <p>Recognize the object-hierarchical impedance mismatch</p> </li> <li> <p>Use in-process or local storage to avoid the network</p> </li> <li> <p>Never assume you own the data or the database</p> </li> <li> <p>Lazy-load infrequently used data</p> </li> <li> <p>Eager-load frequently used data</p> </li> <li> <p>Batch SQL work to avoid round-trips</p> </li> <li> <p>Know your JDBC provider</p> </li> <li> <p>Tune your SQL</p> </li> </ol> <h1>Presentation</h1> <ol> <li> <p>Consider rich-client UI technologies</p> </li> <li> <p>Keep HTML minimal</p> </li> <li> <p>Separate presentation from processing</p> </li> <li> <p>Keep style separate from content</p> </li> <li> <p>Pregenerate content to minimize processing</p> </li> <li> <p>Validate early, validate everything</p> </li> </ol> <h1>Security</h1> <ol> <li> <p>Security is a process, not a product</p> </li> <li> <p>Remember that security is not just prevention</p> </li> <li> <p>Establish a threat model</p> </li> <li> <p>Assume insecurity</p> </li> <li> <p>Always validate user input</p> </li> <li> <p>Turn on platform security</p> </li> <li> <p>Use role-based authorization</p> </li> <li> <p>Use SignedObject to provide integrity of Serialized objects</p> </li> <li> <p>Use SealedObject to provide confidentiality of Serializable&nbsp;objects</p> </li> <li> <p>Use GuardedObject to provide access control on objects</p> </li> </ol> <h1>System</h1> <ol> <li> <p>Aggressively release resources</p> </li> <li> <p>Tune the JVM</p> </li> <li> <p>Use independent JREs for side-by-side versioning</p> </li> <li> <p>Recognize ClassLoader boundaries</p> </li> <li> <p>Understand Java Object Serialization</p> </li> <li> <p>Don’t fight the garbage collector</p> </li> <li> <p>Prefer container-managed resource management</p> </li> <li> <p>Use reference objects to augment garbage collection behavior</p> </li> <li> <p>Don’t be afraid of JNI code on the server</p> </li> </ol> https://tedneward.github.io/Research/reading/development/free-programming-books/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/free-programming-books/index.html <p><a href="https://github.com/EbookFoundation/free-programming-books/blob/main/books/free-programming-books-langs.md">Website/Source</a></p> <p>Eventually all of these links should move to their respective pages.</p> <h2>BY PROGRAMMING LANGUAGE</h2> <h3>AutoIt</h3> <ul> <li><a href="https://www.autoitscript.com/autoit3/docs/">AutoIt Docs</a> - Jonathan Bennett (HTML)</li> </ul> <h3>BeanShell</h3> <ul> <li><a href="http://www.beanshell.org/manual/bshmanual.pdf">Beanshell Simple Java Scripting Manual</a> - beanshell.org (PDF)</li> <li><a href="http://www.beanshell.org/manual/bshmanual.html">BeanShell User's Manual</a> - beanshell.org (HTML)</li> </ul> <h3>Cilk</h3> <ul> <li><a href="https://par.tuwien.ac.at/material/manual-5.4.6.pdf">Cilk 5.4.6 Reference Manual</a> (PDF)</li> </ul> <h3>ColdFusion</h3> <ul> <li><a href="https://github.com/mhenke/CFML-in-100-minutes/blob/master/cfml100mins.markdown">CFML In 100 Minutes</a> - J. Casimir</li> <li><a href="http://learncfinaweek.com">Learn CF in a Week</a></li> </ul> <h3>Cool</h3> <ul> <li><a href="https://www.eecis.udel.edu/~cavazos/cisc672/docs/cool-manual.pdf">CoolAid: The Cool 2013 Reference Manual</a> (PDF)</li> </ul> <h3>Coq</h3> <ul> <li><a href="http://adam.chlipala.net/cpdt/html/toc.html">Certified Programming with Dependent Types</a></li> <li><a href="http://www.cis.upenn.edu/~bcpierce/sf/">Software Foundations</a></li> </ul> <h3>CUDA</h3> <ul> <li><a href="https://docs.nvidia.com/pdf/CUDA_C_Best_Practices_Guide.pdf">CUDA C Best Practices Guide</a> (PDF)</li> <li><a href="https://docs.nvidia.com/pdf/CUDA_C_Programming_Guide.pdf">CUDA C Programming Guide</a> (PDF)</li> <li><a href="http://www.nvidia.com/content/cudazone/download/OpenCL/NVIDIA_OpenCL_ProgrammingGuide.pdf">OpenCL Programming Guide for CUDA Architecture</a> (PDF)</li> </ul> <h3>DB2</h3> <ul> <li><a href="http://public.dhe.ibm.com/software/dw/db2/express-c/wiki/Getting_Started_with_DB2_Express_v9.7_p4.pdf">Getting started with DB2 Express-C</a> (PDF)</li> <li><a href="http://public.dhe.ibm.com/software/dw/db2/express-c/wiki/Getting_Started_with_IBM_Data_Studio_for_DB2_p3.pdf">Getting started with IBM Data Studio for DB2</a> (PDF)</li> <li><a href="http://public.dhe.ibm.com/software/dw/db2/express-c/wiki/Getting_Started_with_DB2_App_Dev_p2.pdf">Getting started with IBM DB2 development</a> (PDF)</li> </ul> <h3>DBMS</h3> <ul> <li><a href="http://www.lincoste.com/ebooks/english/pdf/computers/database_management_systems.pdf">Database Management Systems eBooks For All Edition</a> (PDF)</li> </ul> <h3>DTrace</h3> <ul> <li><a href="http://dtrace.org/guide/preface.html">IllumOS Dynamic Tracing Guide</a></li> </ul> <h3>Emacs</h3> <ul> <li><a href="https://www.gnu.org/software/emacs/manual/eintr.html">An Introduction to Programming in Emacs Lisp</a></li> <li><a href="https://www.finseth.com/craft/">Emacs for the Modern World</a> (HTML)</li> <li><a href="http://www.gnu.org/software/emacs/manual/elisp.html">GNU Emacs Lisp Reference Manual</a></li> <li><a href="https://www.gnu.org/software/emacs/manual/emacs.html">GNU Emacs Manual</a></li> </ul> <h3>Embedded Systems</h3> <ul> <li><a href="http://www.learn-c.com">Control and Embedded Systems</a> (HTML)</li> <li><a href="http://www.cs.indiana.edu/~geobrown/book.pdf">Discovering the STM32 Microcontroller</a> (PDF)</li> <li><a href="https://www.phaedsys.com/principals/bytecraft/bytecraftdata/bcfirststeps.pdf">First Steps with Embedded Systems</a> - Byte Craft Limited (PDF)</li> <li><a href="https://ptolemy.berkeley.edu/books/leeseshia/releases/LeeSeshia_DigitalV2_2.pdf">Introduction to Embedded Systems, Second Edition</a> - Edward Ashford Lee, Sanjit Arunkumar Seshia (PDF)</li> <li><a href="http://www.embeddedrelated.com/showarticle/453.php">Introduction to Microcontrollers</a> (HTML)</li> <li><a href="https://freertos.org/Documentation/RTOS_book.html">Mastering the FreeRTOS Real Time Kernel - a Hands On Tutorial Guide</a> - freertos.org (<a href="https://freertos.org/fr-content-src/uploads/2018/07/161204_Mastering_the_FreeRTOS_Real_Time_Kernel-A_Hands-On_Tutorial_Guide.pdf">PDF</a>)</li> </ul> <h3>ESP8266</h3> <ul> <li><a href="https://leanpub.com/ESP8266_ESP32">Kolban's book on the ESP32 &amp; ESP8266</a> <em>(Leanpub account or valid email requested)</em></li> </ul> <h3>Firefox OS</h3> <ul> <li><a href="https://leanpub.com/quickguidefirefoxosdevelopment/read">Quick Guide For Firefox OS App Development: Creating HTML5 based apps for Firefox OS</a> - Andre Garzia</li> </ul> <h3>FreeBSD</h3> <ul> <li><a href="http://www.freebsd.org/docs/books.html">Books and Articles from FreeBSD Site</a></li> <li><a href="http://www.lemis.com/grog/Documentation/CFBSD/">The Complete FreeBSD</a></li> <li><a href="http://www.bitsinthewind.com/about-dac/publications/using-c-on-the-unix-system">Using C on the UNIX System</a> - David A. Curry</li> </ul> <h3>Hack</h3> <ul> <li><a href="https://docs.hhvm.com/hack/">Hack Documentation</a></li> </ul> <h3>Hadoop</h3> <ul> <li><a href="https://www.packtpub.com/free-ebooks/big-data-analytics-hadoop-3">Big Data Analytics with Hadoop 3</a> - Sridhar Alla (Packt account <em>required</em>)</li> <li><a href="https://docs.cloudera.com/documentation/enterprise/latest/PDF/cloudera-impala.pdf">Cloudera Impala</a> - John Russel (PDF)</li> <li><a href="http://lintool.github.io/MapReduceAlgorithms/MapReduce-book-final.pdf">Data-Intensive Text Processing with MapReduce</a> (Jimmy Lin and Chris Dyer) (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/hadoop-for-windows-succinctly">Hadoop for Windows Succinctly</a> - Dave Vickers</li> <li><a href="http://hadoopilluminated.com/index.html">Hadoop Illuminated</a> - Mark Kerzner &amp; Sujee Maniyam</li> </ul> <h3>Idris</h3> <ul> <li><a href="https://leanpub.com/gidti">Gentle Introduction to Dependent Types with Idris</a> - Boro Sitnikovski <em>(Leanpub account or valid email requested)</em></li> </ul> <h3>Icon</h3> <ul> <li><a href="http://www.cs.arizona.edu/icon/ibsale.htm">The Implementation of the Icon Programming Language</a></li> </ul> <h3>IoT</h3> <ul> <li><a href="https://github.com/marcozennaro/IPv6-WSN-book/tree/master/Releases">IoT in five days- V1.1</a> (PDF, EPUB)</li> <li><a href="https://www.packtpub.com/free-ebooks/mastering-internet-things">Mastering Internet of Things</a> - Peter Waher (Packt account <em>required</em>)</li> </ul> <h3>Isabelle/HOL</h3> <ul> <li><a href="http://www21.in.tum.de/~nipkow/Concrete-Semantics/">Concrete Semantics - A Proof Assistant Approach by Tobias Nipkow and Gerwin Klein</a> (PDF)</li> <li><a href="http://isabelle.in.tum.de/doc/tutorial.pdf">Isabelle/HOL - A Proof Assistant for Higher-Order Logic by Tobias Nipkow and Lawrence C. Paulson and Markus Wenzel</a> (PDF)</li> </ul> <h3>J</h3> <ul> <li><a href="http://www.jsoftware.com/books/pdf/arithmetic.pdf">Arithmetic by Kenneth E Iverson</a> (PDF)</li> <li><a href="http://www.jsoftware.com/books/pdf/brief.pdf">Brief Reference by Chris Burke and Clifford Reiter</a> (PDF)</li> <li><a href="http://www.jsoftware.com/books/pdf/calculus.pdf">Calculus by Kenneth E Iverson</a> (PDF)</li> <li><a href="http://www.jsoftware.com/papers/camn.htm">Computers and Mathematical Notation by Kenneth E Iverson</a></li> <li><a href="http://www.jsoftware.com/books/pdf/cmc.pdf">Concrete Math Companion by Kenneth E Iverson</a> (PDF)</li> <li><a href="http://www.jsoftware.com/books/pdf/easyj.pdf">Easy J by Linda Alvord, Norman Thomson</a> (PDF) (<a href="http://www.jsoftware.com/books/doc/easyj_doc.zip">Word DOC</a>)</li> <li><a href="http://www.jsoftware.com/books/pdf/expmath.pdf">Exploring Math by Kenneth E Iverson</a> (PDF)</li> <li><a href="http://www.jsoftware.com/help/jforc/contents.htm">J for C Programmers by Henry Rich</a></li> <li><a href="http://www.jsoftware.com/help/primer/contents.htm">J Primer</a></li> <li><a href="http://www.jsoftware.com/help/learning/contents.htm">Learning J by Roger Stokes- online</a></li> <li><a href="http://www.jsoftware.com/books/pdf/mftl.zip">Math for the Layman by Kenneth E Iverson</a> (zipped html+images)</li> </ul> <h3>Java</h3> <h4>Java Reporting</h4> <ul> <li><a href="https://leanpub.com/itext_pdfabc">The ABC of PDF with iText: PDF Syntax essentials</a> - Bruno Lowagie <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://jasperreports.sourceforge.net/JasperReports-Ultimate-Guide-3.pdf">The JasperReports Ultimate Guide, Third Edition</a> (PDF)</li> </ul> <h4>Solidity</h4> <ul> <li><a href="https://www.tutorialspoint.com/solidity/index.htm">Introductory guide for Solidity</a></li> <li><a href="https://docs.soliditylang.org">The Solidity Reference Guide</a></li> </ul> <h3>Jenkins</h3> <ul> <li><a href="http://www.bogotobogo.com/DevOps/Jenkins/images/Intro_install/jenkins-the-definitive-guide.pdf">Jenkins: The Definitive Guide</a> (PDF)</li> <li><a href="https://www.jenkins.io/user-handbook.pdf">Jenkins User Handbook</a> (PDF)</li> </ul> <h3>LaTeX / TeX</h3> <h4>LaTeX</h4> <ul> <li><a href="http://latex.knobs-dials.com">Arbitrary LaTex Reference</a></li> <li><a href="https://github.com/VoLuong/Begin-Latex-in-minutes">Begin Latex in minutes</a></li> <li><a href="https://en.wikibooks.org/wiki/LaTeX">LaTeX</a> - Wikibooks</li> <li><a href="https://goalkicker.com/LaTeXBook/">LaTex Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://tobi.oetiker.ch/lshort/lshort.pdf">The Not So Short Introduction to LaTeX</a> (PDF)</li> </ul> <h4>TeX</h4> <ul> <li><a href="http://pgfplots.sourceforge.net/TeX-programming-notes.pdf">Notes On Programming in TeX</a> - Christian Feursänger (PDF)</li> <li><a href="http://eijkhout.net/texbytopic/texbytopic.html">TeX by Topic, A TeXnician's Reference</a> - Victor Eijkhout</li> <li><a href="https://www.gnu.org/software/teximpatient/">TeX for the Impatient</a> - Paul Abrahams, Kathryn Hargreaves, and Karl Berry</li> </ul> <h3>Language Agnostic</h3> <ul> <li><a href="free-programming-books-subjects.md">BY SUBJECT</a> This section has been moved to its own file.</li> </ul> <h3>Limbo</h3> <ul> <li><a href="http://doc.cat-v.org/inferno/books/inferno_programming_with_limbo/">Inferno Programming With Limbo</a></li> </ul> <h3>Livecode</h3> <ul> <li><a href="http://www.scribd.com/doc/216789127/LiveCode-userguide">LiveCode userguide</a> (PDF)</li> </ul> <h3>Make</h3> <ul> <li><a href="https://makefiletutorial.com">Makefile tutorial</a> - Chase Lambert</li> <li><a href="https://www.oreilly.com/openbook/make3/book/index.html">Managing Projects with GNU Make</a> - Robert Mecklenburg</li> </ul> <h3>Markdown</h3> <ul> <li><a href="https://www.gitbook.com/book/gitbookio/markdown/details">Learn Markdown</a> - Sammy P., Aaron O. (PDF) (EPUB) (MOBI)</li> </ul> <h3>Mathematica</h3> <ul> <li><a href="http://www.mathprogramming-intro.org">Mathematica® programming: an advanced introduction by Leonid Shifrin</a></li> <li><a href="http://mathematica.stackexchange.com/questions/16485/are-you-interested-in-purchasing-david-wagners-power-programming-with-mathemat/22724">Power Programming with Mathematica</a> - David B. Wagner</li> <li><a href="http://reference.wolfram.com/legacy/v5_2/">Stephen Wolfram's The Mathematica Book</a></li> <li><a href="http://chortle.ccsu.edu/VectorLessons/index.html">Vector Math for 3d Computer Graphics</a></li> <li><a href="https://www.wolfram.com/wolfram-u/catalog/product-training/mathematica/">Wolfram Mathematica Product Training: Wolfram U</a></li> </ul> <h3>MATLAB</h3> <ul> <li><a href="http://www.science.smith.edu/~jcardell/Courses/EGR326/Intro-to-MATLAB.pdf">An Interactive Introduction to MATLAB</a> (PDF)</li> <li><a href="http://www.maths.dundee.ac.uk/software/MatlabNotes.pdf">An Introduction to MATLAB</a> (PDF)</li> <li><a href="http://www.intechopen.com/books/applications-of-matlab-in-science-and-engineering">Applications of MATLAB in Science and Engineering</a></li> <li><a href="http://www.mathworks.com/moler/exm/index.html?requestedDomain=www.mathworks.com&amp;nocookie=true">Experiments with MATLAB</a></li> <li><a href="https://cnx.org/exports/3a643c1f-c1ba-4c2a-8065-317a1f2b1add@18.1.pdf/freshman-engineering-problem-solving-with-matlab-18.1.pdf">Freshman Engineering Problem Solving with MATLAB</a> (PDF)</li> <li><a href="http://www.mathworks.com/tutorials">Interactive Tutorials for MATLAB, Simulink, Signal Processing, Controls, and Computational Mathematics</a></li> <li><a href="https://www.mccormick.northwestern.edu/documents/students/undergraduate/introduction-to-matlab.pdf">Introduction to MATLAB for Engineering Students</a> - David Houcque (PDF) (1.2, 2005)</li> <li><a href="http://www.intechopen.com/books/matlab-a-fundamental-tool-for-scientific-computing-and-engineering-applications-volume-1">MATLAB - A Fundamental Tool for Scientific Computing and Engineering Applications - Volume 1</a></li> <li><a href="http://www.intechopen.com/books/matlab-a-ubiquitous-tool-for-the-practical-engineer">MATLAB - A Ubiquitous Tool for the Practical Engineer</a></li> <li><a href="http://www.intechopen.com/books/matlab-for-engineers-applications-in-control-electrical-engineering-it-and-robotics">MATLAB for Engineers: Applications in Control, Electrical Engineering, IT and Robotics</a></li> <li><a href="https://goalkicker.com/MATLABBook">MATLAB Notes for professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://en.wikibooks.org/wiki/MATLAB_Programming">MATLAB Programming</a> - Wikibooks</li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/matlab">MATLAB Succinctly, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="http://www.mathworks.com/moler/index_ncm.html?requestedDomain=www.mathworks.com&amp;nocookie=true">Numerical Computing with MATLAB</a></li> <li><a href="http://greenteapress.com/matlab/index.html">Physical Modeling in MATLAB</a> - Alan B. Downey</li> <li><a href="https://www.math.ust.hk/~machas/scientific-computing.pdf">Scientific Computing</a> - Jeffrey R. Chasnov (PDF)</li> </ul> <h3>Maven</h3> <ul> <li><a href="https://books.sonatype.com/m2eclipse-book/reference/index.html">Developing with Eclipse and Maven</a></li> <li><a href="http://books.sonatype.com/mvnex-book/reference/public-book.html">Maven by Example</a></li> <li><a href="http://books.sonatype.com/mvnref-book/reference/public-book.html">Maven: The Complete Reference</a></li> <li><a href="http://books.sonatype.com/nexus-book/reference/">Repository Management with Nexus</a></li> </ul> <h3>Mercurial</h3> <ul> <li><a href="https://hginit.github.io">Hg Init: a Mercurial Tutorial</a> - Joel Spolsky</li> <li><a href="http://hgbook.red-bean.com">Mercurial: The Definitive Guide</a></li> <li><a href="https://book.mercurial-scm.org">Mercurial: The Definitive Guide 2nd edition</a></li> </ul> <h3>Mercury</h3> <ul> <li><a href="http://www.mercurylang.org/information/doc-release/user_guide.pdf">The Mercury Users' Guide</a> (PDF)</li> </ul> <h3>Modelica</h3> <ul> <li><a href="http://book.xogeny.com">Modelica by Example</a></li> </ul> <h3>.NET Core</h3> <ul> <li><a href="https://github.com/thangchung/clean-code-dotnet">Clean Code .NET</a></li> <li><a href="https://www.syncfusion.com/succinctly-free-ebooks/entity-frame-work-core-succinctly">Entity Framework Core Succinctly</a> - Ricardo Peres</li> <li><a href="https://docs.microsoft.com/en-us/dotnet/">.NET documentation - Microsoft Docs</a></li> <li><a href="https://www.syncfusion.com/ebooks/using-netcore-docker-and-kubernetes-succinctly">Using .NET Core, Docker, and Kubernetes Succinctly</a> - Michele Aponte</li> </ul> <h3>.NET Framework</h3> <ul> <li><a href="https://www.syncfusion.com/ebooks/akka_net_succinctly">Akka.NET Succinctly</a> - Zoran Maksimovic</li> <li><a href="https://www.syncfusion.com/ebooks/application_security_in_net_succinctly">Application Security in .NET Succinctly</a> - Stan Drapkin</li> <li><a href="https://www.syncfusion.com/ebooks/cryptography_in_net_succinctly">Cryptography in .NET Succinctly</a> - Dirk Strauss</li> <li><a href="http://weblogs.asp.net/zeeshanhirani/my-christmas-present-to-the-entity-framework-community">Entity Framework</a></li> <li><a href="https://en.wikibooks.org/wiki/Game_Creation_with_XNA">Game Creation with XNA</a> - Wikibooks</li> <li><a href="https://www.syncfusion.com/ebooks/getting-the-most-from-linqpad-succinctly">Getting the Most from LINQPad Succinctly</a> - José Roberto Olivas Mendoza</li> <li><a href="https://www.syncfusion.com/ebooks/monogame_succinctly">MonoGame Succinctly</a> - Jim Perry</li> <li><a href="http://foxcentral.net/microsoft/NETforVFPDevelopers.htm">.NET for Visual FoxPro Developers</a></li> <li><a href="https://goalkicker.com/DotNETFrameworkBook/">.NET Framework Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="https://www.red-gate.com/library/net-performance-testing-and-optimization-the-complete-guide">.NET Performance Testing and Optimization - The Complete Guide</a> - Paul Glavich, Chris Farrell (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/nuget-in-house-succinctly">NuGet In-House Succinctly</a> - José Roberto Olivas Mendoza</li> <li><a href="https://www.syncfusion.com/ebooks/rider-succinctly">Rider Succinctly</a> - Dmitri Nesteruk</li> <li><a href="https://assets.red-gate.com/community/books/under-the-hood-of-net-memory-management.pdf">Under the Hood of .NET Memory Management</a> - Chris Farrell, Nick Harrison (PDF)</li> <li><a href="http://www.infoq.com/minibooks/vsnettt">Visual Studio .NET Tips and Tricks</a> (VS 2003-2005 only)</li> <li><a href="https://www.syncfusion.com/ebooks/visual-studio-2019-succinctly">Visual Studio 2019 Succinctly</a> - Alessandro Del Sole</li> </ul> <h3>Oberon</h3> <ul> <li><a href="https://inf.ethz.ch/personal/wirth/AD.pdf">Algorithms and Data-Structures</a> - Niklaus Wirth (PDF)</li> <li><a href="http://ssw.jku.at/Research/Books/Oberon2.pdf">Object-Oriented Programming in Oberon-2</a> - Hanspeter Mössenböck (PDF)</li> <li><a href="https://www.inf.ethz.ch/personal/wirth/ProgInOberonWR.pdf">Programming in Oberon</a> - Niklaus Wirth (PDF)</li> </ul> <h3>Octave</h3> <ul> <li><a href="https://en.wikibooks.org/wiki/Octave_Programming_Tutorial">Octave Programming</a> - Wikibooks</li> </ul> <h3>OpenMP</h3> <ul> <li><a href="http://bisqwit.iki.fi/story/howto/openmp/">A Guide To OpenMP</a></li> <li><a href="http://www.openmp.org/mp-documents/OpenMP4.0.0.pdf">OpenMP Application Programming Interface Standard Version 4.0</a> (PDF)</li> <li><a href="https://www.openmp.org/wp-content/uploads/OpenMP-API-Specification-5.0.pdf">OpenMP Application Programming Interface Standard Version 5.0</a> (PDF)</li> </ul> <h3>OpenResty</h3> <ul> <li><a href="https://www.gitbook.com/book/openresty/programming-openresty/details">Programming OpenResty</a></li> </ul> <h3>OpenSCAD</h3> <ul> <li><a href="https://en.wikibooks.org/wiki/OpenSCAD_User_Manual">OpenSCAD User Manual</a> - Wikibooks</li> </ul> <h3>TrueOS</h3> <ul> <li><a href="https://www.trueos.org/handbook/trueos.html">TrueOS® Users Handbook</a></li> </ul> <h3>PicoLisp</h3> <ul> <li><a href="https://github.com/tj64/picolisp-by-example">PicoLisp by Example</a></li> <li><a href="https://github.com/tj64/picolisp-works">PicoLisp Works</a></li> </ul> <h3>PowerShell</h3> <ul> <li><a href="https://leanpub.com/aunixpersonsguidetopowershell">A Unix Person's Guide to PowerShell</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/creatinghtmlreportsinwindowspowershell">Creating HTML Reports in PowerShell</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/devopstheopsperspective">DevOps: The Ops Perspective</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/ditchexcelmakinghistoricalandtrendreportsinpowershell">Ditch Excel: Making Historical &amp; Trend Reports in PowerShell</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://ravichaganti.com/ebooks/AlaymansguidetoPowerShell2remotingv2.pdf">Layman’s Guide to PowerShell 2.0 remoting</a> (PDF)</li> <li><a href="https://www.packtpub.com/free-ebooks/learn-powershell-core-60">Learn PowerShell Core 6.0</a> - David das Neves, Jan-Hendrik Peters (Packt account <em>required</em>)</li> <li><a href="https://learnxinyminutes.com/docs/powershell/">Learn PowerShell in Y Minutes</a></li> <li><a href="http://community.idera.com/powershell/powertips/b/ebookv2#pi619PostSortOrder=Ascending">Mastering PowerShell v2</a></li> <li><a href="https://leanpub.com/powershell101">PowerShell 101: The No-Nonsense Beginner’s Guide to PowerShell</a> - Mike F. Robbins <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://www.jonathanmedd.net/wp-content/uploads/2010/09/PowerShell_2_One_Cmdlet_at_a_Time.pdf">PowerShell 2.0 – One CMDLET At A Time</a> (PDF)</li> <li><a href="http://goalkicker.com/PowerShellBook/">PowerShell Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/powershell">PowerShell Succinctly, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://leanpub.com/powershelltips">PowerShell Tips to Write By</a> - Adam Bertram <em>(Leanpub account or valid email requested)</em> (:construction: <em>in process</em>)</li> <li><a href="https://leanpub.com/secretsofpowershellremoting">Secrets of PowerShell Remoting</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/thebigbookofpowershellerrorhandling">The Big Book of PowerShell Error Handling</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/thebigbookofpowershellgotchas">The Big Book of PowerShell Gotchas</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/themonadmanifestoannotated">The Monad Manifesto - Annotated</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/windowspowershellnetworkingguide">The PowerShell + DevOps Global Summit Manual for Summiteers</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/whypowershell">Why PowerShell?</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/windowspowershellnetworkingguide">Windows PowerShell Networking Guide</a> - The DevOps Collective, Inc. (PDF, ePub, MOBI, HTML) <em>(Leanpub account or valid email requested)</em></li> </ul> <h3>Processing</h3> <ul> <li><a href="http://natureofcode.com/book/">The Nature of Code: Simulating Natural Systems with Processing</a></li> </ul> <h3>PureScript</h3> <ul> <li><a href="https://leanpub.com/purescript/read">PureScript By Example</a> - Phil Freeman</li> </ul> <h3>QML</h3> <ul> <li><a href="http://qmlbook.github.io">Qt5 Cadaques</a> - Juergen Bocklage-Ryannel and Johan Thelin (HTML, PDF, ePub) (:construction: <em>in process</em>)</li> </ul> <h3>R</h3> <ul> <li><a href="http://adv-r.had.co.nz">Advanced R Programming</a> - Hadley Wickham</li> <li><a href="https://cran.r-project.org/doc/manuals/R-intro.html">An Introduction to R</a> -David M. Smith and William N. Venables</li> <li><a href="https://web.stanford.edu/~hastie/ISLR2/ISLRv2_website.pdf">An Introduction to Statistical Learning with Applications in R</a> - Gareth James, Daniela Witten, Trevor Hastie and Robert Tibshirani (PDF)</li> <li><a href="https://bookdown.org/yihui/blogdown/">blogdown: Creating Websites with R Markdown</a> - Yihui Xie, Amber Thomas, Alison Presmanes Hill</li> <li><a href="http://www.cookbook-r.com">Cookbook for R</a> - Winston Chang</li> <li><a href="https://rafalab.github.io/dsbook/">Data Analysis and Prediction Algorithms with R</a> - Rafael A. Irizarry</li> <li><a href="https://leanpub.com/dataanalysisforthelifesciences">Data Analysis for the Life Sciences</a> - Rafael A Irizarry, Michael I Love <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://csgillespie.github.io/efficientR/">Efficient R programming</a> - Colin Gillespie, Robin Lovelace</li> <li><a href="https://leanpub.com/exdata">Exploratory Data Analysis with R</a> - Roger D. Peng <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://otexts.com/fpp3/">Forecasting: Principles and Practice</a> - Rob J Hyndman and George Athanasopoulos</li> <li><a href="https://dcl-prog.stanford.edu">Functional Programming</a> - Sara Altman, Bill Behrman and Hadley Wickham</li> <li><a href="https://github.com/gjkerns/IPSUR">Introduction to Probability and Statistics Using R</a> - G. Jay Kerns (PDF)</li> <li><a href="https://learningstatisticswithr.com/book/">Learning Statistics with R</a> - Danielle Navarro</li> <li><a href="https://ismayc.github.io/moderndiver-book/">ModernDive</a> - Chester Ismay and Albert Y. Kim</li> <li><a href="http://cran.r-project.org/doc/contrib/Faraway-PRA.pdf">Practical Regression and Anova using R</a> - Julian J. Faraway (PDF)</li> <li><a href="http://r4ds.had.co.nz">R for Data Science</a> - Garrett Grolemund and Hadley Wickham</li> <li><a href="http://www.columbia.edu/~cjd11/charles_dimaggio/DIRE/resources/spatialEpiBook.pdf">R for Spatial Analysis</a> (PDF)</li> <li><a href="http://www.johndcook.com/blog/r_language_for_programmers">R Language for Programmers</a> - John D. Cook</li> <li><a href="https://goalkicker.com/RBook/">R Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="http://r-pkgs.had.co.nz">R Packages</a> - Hadley Wickham</li> <li><a href="http://www.columbia.edu/~cjd11/charles_dimaggio/DIRE/resources/R/practicalsBookNoAns.pdf">R Practicals</a> (PDF)</li> <li><a href="https://en.wikibooks.org/wiki/R_Programming">R Programming</a> - Wikibooks</li> <li><a href="https://leanpub.com/rprogramming">R Programming for Data Science</a> - Roger D. Peng <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/rsuccinctly">R Succinctly, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://rcompanion.org/handbook/index.html">Summary and Analysis of Extension Program Evaluation in R</a> - Salvatore S. Mangiafico</li> <li><a href="http://topepo.github.io/caret/index.html">The caret Package</a> - Max Kuhn</li> <li><a href="http://www.burns-stat.com/pages/Tutor/R_inferno.pdf">The R Inferno</a> - Patrick Burns (PDF)</li> <li><a href="http://stat.ethz.ch/R-manual/R-patched/doc/html">The R Language</a></li> <li><a href="http://cran.r-project.org/manuals.html">The R Manuals</a></li> <li><a href="http://tidytextmining.com">Tidy Text Mining with R</a> - Julia Silge and David Robinson</li> </ul> <h3>Racket</h3> <ul> <li><a href="https://htdp.org/2019-02-24/">How to Design Programs</a> - Matthias Felleisen, Robert Bruce Findler, Matthew Flatt, Shriram Krishnamurthi</li> <li><a href="http://cs.brown.edu/courses/cs173/2012/book/index.html">Programming Languages: Application and Interpretation</a></li> <li><a href="http://docs.racket-lang.org/guide/index.html">The Racket Guide</a></li> </ul> <h3>Raku</h3> <ul> <li><a href="https://kyclark.gitbooks.io/metagenomics">Metagenomics</a> - Ken Youens-Clark</li> <li><a href="https://andrewshitov.com/wp-content/uploads/2020/01/Perl-6-at-a-Glance.pdf">Perl 6 at a Glance</a> - Andrew Shitov (PDF)</li> <li><a href="https://raku.guide">Raku Guide</a> (HTML) <a href="https://github.com/hankache/rakuguide">(PDF)</a></li> <li><a href="https://andrewshitov.com/wp-content/uploads/2020/01/Raku-One-Liners.pdf">Raku One-Liners</a> - Andrew Shitov (PDF)</li> <li><a href="https://en.wikibooks.org/wiki/Raku_Programming">Raku Programming</a> - Wikibooks (HTML)</li> <li><a href="https://github.com/LaurentRosenfeld/think_raku/raw/master/PDF/think_raku.pdf">Think Raku</a> - Laurent Rosenfeld, with Allen B. Downey (PDF)</li> <li><a href="https://github.com/perl6/book/">Using Perl 6</a> (:construction: <em>project is dead</em>)</li> <li><a href="https://learnxinyminutes.com/docs/raku/">X=Raku</a> - Learn X in Y minutes (HTML)</li> </ul> <h3>Raspberry Pi</h3> <ul> <li><a href="https://leanpub.com/RPiMRE/read">Raspberry Pi: Measure, Record, Explore</a> - Malcolm Maclean (HTML)</li> <li><a href="http://www.cs.unca.edu/~bruce/Fall14/360/RPiUsersGuide.pdf">Raspberry Pi Users Guide - (2012)</a> - Eben Upton (PDF)</li> <li><a href="https://magpi.raspberrypi.com/books/projects-1">The Official Raspberry Pi Project Book 1 (2015)</a> (PDF)</li> </ul> <h3>REBOL</h3> <ul> <li><a href="http://www.lulu.com/shop/nick-antonaccio/learn-rebol/ebook/product-17383182.html">Learn REBOL</a> - Nick Antonaccio</li> </ul> <h3>Sage</h3> <ul> <li><a href="http://wstein.org/books/sagebook/sagebook.pdf">Sage for Power Users</a> - William Stein (PDF)</li> <li><a href="http://www.sagemath.org/doc/">The Sage Manuals</a></li> </ul> <h3>Scilab</h3> <ul> <li><a href="http://forge.scilab.org/index.php/p/docintrotoscilab/downloads/">Introduction to Scilab</a></li> <li><a href="http://forge.scilab.org/index.php/p/docprogscilab/downloads/">Programming in Scilab</a></li> <li><a href="http://forge.scilab.org/index.php/p/docsciextensions/downloads/">Writing Scilab Extensions</a></li> </ul> <h3>Scratch</h3> <ul> <li><a href="http://scratched.gse.harvard.edu/guide/download.html">An Introductory Computing Curriculum Using Scratch</a></li> <li><a href="https://stwww1.weizmann.ac.il/scratch/scratch_en/">Computer Science Concepts in Scratch</a></li> <li><a href="https://magpi.raspberrypi.com/books/essentials-scratch-v1">Learn to Code with Scratch</a> - The MagPi magazine (PDF)</li> <li><a href="https://inventwithscratch.com/book/">Scratch Programming Playground</a> - Al Sweigart</li> </ul> <h3>Snap</h3> <ul> <li><a href="https://snap.berkeley.edu/snapsource/help/SnapManual.pdf">Snap! Reference Manual</a> - B. Harvey, J. Mönig (PDF)</li> </ul> <h3>Spark</h3> <ul> <li><a href="https://www.gitbook.com/book/databricks/databricks-spark-knowledge-base/details">Databricks Spark Knowledge Base</a></li> <li><a href="https://www.gitbook.com/book/databricks/databricks-spark-reference-applications/details">Databricks Spark Reference Applications</a></li> <li><a href="https://www.gitbook.com/book/jaceklaskowski/mastering-apache-spark/details">Mastering Apache Spark</a></li> </ul> <h3>Splunk</h3> <ul> <li><a href="https://www.packtpub.com/free-ebooks/splunk-7x-quick-start-guide">Splunk 7.x Quick Start Guide</a> - James H. Baxter (Packt account <em>required</em>)</li> </ul> <h3>Subversion</h3> <ul> <li><a href="http://ptgmedia.pearsoncmg.com/images/0131855182/downloads/Nagel_book.pdf">Subversion Version Control</a> (PDF)</li> <li><a href="http://svnbook.red-bean.com">Version Control with Subversion</a></li> </ul> <h3>TEI</h3> <ul> <li><a href="https://books.openedition.org/oep/426">What is the Text Encoding Initiative?</a> - Lou Bernard</li> </ul> <h3>Teradata</h3> <ul> <li><a href="http://www.info.teradata.com">Teradata Books</a></li> </ul> <h3>Tizen</h3> <ul> <li><a href="https://developer.tizen.org/sites/default/files/documentation/guide_to_developing_tizen_native_application_en_2.pdf">Guide to Developing Tizen Native Application</a> - Jung, Dong-Geun (Denis.Jung) (PDF)</li> </ul> <h3>TLA</h3> <ul> <li><a href="http://research.microsoft.com/en-us/um/people/lamport/tla/book.html">Specifying Systems: The TLA+ Language and Tools for Hardware and Software Engineers</a> - Leslie Lamport (Postscript or PDF)</li> </ul> <h4>Deno</h4> <ul> <li><a href="https://deno.land/manual">Deno Manual</a></li> <li><a href="https://flaviocopes.com/deno">FlavioCopes Deno Handbook</a> - <em>free PDF with registration</em></li> </ul> <h3>Unix</h3> <ul> <li><a href="http://www.oliverelliott.org/article/computing/tut_unix/">An Introduction to Unix</a></li> <li><a href="http://beej.us/guide/bgipc/">Beej's Guide to Unix Interprocess Communication</a> - B. Hall (HTML,PDF)</li> <li><a href="http://www.lemis.com/grog/Documentation/Lions/">Commentary on the Sixth Edition UNIX Operating System</a> - J. Lions</li> <li><a href="https://homepages.uc.edu/~thomam/Intro_Unix_Text/TOC.html">INTRODUCTION TO UNIX</a> - Mark A. Thomas</li> <li><a href="https://github.com/mrzool/unix-as-ide">Unix as IDE</a> - Tom Ryder (epub, mobi)</li> <li><a href="http://www.cs.bu.edu/teaching/unix/reference/">UNIX Commands and Concepts</a> - Robert I. Pitts</li> <li><a href="http://web.stanford.edu/class/cs124/kwc-unix-for-poets.pdf">Unix for Poets</a> - Kenneth Ward Church (PDF)</li> <li><a href="https://web.archive.org/web/20210912091852/https://cb.vu/unixtoolbox.xhtml">Unix Toolbox</a> - Colin Barschel <em>(:card_file_box: archived)</em></li> <li><a href="http://www.ee.surrey.ac.uk/Teaching/Unix/">UNIX Tutorial for Beginners</a></li> </ul> <h4>Vulkan</h4> <ul> <li><a href="https://vulkan-tutorial.com">Vulkan Tutorial</a> - Alexander Overvoorde (EPUB, HTML, PDF) (C++)</li> <li><a href="https://github.com/Naitsirc98/Vulkan-Tutorial-Java">Vulkan Tutorial Java</a> - Cristian Herrera et al (Java)</li> <li><a href="https://github.com/bwasty/vulkan-tutorial-rs">Vulkan Tutorial RS</a> - Benjamin Wasty et al. (:construction: <em>in process</em>) (Rust)</li> <li><a href="https://vulkano.rs/guide/introduction">Vulkano</a> - Tomaka et al. (HTML) (Rust)</li> </ul> <h3>Workflow</h3> <ul> <li><a href="https://leanpub.com/declarepeaceonvms/read">Declare Peace on Virtual Machines. A guide to simplifying vm-based development on a Mac</a></li> </ul> <h3>xBase (dBase / Clipper / Harbour)</h3> <ul> <li><a href="https://en.wikibooks.org/wiki/Application_Development_with_Harbour">Application Development with Harbour</a> - Wikibooks</li> <li><a href="https://web.archive.org/web/20190516192814/http://www.ousob.com/ng/clguide/">CA-Clipper 5.2 Norton Guide</a></li> <li><a href="https://en.wikibooks.org/wiki/Clipper_Tutorial%3A_a_Guide_to_Open_Source_Clipper(s)">Clipper Tutorial: a Guide to Open Source Clipper(s)</a> - Wikibooks</li> </ul> <h2>BY SUBJECT</h2> <h3>Cellular Automata</h3> <ul> <li><a href="https://www.wolframscience.com/nksonline/toc.html">A New Kind of Science</a> - Stephen Wolfram</li> </ul> <h3>Competitive Programming</h3> <ul> <li><a href="https://cses.fi/book/book.pdf">Competitive Programmer's Handbook</a> - Antti Laaksonen (PDF)</li> <li><a href="https://cpbook.net/#CP1details">Competitive Programming, 1st Edition</a> - Steven Halim <a href="https://www.comp.nus.edu.sg/~stevenha/myteaching/competitive_programming/cp1.pdf">(PDF)</a></li> <li><a href="https://cpbook.net/#CP2details">Competitive Programming, 2nd Edition</a> - Steven Halim <a href="https://www.comp.nus.edu.sg/~stevenha/myteaching/competitive_programming/cp2.pdf">(PDF)</a></li> <li><a href="http://www.csc.kth.se/~jsannemo/slask/main.pdf">Principles of Algorithmic Problem Solving</a> - Johan Sannemo (PDF)</li> </ul> <h3>Computer Organization and Architecture</h3> <ul> <li><a href="https://www.cse.iitd.ac.in/~srsarangi/archbooksoft.html">Basic Computer Architecture</a> - Smruti R. Sarangi (HTML, PDF, Slides, Videos)</li> <li><a href="https://faculty.etsu.edu/tarnoff/138292">Computer Organization and Design Fundamentals</a> - David Tarnoff (PDF)</li> </ul> <h3>Computer Science</h3> <ul> <li><a href="https://www.cs.cmu.edu/~15110-s13/Wing06-ct.pdf">Computational Thinking</a> - Jeannette Wing, Carnegie-Mellon University (PDF)</li> <li><a href="https://bitbucket.org/chrisbourke/computersciencei/src">Computer Science I - Draft</a> - Dr. Chris Bourke (PDF) (:construction: <em>in process</em>)</li> <li><a href="http://cse.unl.edu/~cbourke/ComputerScienceTwo.pdf">Computer Science II - Draft</a> - Dr. Chris Bourke (PDF) (:construction: <em>in process</em>)</li> <li><a href="https://www.openbookproject.net/books/StudentCSP/">CS Principles: Big Ideas in Programming</a> - Mark Guzdial and Barbara Ericson (HTML)</li> <li><a href="https://leanpub.com/whattolookforinacodereview">What to Look for in a Code Review</a> - Trisha Gee (HTML, PDF, EPUB, Kindle) <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://leanpub.com/firstyearincode">Your First Year in Code</a> - Isaac Lyman (HTML, PDF, EPUB, Kindle) <em>(Leanpub account or valid email requested)</em></li> </ul> <h3>Computer Vision</h3> <ul> <li><a href="http://homepages.inf.ed.ac.uk/rbf/BOOKS/BANDB/bandb.htm">Computer Vision</a> - Dana Ballard, Chris Brown</li> <li><a href="http://szeliski.org/Book/">Computer Vision: Algorithms and Applications</a> - Richard Szeliski</li> <li><a href="http://www.computervisionmodels.com">Computer Vision: Models, Learning, and Inference</a> - Simon J.D. Prince</li> <li><a href="http://programmingcomputervision.com">Programming Computer Vision with Python</a> - Jan Erik Solem</li> </ul> <h3>Misc</h3> <ul> <li><a href="https://ithemes.com/wp-content/uploads/downloads/2012/09/10-keys-to-great-landing-pages-eBook.pdf">10 Keys to Great Landing Pages</a> - iThemes Media (PDF)</li> <li><a href="https://www.oreilly.com/radar/2016-european-software-development-salary-survey/">2016 European Software Development Salary Survey</a></li> <li><a href="https://www.oreilly.com/radar/2016-software-development-salary-survey-report/">2016 Software Development Salary Survey</a></li> <li><a href="https://leanpub.com/97-Things-Every-Programmer-Should-Know-Extended">97 Things Every Programmer Should Know - Extended</a> <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://ocaml-book.com/s/popbook.pdf">A MACHINE MADE THIS BOOK ten sketches of computer science</a> - JOHN WHITINGTON (PDF)</li> <li><a href="https://www.ansible.com/ebooks">Ansible Up &amp; Running (first three chapters)</a> <em>(account required)</em></li> <li><a href="http://solmu.org/pub/help/Asterisk/3nd_Edition_for_Asterisk_1.8">Asterisk™: The Definitive Guide</a></li> <li><a href="http://www.tec-it.com/download/PDF/Barcode_Reference_EN.pdf">Barcode Overview</a> (PDF)</li> <li><a href="http://www.wisdom.weizmann.ac.il/~playbook/">Come, Let's Play: Scenario-Based Programming Using Live Sequence Charts</a></li> <li><a href="http://www.usingcsp.com/cspbook.pdf">Communicating Sequential Processes</a> - Tony Hoare (PDF)</li> <li><a href="http://www.jackkinsella.ie/books/confessions_of_an_unintentional_cto">Confessions of an Unintentional CTO: Lessons in Growing a Web App</a> - Jack Kinsella</li> <li><a href="http://hintjens.com/books">Culture &amp; Empire: Digital Revolution</a> (PDF)</li> <li><a href="http://designwithfontforge.com/en-US/index.html">Design With FontForge</a></li> <li><a href="http://designinginterfaces.com">Designing Interfaces</a> - Jennifer Tidwell</li> <li><a href="http://devdocs.io">DevDocs</a> - Documents for Developers in 1 place</li> <li><a href="http://www.ibm.com/ibm/devops/us/en/resources/dummiesbooks/">DevOps For Dummies (IBM Edition)</a> (PDF)</li> <li><a href="http://www.sp4comm.org">Digital Signal Processing For Communications</a></li> <li><a href="http://www.dspguide.com">Digital Signal Processing For Engineers and Scientists</a></li> <li><a href="https://greenteapress.com/wp/think-dsp">Digital Signal Processing in Python</a> - Allen B. Downey</li> <li><a href="http://www.lulu.com/shop/luis-enr%C3%ADquez-a/dynamic-linked-libraries-paradigms-of-the-gpl-license-in-contemporary-software/ebook/product-21419788.html">"DYNAMIC LINKED LIBRARIES": Paradigms of the GPL license in contemporary software</a> - Luis A. Enríquez</li> <li><a href="https://images.guide">Essential Image Optimization</a> - Addy Osmani</li> <li><a href="https://openmymind.net/FoundationsOfProgramming.pdf">Foundations of Programming</a> - Karl Seguin (PDF)</li> <li><a href="https://basecamp.com/books/getting-real">Getting Real</a> - Basecamp, 37signals (<a href="https://basecamp.com/gettingreal">HTML</a>, <a href="https://basecamp.com/gettingreal/getting-real.pdf">PDF</a>)</li> <li><a href="https://www.syncfusion.com/ebooks/google_maps_api_succinctly">Google Maps API Succinctly</a> - Mark Lewin</li> <li><a href="http://www.lulu.com/shop/ed-johnson/hacknot-essays-on-software-development/ebook/product-17544641.html">Hacknot: Essays on Software Development</a> - Ed Johnson</li> <li><a href="https://andreask.cs.illinois.edu/Teaching/HPCFall2012">High-Performance Scientific Computing</a> (class lectures and slides)</li> <li><a href="https://www.syncfusion.com/ebooks/hololens_succinctly">HoloLens Succinctly</a> - Lars Klint</li> <li><a href="http://www.fastchip.net/howcomputerswork/p1.html">How Computers Work</a> - R. Young</li> <li><a href="http://softwarebyrob.wpengine.netdna-cdn.com/assets/Software_by_Rob%20_How_to_Become_a%20_Programmer_1.0.pdf">How to Become a Programmer</a> (PDF)</li> <li><a href="https://www.digitalocean.com/community/books/how-to-manage-remote-servers-with-ansible-ebook">How To Manage Remote Servers with Ansible</a> - Erika Heidi (PDF, EPUB)</li> <li><a href="http://openbookproject.net/thinkcs/">How to Think Like a Computer Scientist</a> - Peter Wentworth, Jeffrey Elkner, Allen B. Downey, and Chris Meyers</li> <li><a href="http://homepages.inf.ed.ac.uk/rbf/BOOKS/PHILLIPS/">Image Processing in C: Analyzing and Enhancing Digital Images</a> - Dwayne Phillips</li> <li><a href="https://web.archive.org/web/20200731035935/https://florida.theorangegrove.org/og/file/49843a6a-9a9d-4bad-b4d4-d053f9cdf73e/1/InfoTechNetworkedEconomy.pdf">Information Technology and the Networked Economy</a> - Patrick McKeown (PDF) <em>(:card_file_box: archived)</em></li> <li><a href="https://web.corral.tacc.utexas.edu/CompEdu/pdf/isp/EijkhoutIntroSciProgramming-book.pdf">Introduction to Scientific Programming in C++ and Fortran</a> - Victor Eijkhout (PDF)</li> <li><a href="https://www.gitbook.com/book/scemama/irpf90/details">IRPF90 Fortran code generator</a> - Anthony Scemama</li> <li><a href="https://progbook.org">Learn Programming</a> - Antti Salonen</li> <li><a href="https://pine.fm/LearnToProgram/">Learn to Program</a></li> <li><a href="https://blog.openshift.com/learning-30-technologies-in-30-days-a-developer-challenge/">Learning 30 Technologies in 30 Days: A Developer Challenge</a> - Shekhar Gulati</li> <li><a href="http://patterns.dataincubator.org/book/">Linked Data Patterns: A pattern catalogue for modelling, publishing, and consuming Linked Data</a> - Leigh Dodds, Ian Davis</li> <li><a href="http://worrydream.com/#!/MagicInk">Magic Ink: Information Software and The Graphical Interface</a> - Bret Victor</li> <li><a href="https://leanpub.com/mobiledevelopersguide">Mobile Developer's Guide to the Galaxy</a> <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://www.wisdom.weizmann.ac.il/~harel/reactive_systems.html">Modeling Reactive Systems with Statecharts</a></li> <li><a href="https://www.syncfusion.com/ebooks/msix-succinctly">MSIX Succinctly</a> - Matteo Pagani</li> <li><a href="http://www.cs.cornell.edu/home/kleinber/networks-book/">Networks, Crowds, and Markets: Reasoning About a Highly Connected World</a></li> <li><a href="http://win.ua.ac.be/~sdemey/">Object-Oriented Reengineering Patterns</a> - Serge Demeyer, Stéphane Ducasse and Oscar Nierstrasz</li> <li><a href="https://github.com/oreillymedia/open_government">Open Government; Collaboration, Transparency, and Participation in Practice</a> - Daniel Lathrop, Laurel Ruma</li> <li><a href="https://leanpub.com/PDQ">PDQ: Pretty Darn Quick: An Agile, All-Purpose Methodology</a> - Jeff Franz-Lien <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://www.cse.buffalo.edu/~rapaport/Papers/phics.pdf">Philosophy of Computer Science</a> (PDF)</li> <li><a href="http://www.libpng.org/pub/png/book/">PNG: The Definitive Guide</a></li> <li><a href="http://cslibrary.stanford.edu/102/PointersAndMemory.pdf">Pointers And Memory</a> (PDF)</li> <li><a href="http://ocw.mit.edu/resources/res-6-004-principles-of-computer-system-design-an-introduction-spring-2009/online-textbook/part_ii_open_5_0.pdf">Principles of Computer System Design</a> (PDF)</li> <li><a href="https://press.rebus.community/programmingfundamentals/">Programming Fundamentals</a> - Kenneth Leroy Busbee, Dave Braunschweig</li> <li><a href="http://unicodebook.readthedocs.org">Programming with Unicode</a></li> <li><a href="https://www.oreilly.com/ideas/real-world-maintainable-software">Real-World Maintainable Software</a> - Abraham Marin-Perez</li> <li><a href="https://leanpub.com/manualToAutomatedWithSeleniumIDEAndSahi">Record-Playback Test Automation: Sahi &amp; Selenium IDE: Critical Evaluation of Record-Playback Automation Tools</a> - Shashikant Jagtap <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://divakarvi.github.io/bk-spca/spca.html">Scientific Programming and Computer Architecture</a> - Divakar Viswanath</li> <li><a href="http://faculty.washington.edu/stiber/pubs/Signal-Computing/Signal%20Computing.pdf">Signal Computing: Digital Signals in the Software Domain</a> - Michael Stiber, Bilin Zhang Stiber, Eric C. Larson (PDF)</li> <li><a href="https://smallmemory.charlesweir.com/book.html">Small Memory Software</a> - Charles Weir, James Noble (HTML)</li> <li><a href="https://almanac.httparchive.org/static/pdfs/web_almanac_2019_en.pdf">Web Almanac</a> (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/writing_native_mobile_apps_in_a_functional_language_succinctly">Writing Native Mobile Apps in a Functional Language Succinctly</a> - Vassili Kaplan</li> </ul> <h3>Open Source Ecosystem</h3> <ul> <li><a href="http://datajournalismhandbook.org">Data Journalism Handbook</a></li> <li><a href="https://archive.org/details/faif-2.0">Free as in Freedom: Richard Stallman and the free software revolution</a> - Sam Williams (PDF)</li> <li><a href="https://unglue.it/work/136445/">Free for All</a> - Peter Wayner</li> <li><a href="http://shop.fsf.org/product/free-software-free-society-2/">Free Software, Free Society: Selected Essays of Richard M. Stallman</a></li> <li><a href="http://www.oreilly.com/programming/free/getting-started-with-innersource.csp">Getting Started with InnerSource</a> (email address <em>requested</em>, not required)</li> <li><a href="http://public.dhe.ibm.com/software/dw/db2/express-c/wiki/Getting_started_with_open_source_development_p2.pdf">Getting started with Open source development</a> (PDF)</li> <li><a href="https://about.gitlab.com/handbook/">GitLab Handbook</a></li> <li><a href="https://opensource.com/resources/ebook/how-get-started-open-source">How to get started with open source</a> (ePub &amp; ODT)</li> <li><a href="http://dreamsongs.com/IHE/IHE.html">Innovation Happens Elsewhere</a> - Ron Goldman, Richard P. Gabriel</li> <li><a href="https://do1.dr-chuck.net/net-intro/EN_us/net-intro.pdf">Introduction to Networking</a> - Charles Severance (PDF)</li> <li><a href="http://open-advice.org">Open Advice: FOSS: What We Wish We Had Known When We Started</a></li> <li><a href="https://www.oreilly.com/ideas/open-source-in-brazil">Open source in Brazil</a> - Andy Oram</li> <li><a href="http://producingoss.com">Producing Open Source Software</a> - Karl Fogel</li> <li><a href="http://www.aosabook.org/en/index.html">The Architecture of Open Source Applications: Vol. 1: Elegance, Evolution, and a Few Fearless Hacks; Vol. 2: Structure, Scale, and a Few More Feerless Hacks</a></li> <li><a href="http://artofcommunityonline.org/Art_of_Community_Second_Edition.pdf">The Art of Community</a> - Jono Bacon (PDF)</li> <li><a href="http://www.catb.org/esr/writings/cathedral-bazaar/">The Cathedral and the Bazaar</a> - Eric S. Raymond</li> <li><a href="http://futureoftheinternet.org">The Future of the Internet</a> - Jonathan Zittrain</li> <li><a href="http://www.theopensourceway.org/book/">The Open Source Way</a></li> <li><a href="http://cyber.law.harvard.edu/wealth_of_networks/Main_Page">The Wealth of Networks: How Social Production Transforms Markets and Freedom</a> - Yochai Benkler</li> </ul> <h3>Partial Evaluation</h3> <ul> <li><a href="http://www.itu.dk/people/sestoft/pebook/">Partial Evaluation and Automatic Program Generation</a> - (PDF) Jones, Gomard and Sestoft</li> </ul> <h3>Professional Development</h3> <ul> <li><a href="https://www.gitbook.com/book/ccd_school/clean-code-developer-com/details">Clean Code Developer: An initiative for more professionalism in software development</a> (:construction: <em>in process</em>)</li> <li><a href="https://www.red-gate.com/library/confessions-of-an-it-manager">Confessions of an IT Manager</a> - Phil Factor (PDF)</li> <li><a href="https://www.red-gate.com/library/dont-just-roll-the-dice">Don't Just Roll the Dice</a> - Neil Davidson (PDF)</li> <li><a href="https://leanpub.com/dowhatyoulove/read">How to Do What You Love &amp; Earn What You’re Worth as a Programmer</a> - Reginald Braithwaite</li> <li><a href="https://mixmastamyk.bitbucket.io/pro_soft_dev/intro.html">Professional Software Development For Students</a> - Mike G. Miller</li> <li><a href="https://abseil.io/resources/swe-book">Software Engineering at Google</a> - Titus Winters, Tom Manshreck &amp; Hyrum Wright</li> <li><a href="https://softwareconcepts.vercel.app">Software Environment Concepts</a> - Amr Elmohamady (:construction: <em>in process</em>)</li> <li><a href="https://leanpub.com/shippingsoftware/read">What I've Learned From Failure</a> - Reginald Braithwaite</li> </ul> <h3>Regular Expressions</h3> <ul> <li><a href="https://learnbyexample.github.io/learn_js_regexp/">JavaScript RegExp</a> - Sundeep Agarwal</li> <li><a href="https://learnbyexample.github.io/py_regular_expressions/">Python re(gex)?</a> - Sundeep Agarwal</li> <li><a href="https://refrf.shreyasminocha.me">Regular Expressions for Regular Folk</a> - Shreyas Minocha</li> <li><a href="http://www.rexegg.com">RexEgg</a></li> <li><a href="https://learnbyexample.github.io/Ruby_Regexp/">Ruby Regexp</a> - Sundeep Agarwal</li> <li><a href="http://www.codeproject.com/Articles/9099/The-Minute-Regex-Tutorial">The 30 Minute Regex Tutorial</a> - Jim Hollenhorst</li> <li><a href="https://leanpub.com/bastards-regexes">The Bastards Book of Regular Expressions: Finding Patterns in Everyday Text</a> - Dan Nguyen <em>(Leanpub account or valid email requested)</em></li> </ul> <h3>Search Engines</h3> <ul> <li><a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/index.html">Elasticsearch: The Definitive Guide</a> (<a href="https://github.com/elastic/elasticsearch-definitive-guide">fork it on GH</a>)</li> <li><a href="https://ciir.cs.umass.edu/irbook">Search Engines: Information Retrieval in Practice</a> - W. Bruce Croft, Donald Metzler, Trevor Strohman (PDF)</li> <li><a href="https://github.com/hectorcorrea/solr-for-newbies">Solr for newbies workshop (2019)</a> - Hector Correa (<a href="https://github.com/hectorcorrea/solr-for-newbies/blob/master/tutorial.pdf">PDF</a>)</li> </ul> <h3>Security</h3> <ul> <li><a href="http://toc.cryptobook.us">A Graduate Course in Applied Cryptography</a></li> <li><a href="https://www.crypto101.io">Crypto 101 - Crypto for everyone</a></li> <li><a href="https://en.wikibooks.org/wiki/Cryptography">Cryptography</a> - Wikibooks (HTML) (:construction: <em>in process</em>)</li> <li><a href="https://unglue.it/work/141611/">CryptoParty Handbook</a></li> <li><a href="http://cacr.uwaterloo.ca/hac/index.html">Handbook of Applied Cryptography</a></li> <li><a href="https://howhttps.works">How HTTPS works</a> - dnsimple</li> <li><a href="https://github.com/MHM5000/pass">How to deal with Passwords</a></li> <li><a href="http://ptgmedia.pearsoncmg.com/images/0131407333/downloads/0131407333.pdf">Intrusion Detection Systems with Snort</a> (PDF)</li> <li><a href="https://www.feistyduck.com/library/openssl-cookbook/">OpenSSL Cookbook</a></li> <li><a href="https://mobile-security.gitbook.io/mobile-security-testing-guide/">OWASP Mobile Security Testing Guide</a> - Bernhard Mueller et al.</li> <li><a href="https://www.owasp.org/images/1/19/OTGv4.pdf">OWASP Testing Guide 4.0</a> (PDF)</li> <li><a href="http://www.troyhunt.com/2011/12/free-ebook-owasp-top-10-for-net.html">OWASP Top 10 for .NET Developers</a></li> <li><a href="http://www.cl.cam.ac.uk/~rja14/book.html">Security Engineering</a></li> <li><a href="https://web.engr.oregonstate.edu/~rosulekm/crypto/">The Joy of Cryptography (2021)</a> - Mike Rosulek (PDF) (:construction: <em>in process</em>)</li> </ul> <h3>Standards</h3> <ul> <li><a href="http://refspecs.linuxfoundation.org/lsb.shtml">Linux Standard Base</a></li> <li><a href="https://github.com/geoff-codes/posix-standard">UNIX - The POSIX Standard - IEEE Std 1003.1</a></li> </ul> <h3>Theoretical Computer Science</h3> <ul> <li><a href="http://mfleck.cs.illinois.edu/building-blocks/index.html">Building Blocks for Theoretical Computer Science</a> - Margaret M. Fleck</li> <li><a href="http://www.tac.mta.ca/tac/reprints/articles/22/tr22.pdf">Category Theory for Computing Science</a> (PDF)</li> <li><a href="https://github.com/hmemcpy/milewski-ctfp-pdf">Category Theory for Programmers</a> - Bartosz Milewski (PDF)</li> <li><a href="https://textbooks.open.tudelft.nl/textbooks/catalog/book/13">Delftse Foundations of Computation</a> - Stefan Hugtenburgand, Neil Yorke-Smith @ TU Delft Open (PDF)</li> <li><a href="https://www.smashwords.com/books/view/639609">Design of a Programmer</a> - Prakash Hegade (PDF)</li> <li><a href="http://homotopytypetheory.org/book/">Homotopy Type Theory: Univalent Foundations of Mathematics</a> (PDF)</li> <li><a href="http://www.cse.iitd.ernet.in/~suban/CSL102/">Introduction to Computer Science</a> - Prof. Subhashis Banerjee, IIT Delhi</li> <li><a href="http://www.computingbook.org">Introduction to Computing</a></li> <li><a href="http://cglab.ca/~michiel/TheoryOfComputation/">Introduction to Theory of Computation</a> - Anil Maheshwari and Michiel Smid (PDF)</li> <li><a href="http://cs.brown.edu/people/jes/book/">Models of Computation</a> - John E. Savage</li> <li><a href="https://www.cs.cmu.edu/~rwh/pfpl/2nded.pdf">Practical Foundations for Programming Languages, Preview</a> - Robert Harper (PDF)</li> <li><a href="https://web.archive.org/web/20150418034451/http://www.cs.jhu.edu/~scott/pl/book/dist/">Principles of Programming Languages</a> - Scott F. Smith</li> <li><a href="https://papl.cs.brown.edu/2019/">Programming and Programming Languages</a> - Shriram Krishnamurthi</li> <li><a href="http://www.cse.chalmers.se/research/group/logic/book/">Programming in Martin-Löf's Type Theory</a> - Bengt Nordstroem</li> <li><a href="http://cs.brown.edu/~sk/Publications/Books/ProgLangs/">Programming Languages: Application and Interpretation (2nd Edition)</a> - Shriram Krishnamurthi</li> <li><a href="http://people.cs.uchicago.edu/~blume/classes/aut2008/proglang/text/offline.pdf">Programming Languages: Theory and Practice</a> - Robert Harper (PDF)</li> <li><a href="http://www.cs.ru.nl/~herman/onderwijs/semantics2019/wiley.pdf">Semantics with Applications: A Formal Introduction</a> - Hanne Riis Nielson, Flemming Nielson (PDF)</li> <li><a href="https://mitpress.mit.edu/sicp/">Structure and Interpretation of Computer Programs</a></li> <li><a href="http://sarabander.github.io/sicp/html/index.xhtml">Structure and Interpretation of Computer Programs</a></li> <li><a href="http://self.gutenberg.org/wplbn0002828847-the-black-art-of-programming-by-mcilroy-mark.aspx?">The Black Art of Programming</a> - Mark McIlroy</li> <li><a href="https://kilthub.cmu.edu/articles/The_Craft_of_Programming/6610514">The Craft of Programming</a> - John C. Reynolds</li> <li><a href="https://joshua.smcvt.edu/computation">Theory of Computation, Making Connections</a> - Jim Hefferon (PDF)</li> <li><a href="https://greenteapress.com/wp/think-complexity-2e/">Think Complexity</a> - - Allen B. Downey (2nd Edition) (PDF, HTML)</li> </ul> <h3>Web Performance</h3> <ul> <li><a href="http://www.bookofspeed.com">Book of Speed</a> - Stoyan Stefanov</li> <li><a href="http://designingforperformance.com">Designing for Performance</a> - Lara Hogan</li> <li><a href="https://thisisyuu.github.io/ebook">High Performance Accelerated Websites</a> - Anshul (HTML) (:construction: <em>in process</em>)</li> <li><a href="https://hpbn.co">High Performance Browser Networking</a> - Ilya Grigorik</li> <li><a href="http://carlos.bueno.org/optimization/mature-optimization.pdf">Mature Optimization</a> - Carlos Bueno (PDF)</li> </ul> <h3>Web Services</h3> <ul> <li><a href="http://restfulwebapis.org/RESTful_Web_Services.pdf">RESTful Web Services</a> (PDF)</li> </ul> <h3>Workflow</h3> <ul> <li><a href="https://leanpub.com/declarepeaceonvms/read">Declare Peace on Virtual Machines. A guide to simplifying vm-based development on a Mac</a></li> </ul> https://tedneward.github.io/Research/reading/development/dynamic-binary-instrumentation/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/dynamic-binary-instrumentation/index.html <ul> <li>Dynamic Binary Instrumentation Primer <ul> <li><a href="http://deniable.org/reversing/binary-instrumentation">http://deniable.org/reversing/binary-instrumentation</a></li> <li><a href="https://github.com/fdiskyou/DBI">https://github.com/fdiskyou/DBI</a></li> </ul> </li> <li>Cross-ISA Machine Instrumentation using Fast and Scalable Dynamic Binary Translation <ul> <li>Virtual Execution Environments (VEE) 2019</li> <li>Emilio G. Cota, Luca P. Carloni</li> <li><a href="http://www.cs.columbia.edu/~cota/pubs/cota_vee19.pdf">http://www.cs.columbia.edu/~cota/pubs/cota_vee19.pdf</a></li> <li><a href="https://www.sld.cs.columbia.edu/qemu-vee.html">https://www.sld.cs.columbia.edu/qemu-vee.html</a></li> </ul> </li> <li>Optimization of Naïve Dynamic Binary Instrumentation Tools <ul> <li>2011 Ph.D. Dissertation; Reid Kleckner</li> <li><a href="https://dspace.mit.edu/handle/1721.1/76984">https://dspace.mit.edu/handle/1721.1/76984</a></li> </ul> </li> <li>Security Evaluation of Dynamic Binary Instrumentation Engines <ul> <li>DBI Engine Detection Tool and all PoC code <ul> <li><a href="https://github.com/zhechkoz/PwIN">https://github.com/zhechkoz/PwIN</a></li> </ul> </li> <li>2018 Master Thesis; Zhechko Zhechev <ul> <li><a href="https://kirschju.re/static/ma_zhechev_2018.pdf">https://kirschju.re/static/ma_zhechev_2018.pdf</a></li> </ul> </li> <li>REcon Montreal 2018 <ul> <li><a href="https://kirschju.re/static/recon_2018_kirsch_zhechev_pwin.pdf">https://kirschju.re/static/recon_2018_kirsch_zhechev_pwin.pdf</a></li> <li><a href="https://infocondb.org/con/recon/recon-2018/pwning-intel-pin-reconsidering-intel-pin-in-context-of-security">https://infocondb.org/con/recon/recon-2018/pwning-intel-pin-reconsidering-intel-pin-in-context-of-security</a></li> <li><a href="https://recon.cx/2018/montreal/schedule/events/145.html">https://recon.cx/2018/montreal/schedule/events/145.html</a></li> </ul> </li> <li>Pwning Intel piN – Why DBI is unsuitable for security applications <ul> <li>ESORICS 2018</li> <li>Julian Kirsch, Zhechko Zhechev, Bruno Bierbaumer, Thomas Kittel</li> <li><a href="https://link.springer.com/chapter/10.1007/978-3-319-99073-6_18">https://link.springer.com/chapter/10.1007/978-3-319-99073-6_18</a></li> </ul> </li> </ul> </li> <li>SoK: Using Dynamic Binary Instrumentation for Security (And How You May Get Caught Red Handed) <ul> <li>Asia Conference on Computer and Communications Security (AsiaCCS) 2019</li> <li>Daniele Cono D’Elia, Emilio Coppa, Simone Nicchi, Federico Palmaro, Lorenzo Cavallaro</li> <li><a href="https://doi.org/10.1145/3321705.3329819">https://doi.org/10.1145/3321705.3329819</a></li> <li><a href="https://www.diag.uniroma1.it/~delia/papers/asiaccs2019.pdf">https://www.diag.uniroma1.it/~delia/papers/asiaccs2019.pdf</a></li> <li><a href="https://github.com/season-lab/sok-dbi-security/">https://github.com/season-lab/sok-dbi-security/</a></li> </ul> </li> <li>Using Binary Instrumentation for Vulnerability Discovery (or even mitigation!) <ul> <li>EuskalHack 2018; Joxean Koret</li> <li><a href="https://github.com/joxeankoret/membugtool/blob/master/docs/Using%20Binary%20Instrumentation%20for%20Vulnerability%20Discovery%20(or%20even%20mitigation!).pdf">https://github.com/joxeankoret/membugtool/blob/master/docs/Using%20Binary%20Instrumentation%20for%20Vulnerability%20Discovery%20(or%20even%20mitigation!).pdf</a></li> <li><a href="https://docs.google.com/presentation/d/e/2PACX-1vQYw4HJ3kzdmjnxklyBK2nyoMV3Iftx5G6IQmaas7z1cJdP04sX9WsWZmZKqtcTOsqbYukDdUyovhXb/pub">https://docs.google.com/presentation/d/e/2PACX-1vQYw4HJ3kzdmjnxklyBK2nyoMV3Iftx5G6IQmaas7z1cJdP04sX9WsWZmZKqtcTOsqbYukDdUyovhXb/pub</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/essays/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/essays/index.html <ul> <li> <p><a href="https://www.codemotion.com/magazine/dev-life/a-great-programmer-removes-doesnt-add/?ref=dailydev">A Great Programmer Removes, Doesn’t Add</a></p> </li> <li> <p><a href="https://jv-la.com/journal/2022/lessons-earlier-professional-career/">4 Lessons I wish I knew earlier on in my professional career</a></p> </li> <li> <p><a href="https://andrewkelley.me/post/why-we-cant-have-nice-software.html">Why we can't have nice software</a>: We keep improving past perfection.</p> </li> <li> <p><a href="https://arstechnica.com/information-technology/2012/11/how-team-obamas-tech-efficiency-left-romney-it-in-dust/">How Team Obama's tech efficiency left Romney IT in dust</a></p> </li> <li> <p><a href="https://jonathanadams.pro/blog-articles/how-good-was-nasa-at-fortran-in-the-1960s.php">How Good was NASA at FORTRAN in the 1960s?</a></p> </li> <li> <p><a href="https://biology.stackexchange.com/questions/30116/does-dna-have-the-equivalent-of-if-statements-while-loops-or-function-calls-h">"Does DNA have the equivalent of IF-statements, WHILE loops, or function calls? How about GOTO?"</a>: Short answer, sort of:</p> <p>"Molecular biological processes cannot be directly compared to a computer code. It is the underlying logic that is important and not the statement construct itself and these examples should not be taken as absolute analogies. It is also to be noted that DNA is just a set of instructions and not really a fully functional entity (it is functional to some extent). However, even being just a code it is comparable to a HLL code that has to be compiled to execute its functions. See this post too.</p> <p>"It is also important to note that the cell, like many other physical systems, is analog in nature. Therefore, in most situations there is no 0/1 (binary) value of variables. Consider gene expression. If a transcriptional activator is present, the gene will be transcribed. However, if you keep increasing the concentration of the activator, the expression of that gene will increase until it reaches a saturation point. So there is no digital logic here. Having said that, I would add that switching behaviour is possible in biological systems (including gene expression) and is also used in many cases. Certain kinds of regulatory network structures can give rise to such dynamics. Co-operativity with or without positive feedback is one of the mechanisms that can implement switching behaviour."</p> </li> <li> <p><a href="http://doc.cat-v.org/programming/worse_is_better">The Rise of "Worse is Better"</a> by Richard P. Gabriel</p> </li> <li> <p><a href="https://workweave.dev/blog/the-price-of-mandatory-code-reviews">The Price of Mandatory Code Reviews</a></p> </li> </ul> <h2>Steve Yegge's rants</h2> <ul> <li><a href="http://steve-yegge.blogspot.com/2008/10/universal-design-pattern.html">Universal Design Pattern</a>: "Today I thought I'd talk about a neat design pattern that doesn't seem to get much love: the Properties Pattern. In its fullest form it's also sometimes called the Prototype Pattern." Talks about three forms of modeling: <ul> <li>Class modeling: "Class-based OO design is the 800-pound gorilla of domain modeling these days. Its appeal is that it's a natural match for the way we already model things in everyday life. Although the industry loves OO design, it's not especially well liked as an academic topic. This is because OO design has no real mathematical foundation to support it — at least, not until someone comes along and creates a formal model for side effects. The concepts of OOP stem not from mathematics but from fuzzy intuition. This in some sense explains its popularity, and it also explains why OOP has so many subtly different flavors in practice: whether (and how) to support multiple inheritance, static members, method overloading vs. rich signatures, and so on. Industry folks can never quite agree on what OOP is, but we love it all the same."</li> <li>Relational modeling: "Relational database modeling is a bit harder and takes more practice, because its strength stems from its mathematical foundation. Relational modeling can be intuitive, depending on the problem domain, but most people would agree that it is not necessarily so: it takes some real skill to learn how to model arbitrary problems as relational schemas. Object modeling and relational modeling produce very different designs, each with its strengths and weaknesses, and one of the trickiest problems we face in our industry has always been the object-relational mapping (ORM) problem. It's a big deal. Some people may have let you to believe that it's simple, or that it's automatically handled by frameworks such as Rails or Hibernate. Those who know better know just how hard ORM is in real-world production schemas and systems.</li> <li>XML (hierarchical) modeling: "XML provides yet another technique for modeling problems. Usually XML is used to model data, but it can also be used to model code. For instance, XML-based frameworks such as Apache Ant and XSLT offer computational facilities: loops or recursion, conditional expressions and setting variables." (He goes on to say that this is a "mostly solved problem" and he basically loses any respect I have for this essay at that point. Mostly.)</li> </ul> <p>"Two other obvious candidates are Functional modeling (in the sense of Functional Programming, with roots in the lambda calculus) and Prolog-style logical modeling. Both are mature problem-modeling strategies, each with its pros and cons, and each having varying degrees of overlap with other strategies. And there are still other schools, perhaps dozens of them."</p> <p>(TODO: Fill in the rest: Hofstadter, properties modeling, etc)</p> </li> </ul> <h2><a href="https://www.benkuhn.net/progessays/">Essays on programming I think about a lot</a>:</h2> <h3><a href="https://blog.nelhage.com/post/computers-can-be-understood/">Computers can be understood</a>:</h3> <blockquote> <p>I approach software with a deep-seated belief that computers and software systems can be understood. …</p> <p>In some ways, this belief feels radical today. Modern software and hardware systems contain almost unimaginable complexity amongst many distinct layers, each building atop each other. …</p> <p>In the face of this complexity, it’s easy to assume that there’s just too much to learn, and to adopt the mental shorthand that the systems we work with are best treated as black boxes, not to be understood in any detail.</p> <p>I argue against that approach. You will never understand every detail of the implementation of every level on that stack; but you can understand all of them to some level of abstraction, and any specific layer to essentially any depth necessary for any purpose.</p> </blockquote> <h3><a href="https://mcfunley.com/choose-boring-technology">Choose Boring Technology</a>:</h3> <blockquote> <p>Let’s say every company gets about three innovation tokens. You can spend these however you want, but the supply is fixed for a long while. You might get a few more after you achieve a certain level of stability and maturity, but the general tendency is to overestimate the contents of your wallet. Clearly this model is approximate, but I think it helps.</p> <p>If you choose to write your website in NodeJS, you just spent one of your innovation tokens. If you choose to use MongoDB, you just spent one of your innovation tokens. If you choose to use service discovery tech that’s existed for a year or less, you just spent one of your innovation tokens. If you choose to write your own database, oh god, you’re in trouble.</p> </blockquote> <h3><a href="https://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction">The Wrong Abstraction</a>:</h3> <blockquote> <p>Time passes. A new requirement appears for which the current abstraction is almost perfect. Programmer B gets tasked to implement this requirement. Programmer B feels honor-bound to retain the existing abstraction, but since isn’t exactly the same for every case, they alter the code to take a parameter.... ... Loop until code becomes incomprehensible. You appear in the story about here, and your life takes a dramatic turn for the worse.</p> </blockquote> <h3><a href="https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/">Falsehoods Programmers Believe About Names</a></h3> <h3><a href="https://sockpuppet.org/blog/2015/03/06/the-hiring-post/">The Hiring Post</a>:</h3> <blockquote> <p>Nothing in Alex’s background offered a hint that this would happen. He had Walter White’s resume, but Heisenberg’s aptitude. None of us saw it coming. My name is Thomas Ptacek and I endorse this terrible pun. Alex was the one who nonced.</p> <p>A few years ago, Matasano couldn’t have hired Alex, because we relied on interviews and resumes to hire. Then we made some changes, and became a machine that spotted and recruited people like Alex: line of business .NET developers at insurance companies who pulled Rails core CVEs out of their first hour looking at the code. Sysadmins who hardware-reversed assembly firmware for phone chipsets. Epiphany: the talent is out there, but you can’t find it on a resume.</p> <p>Our field selects engineers using a process that is worse than reading chicken entrails. Like interviews, poultry intestine has little to tell you about whether to hire someone. But they’re a more pleasant eating experience than a lunch interview.</p> </blockquote> <h3><a href="https://blog.pragmaticengineer.com/the-product-minded-engineer/">The Product-Minded Engineer</a>:</h3> <blockquote> <p>Proactive with product ideas/opinions • Interest in the business, user behavior and data on this • Curiosity and a keen interest in "why?" • Strong communicators and great relationships with non-engineers • Offering product/engineering tradeoffs upfront • Pragmatic handling of edge cases • Quick product validation cycles • End-to-end product feature ownership • Strong product instincts through repeated cycles of learning</p> </blockquote> <h3><a href="https://programmingisterrible.com/post/139222674273/write-code-that-is-easy-to-delete-not-easy-to">Write code that is easy to delete, not easy to extend</a>:</h3> <blockquote> <p>If we see ‘lines of code’ as ‘lines spent’, then when we delete lines of code, we are lowering the cost of maintenance. Instead of building re-usable software, we should try to build disposable software.</p> <p>Business logic is code characterised by a never ending series of edge cases and quick and dirty hacks. This is fine. I am ok with this. Other styles like ‘game code’, or ‘founder code’ are the same thing: cutting corners to save a considerable amount of time.</p> <p>The reason? Sometimes it’s easier to delete one big mistake than try to delete 18 smaller interleaved mistakes. A lot of programming is exploratory, and it’s quicker to get it wrong a few times and iterate than think to get it right first time.</p> </blockquote> <h3><a href="https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/">The Law of Leaky Abstractions</a>:</h3> <blockquote> <p>Back to TCP. Earlier for the sake of simplicity I told a little fib, and some of you have steam coming out of your ears by now because this fib is driving you crazy. I said that TCP guarantees that your message will arrive. It doesn’t, actually. If your pet snake has chewed through the network cable leading to your computer, and no IP packets can get through, then TCP can’t do anything about it and your message doesn’t arrive. If you were curt with the system administrators in your company and they punished you by plugging you into an overloaded hub, only some of your IP packets will get through, and TCP will work, but everything will be really slow.</p> <p>This is what I call a leaky abstraction. TCP attempts to provide a complete abstraction of an underlying unreliable network, but sometimes, the network leaks through the abstraction and you feel the things that the abstraction can’t quite protect you from. This is but one example of what I’ve dubbed the Law of Leaky Abstractions:</p> <p><strong>All non-trivial abstractions, to some degree, are leaky.</strong></p> <p>Abstractions fail. Sometimes a little, sometimes a lot. There’s leakage. Things go wrong. It happens all over the place when you have abstractions. Here are some examples.</p> </blockquote> <h3><a href="https://blog.nelhage.com/post/reflections-on-performance/">Reflections on software performance</a>:</h3> <blockquote> <p>It’s probably fairly intuitive that users prefer faster software, and will have a better experience performing a given task if the tools are faster rather than slower.</p> <p>What is perhaps less apparent is that having faster tools changes how users use a tool or perform a task. Users almost always have multiple strategies available to pursue a goal — including deciding to work on something else entirely — and they will choose to use faster tools more and more frequently. Fast tools don’t just allow users to accomplish tasks faster; they allow users to accomplish entirely new types of tasks, in entirely new ways. I’ve seen this phenomenon clearly while working on both Sorbet and Livegrep:</p> </blockquote> <h3>Brandur Leach’s series on using databases to ensure correct edge-case behavior:</h3> <blockquote> <p>I want to convince you that ACID databases are one of the most important tools in existence for ensuring maintainability and data correctness in big production systems. Lets start by digging into each of their namesake guarantees.</p> <p>There’s a surprising symmetry between an HTTP request and a database’s transaction. Just like the transaction, an HTTP request is a transactional unit of work – it’s got a clear beginning, end, and result. The client generally expects a request to execute atomically and will behave as if it will (although that of course varies based on implementation). Here we’ll look at an example service to see how HTTP requests and transactions apply nicely to one another.</p> <p>In APIs <em>idempotency</em> is a powerful concept. An idempotent endpoint is one that can be called any number of times while guaranteeing that the side effects will occur only once. In a messy world where clients and servers that may occasionally crash or have their connections drop partway through a request, it’s a huge help in making systems more robust to failure. Clients that are uncertain whether a request succeeded or failed can simply keep retrying it until they get a definitive response.</p> </blockquote> <ul> <li><a href="https://brandur.org/acid">Building Robust Systems with ACID and Constraints</a></li> <li><a href="https://brandur.org/http-transactions">Using Atomic Transactions to Power an Idempotent API</a></li> <li>[Transactionally Staged Job Drains in Postgres](<a href="https://brandur.org/job-drain">https://brandur.org/job-drain</a></li> <li><a href="https://brandur.org/idempotency-keys">Implementing Stripe-like Idempotency Keys in Postgres</a>:</li> </ul> <h3><a href="http://web.mit.edu/Saltzer/www/publications/endtoend/endtoend.pdf">End-to-End Arguments in System Design</a>:</h3> <blockquote> <p>This paper presents a design principle that helps guide placement of functions among the modules of a distributed computer system. The principle, called the end-to-end argument, suggests that functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level. Examples discussed in the paper include bit error recovery, security using encryption, duplicate message suppression, recovery from system crashes, and delivery acknowledgement. Low level mechanisms to support these functions are justified only as performance enhancements.</p> </blockquote> <h3><a href="https://vimeo.com/36579366">Inventing on Principle</a>:</h3> <blockquote> <p>I’ve spent a lot of time over the years making creative tools, using creative tools, thinking about them a lot, and here’s something I’ve come to believe: Creators need an immediate connection to what they’re creating.</p> </blockquote> https://tedneward.github.io/Research/reading/development/dynamic-binary-translation/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/dynamic-binary-translation/index.html <ul> <li>A Retargetable System-Level DBT Hypervisor <ul> <li>2019 USENIX Annual Technical Conference</li> <li>Tom Spink, Harry Wagstaff, Björn Franke</li> <li><a href="https://www.usenix.org/conference/atc19/presentation/spink">https://www.usenix.org/conference/atc19/presentation/spink</a></li> </ul> </li> <li>Unleashing the Power of Learning: An Enhanced Learning-Based Approach for Dynamic Binary Translation <ul> <li>2019 USENIX Annual Technical Conference</li> <li>Changheng Song, Wenwen Wang, Pen-Chung Yew, Antonia Zhai, Weihua Zhang</li> <li><a href="https://www.usenix.org/conference/atc19/presentation/song">https://www.usenix.org/conference/atc19/presentation/song</a></li> </ul> </li> <li>Acceleration of memory accesses in dynamic binary translation <ul> <li>2018 PhD Dissertation; Antoine Faravelon</li> <li><a href="https://tel.archives-ouvertes.fr/tel-02004524/">https://tel.archives-ouvertes.fr/tel-02004524/</a></li> </ul> </li> <li>Cross-ISA Machine Emulation for Multicores <ul> <li>Code Generation and Optimization (CGO) 2017</li> <li>Emilio G. Cota, Paolo Bonzini, Alex Bennée, Luca P. Carloni</li> <li><a href="https://sld.cs.columbia.edu/pubs/cota_cgo17.pdf">https://sld.cs.columbia.edu/pubs/cota_cgo17.pdf</a></li> </ul> </li> <li>Dynamic Binary Modification: Tools, Techniques, and Applications <ul> <li>2011 Book; Kim Hazelwood</li> <li><a href="http://www.morganclaypool.com/doi/abs/10.2200/S00345ED1V01Y201104CAC015">http://www.morganclaypool.com/doi/abs/10.2200/S00345ED1V01Y201104CAC015</a></li> </ul> </li> <li>Dynamically Translating x86 to LLVM using QEMU <ul> <li>2010 Technical Report; Vitaly Chipounov and George Candea</li> <li><a href="https://infoscience.epfl.ch/record/149975/files/x86-llvm-translator-chipounov_">https://infoscience.epfl.ch/record/149975/files/x86-llvm-translator-chipounov_</a></li> </ul> </li> <li>Efficient and Retargetable SIMD Translation in a Dynamic Binary Translator <ul> <li>Software: Practice and Experience, 48(6) 2018</li> <li>Sheng-Yu Fu, Ding-Yong Hong, Yu-Ping Liu, Jan-Jan Wu, Wei-Chung Hsu</li> <li><a href="https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.2573?af=R">https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.2573?af=R</a></li> </ul> </li> <li>Hermes: A fast cross-ISA binary translator with post-optimization <ul> <li>Code Generation and Optimization (CGO) 2015</li> <li>Xiaochun Zhang, Qi Guo, Yunji Chen, Tianshi Chen, Weiwu Hu</li> <li><a href="http://ieeexplore.ieee.org/document/7054204/">http://ieeexplore.ieee.org/document/7054204/</a></li> <li><a href="https://dl.acm.org/citation.cfm?id=2738631">https://dl.acm.org/citation.cfm?id=2738631</a></li> </ul> </li> <li>Improving Dynamically-Generated Code Performance on Dynamic Binary Translators <ul> <li>Virtual Execution Environments VEE 2018</li> <li>Wenwen Wang, Jiacheng Wu, Xiaoli Gong, Tao Li, Pen-Chung Yew</li> <li><a href="https://dl.acm.org/citation.cfm?id=3186413">https://dl.acm.org/citation.cfm?id=3186413</a></li> </ul> </li> <li>Improving SIMD Parallelism via Dynamic Binary Translation <ul> <li>Ding-Yong Hong, Yu-Ping Liu, Sheng-Yu Fu, Jan-Jan Wu, Wei-Chung Hsu</li> <li>ACM Transactions on Embedded Computing Systems (TECS) 17(3) 2018</li> <li><a href="https://www.researchgate.net/publication/323149934_Improving_SIMD_Parallelism_via_Dynamic_Binary_Translation">https://www.researchgate.net/publication/323149934_Improving_SIMD_Parallelism_via_Dynamic_Binary_Translation</a></li> <li><a href="https://www.researchgate.net/publication/323865399_Improving_Dynamically-Generated_Code_Performance_on_Dynamic_Binary_Translators">https://www.researchgate.net/publication/323865399_Improving_Dynamically-Generated_Code_Performance_on_Dynamic_Binary_Translators</a></li> </ul> </li> <li>libdetox: A Framework for Online Program Transformation <ul> <li>Forming an Ecosystem Around Software Transformation FEAST 2016</li> <li>Mathias Payer</li> <li><a href="https://nebelwelt.net/publications/files/16FEAST.pdf">https://nebelwelt.net/publications/files/16FEAST.pdf</a></li> <li><a href="https://github.com/HexHive/libdetox">https://github.com/HexHive/libdetox</a></li> </ul> </li> <li>Optimizing Binary Translation of Dynamically Generated Code <ul> <li>Code Generation and Optimization (CGO) 2015</li> <li>Byron Hawkins, Brian Demsky, Derek Bruening, Qin Zhao</li> <li><a href="https://research.google.com/pubs/archive/46470.pdf">https://research.google.com/pubs/archive/46470.pdf</a></li> <li><a href="http://demsky.eecs.uci.edu/~bhawkins/papers/cgo-2015.slides.pdf">http://demsky.eecs.uci.edu/~bhawkins/papers/cgo-2015.slides.pdf</a></li> </ul> </li> <li>Optimizing indirect branches in a system-level dynamic binary translator <ul> <li>SYSTOR 2012</li> <li>Toshihiko Koju, Xin Tong, Ali Ijaz Sheikh, Moriyoshi Ohara, Toshio Nakatani</li> <li><a href="http://doi.acm.org/10.1145/2367589.2367599">http://doi.acm.org/10.1145/2367589.2367599</a></li> </ul> </li> <li>Processor-Tracing Guided Region Formation in Dynamic Binary Translation <ul> <li>ACM Transactions on Architecture and Code Optimization, Vol. 15, No. 4, Article 52, 2018</li> <li>Ding-Yong Hong, Jan-Jan Wu, Yu-Ping Liu, Sheng-Yu Fu, Wei-Chung Hsu</li> <li><a href="http://dl.acm.org/citation.cfm?id=3281664">http://dl.acm.org/citation.cfm?id=3281664</a></li> </ul> </li> <li>Requirements for Fast Binary Translation <ul> <li>Mathias Payer, Thomas R. Gross</li> <li>AMAS-BT: Workshop on Architectural and Microarchitectural Support for Binary Translation 2009</li> <li><a href="https://nebelwelt.net/publications/files/09AMAS-BT.pdf">https://nebelwelt.net/publications/files/09AMAS-BT.pdf</a></li> <li><a href="https://nebelwelt.net/publications/files/09AMAS-BT-presentation.pdf">https://nebelwelt.net/publications/files/09AMAS-BT-presentation.pdf</a></li> </ul> </li> <li>rv8: a high performance RISC-V to x86 binary translator <ul> <li>Workshop on Computer Architecture Research with RISC-V (CARRV) 2017</li> <li>Michael Clark and Bruce Hoult</li> <li><a href="https://anarch128.org/~mclark/rv8-carrv.pdf">https://anarch128.org/~mclark/rv8-carrv.pdf</a></li> <li><a href="https://carrv.github.io/papers/clark-rv8-carrv2017.pdf">https://carrv.github.io/papers/clark-rv8-carrv2017.pdf</a></li> <li><a href="https://rv8.io/">https://rv8.io/</a></li> <li><a href="https://github.com/rv8-io/rv8">https://github.com/rv8-io/rv8</a></li> </ul> </li> <li>Scalable Emulation of Heterogeneous Systems <ul> <li>2019 Ph.D. Dissertation; Luca Carloni</li> <li><a href="https://academiccommons.columbia.edu/doi/10.7916/d8-a78j-z392">https://academiccommons.columbia.edu/doi/10.7916/d8-a78j-z392</a></li> <li>"in this dissertation we first present a novel machine emulator design based on dynamic binary translation that makes the following improvements over the state of the art: it scales on multicore hosts while remaining memory efficient, correctly handles cross-ISA differences in atomic instruction semantics, leverages the host floating point (FP) unit to speed up FP emulation without sacrificing correctness, and can be efficiently instrumented to---among other possible uses---drive the execution of a full-system, cross-ISA simulator with support for accelerators."</li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/executables/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/executables/index.html <p>Executable files, debugging data, object code, shared libraries - file formats information, specifications, software - with relevance to compiler toolchains, debuggers, and general program analysis.</p> <p>Organization: Preference given to the most specific category; e.g., if an article discusses DLL-specific information, then it belongs to the DLL section (in preference to the more general PE).</p> <ul> <li>Comparison of executable file formats - <a href="https://en.wikipedia.org/wiki/Comparison_of_executable_file_formats">https://en.wikipedia.org/wiki/Comparison_of_executable_file_formats</a></li> <li>Executable and object file formats - <a href="https://en.wikipedia.org/wiki/Template:Executables">https://en.wikipedia.org/wiki/Template:Executables</a></li> <li>From Hack to Elaborate Technique—A Survey on Binary Rewriting <ul> <li>ACM Computing Surveys 52(3) 2019</li> <li>Matthias Wenzl, Georg Merzdovnik, Johanna Ullrich, and Edgar Weippl</li> <li><a href="https://dl.acm.org/citation.cfm?doid=3341324.3316415">https://dl.acm.org/citation.cfm?doid=3341324.3316415</a></li> <li><a href="https://publications.sba-research.org/publications/201906%20-%20GMerzdovnik%20-%20From%20hack%20to%20elaborate%20technique.pdf">https://publications.sba-research.org/publications/201906%20-%20GMerzdovnik%20-%20From%20hack%20to%20elaborate%20technique.pdf</a></li> </ul> </li> <li>Linux Foundation Referenced Specifications: ABI, ELF, DWARF - <a href="https://refspecs.linuxfoundation.org/">https://refspecs.linuxfoundation.org/</a></li> <li>Type Inference on Executables <ul> <li>ACM Computing Surveys, Vol. 48, No. 4, Article 65, 2016</li> <li>Juan Caballero and Zhiqiang Lin</li> <li><a href="https://www.utdallas.edu/~zxl111930/file/CSUR16.pdf">https://www.utdallas.edu/~zxl111930/file/CSUR16.pdf</a></li> <li><a href="http://web.cse.ohio-state.edu/~lin.3021/file/CSUR16.pdf">http://web.cse.ohio-state.edu/~lin.3021/file/CSUR16.pdf</a></li> </ul> </li> <li>Visual Documentation - File formats - Executables: ELF, Mach-O, PE (SVG and PDF available) <ul> <li><a href="https://github.com/corkami/pics/blob/master/binary/README.md#executables">https://github.com/corkami/pics/blob/master/binary/README.md#executables</a></li> <li>elf101: <a href="https://github.com/corkami/pics/tree/master/binary/elf101">https://github.com/corkami/pics/tree/master/binary/elf101</a></li> <li>macho101: <a href="https://github.com/corkami/pics/tree/master/binary/macho101">https://github.com/corkami/pics/tree/master/binary/macho101</a></li> <li>pe101: <a href="https://github.com/corkami/pics/tree/master/binary/pe101">https://github.com/corkami/pics/tree/master/binary/pe101</a></li> <li>pe102: <a href="https://github.com/corkami/pics/tree/master/binary/pe102">https://github.com/corkami/pics/tree/master/binary/pe102</a></li> </ul> </li> </ul> <h2>Talks</h2> <ul> <li>Reverse Engineering Binaries <ul> <li>DevConf.CZ 2019; Divya Basant Kumar</li> <li><a href="https://www.youtube.com/watch?v=TZVBK5hu0sk">https://www.youtube.com/watch?v=TZVBK5hu0sk</a></li> </ul> </li> <li>My Little Object File: How Linkers Implement C++ <ul> <li>CppCon 2017; Michael Spencer</li> <li>(ELF, MachO, COFF)</li> <li><a href="https://www.youtube.com/watch?v=a5L66zguFe4">https://www.youtube.com/watch?v=a5L66zguFe4</a></li> </ul> </li> <li>The Life of Binaries <ul> <li><a href="http://www.opensecuritytraining.info/LifeOfBinaries.html">http://www.opensecuritytraining.info/LifeOfBinaries.html</a></li> <li><a href="https://www.youtube.com/playlist?list=PLUFkSN0XLZ-n_Na6jwqopTt1Ki57vMIc3">https://www.youtube.com/playlist?list=PLUFkSN0XLZ-n_Na6jwqopTt1Ki57vMIc3</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/designing-data-intensive-apps/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/designing-data-intensive-apps/index.html <p><em>(by Martin Kleppmann)</em></p> <h3>Chapter 1: Reliable, Scalable, and Maintainable Applications</h3> <ul> <li>Many applications are data-intensive and not compute intensive, meaning the biggest problems are usually the amount of data, the complexity of data, and the speed at which it is changing.</li> </ul> <h4>Thinking About Data Systems</h4> <ul> <li>No single tool can meet all data processing and storage needs. Instead work is broken down into tasks that can be performed efficiently on a single tool, and application code stitches those tools together.</li> <li>When you create an API in front of several tools, you have created a new, special-purpose data system from smaller, general-purpose components.</li> <li>This book focuses on three main concerns: reliability, scalability, and maintainability.</li> </ul> <h4>Reliability</h4> <ul> <li>Reliability means the system continues to work correctly, even when things go wrong.</li> <li>Things that can go wrong are called <em>faults</em>, and systems that anticipate faults and can cope with them are called <em>fault-tolerant</em> or <em>resilient</em>.</li> <li>A <em>fault</em> is when one component of the system deviates from its spec, whereas a <em>failure</em> is when the system as a whole stops providing the required service to the user.</li> <li>You cannot reduce the possibility of a fault to zero, and so you must design fault-tolerance mechanisms that prevent faults from causing failures.</li> </ul> <h5>Hardware Faults</h5> <ul> <li>Hard disks have a MTTF of 10 to 50 years, and so on a storage cluster with 10,000 disks, we should expect on average one disk to die per day.</li> </ul> <h5>Software Errors</h5> <ul> <li>Systemic errors, such as leap second bugs or cascading failures, are correlated across nodes and so tend to cause more system failures than uncorrelated hardware faults.</li> </ul> <h5>Human Errors</h5> <ul> <li>Allow quick recovery from human errors, such as making it fast to rollback configuration changes, and to roll out code gradually.</li> <li>Set up detailed and clear monitoring, such as performance metrics and error rates.</li> </ul> <h4>Scalability</h4> <ul> <li>Scalability is the term we use to describe a system's ability to cope with increased load.</li> </ul> <h5>Describing Load</h5> <ul> <li>Load can be described with a few numbers that we call <em>load parameters</em>. The best choice of parameters depends on the architecture of your system.</li> <li>For Twitter, the distribution of followers per user is a key load parameter for describing scalability, as it determines the fan-out load.</li> </ul> <h5>Describing Performance</h5> <ul> <li>You can look at what happens when load increases in two ways:</li> <li>When you increase a load parameter and keep system resources fixed, how is the performance of your system affected?</li> <li>When you increase a load parameter, how much do you need to increase the resources if you want to keep performance unchanged?</li> <li>Response time what the client sees, including the time to process the request (the service time) and network delays and queuing delays.</li> <li>Latency is the duration that a request is waiting to be handled – during which it is <em>latent</em>, awaiting service.</li> <li>The mean is not a good metrics to describe the "typical" response time, as it doesn't tell you how many users experienced that delay. Prefer percentiles instead.</li> <li>High percentiles of response times, or tail latencies, are important because they directly affect users' experience of the service.</li> <li>Reducing response times at high percentiles is difficult because they are easily affected by random events outside your control, and the benefits are diminishing.</li> <li>When artificially generating load to test scalability, the client must send requests independently of the response time, or else it will artificially keep the queues shorter than in reality, thereby skewing the measurements.</li> <li>Even if you call multiple backend services in parallel, the end-user request must wait for the slowest of the parallel calls to complete.</li> <li>Averaging percentiles, such as when combining data from several machines, is mathematically meaningless. The right way of aggregating response time data is to add the histograms.</li> </ul> <h5>Approaches for Coping with Load</h5> <ul> <li>Scaling up, or vertical scaling, is moving to a more powerful machine. Scaling out, or horizontal scaling, distributes the load across multiple smaller machines.</li> <li>An architecture that scales well for a particular application is built around assumptions of operations will be common and which will be rare&nbsp;– the load parameters.</li> <li>In an early-stage startup it's more important to be able to iterate quickly on product features than to scale beyond some hypothetical future load.</li> </ul> <h4>Maintainability</h4> <ul> <li>The majority of the cost of software is not in its initial development, but its ongoing maintenance.</li> <li>Three important design principles for maintainability are:</li> <li>Operability: Make it easy for the operations teams to keep the system running smoothly.</li> <li>Simplicity: Make it easy for new engineers to understand the system by removing complexity.</li> <li>Evolvability (or extensibility): Make it easy for engineers to adapt the system to unanticipated use cases as requirements change.</li> </ul> <h5>Operability: Making Life Easy for Operations</h5> <ul> <li>Good operations can work around the limitations of bad software, but good software cannot run reliably with bad operations.</li> <li>A good operations team works to preserve the organization's knowledge about the system, even as individuals come and go.</li> <li>Data systems with good operability provide good default behavior but allow overriding, and exhibit predictable behavior to minimize surprises.</li> </ul> <h5>Simplicity: Managing Complexity</h5> <ul> <li>Making a system simpler does not necessarily mean reducing its functionality; it can also mean removing <em>accidental complexity</em>.</li> <li>Accidental complexity is complexity that is not inherent in the problem that the software solves, but arises only from the implementation.</li> <li>Good abstractions are one of the best tools for removing accidental complexity by hiding implementation details, but finding good abstractions is very hard.</li> </ul> <h5>Evolvability: Making Change Easy</h5> <ul> <li>Simple and easy-to-understand systems are usually easier to modify than complex ones.</li> </ul> <h4>Summary</h4> <ul> <li>Functional requirements are what an application should do.</li> <li>Nonfunctional requirements are general properties like security, reliability, compliance, scalability, compatibility, and maintainability.</li> </ul> <h3>Chapter 2: Data Models and Query Languages</h3> <ul> <li>Data models are the most important part of developing software, because they affect not only how the software is written, but how we think about the problem that we're solving.</li> <li>Layered software abstractions allow different groups of people to work together effectively.</li> </ul> <h4>Relational Model Versus Document Model</h4> <h5>The Birth of NoSQL</h5> <ul> <li>The term NoSQL has been retroactively reinterpreted as <em>Not Only SQL</em>.</li> </ul> <h5>The Object-Relational Mismatch</h5> <ul> <li>Impedance mismatch is the disconnect between objects in the application code and the database model of tables, rows, and columns.</li> <li>A JSON representation of a document-oriented database has better locality than an equivalent multi-table schema.</li> </ul> <h5>Many-to-One and Many-to-Many Relationships</h5> <ul> <li>In a document-oriented database, if information is duplicated and that information is changed, then all redundant copies need to be updated. Normalization is the key idea behind removing such duplication.</li> </ul> <h5>Are Document Databases Repeating History?</h5> <h6>The relational model</h6> <ul> <li>A key insight of the relational model was that you only need to build a query optimizer once, and then all database clients can benefit from it.</li> </ul> <h6>Comparison to document databases</h6> <ul> <li>For many-to-one and many-to-many relationships, a related item is referenced by a <em>foreign key</em> in the relational model, and a <em>document reference</em> in the document model.</li> </ul> <h5>Relational Versus Document Databases Today</h5> <ul> <li>The document data model offers greater schema flexibility, better performance from locality, and for some applications it is closer to the data structures used by the application.</li> <li>The relational model offers better support for joins, and many-to-one and many-to-many relationships.</li> </ul> <h6>Which data model leads to simpler application code?</h6> <ul> <li>If your application data has a document-like structure (i.e. a tree of one-to-many relationships, where typically the entire tree is loaded at once), then a document model is likely a good idea.</li> <li>But if your application uses many-to-many relationships, the document model becomes less appealing.</li> </ul> <h6>Schema flexibility in the document model</h6> <ul> <li>Document databases are sometimes called <em>schemaless</em>, but that's misleading – there is an implicit schema assumed by the application, but not enforced by the database.</li> <li>A more accurate term is <em>schema-on-read</em> in contrast with <em>schema-on-write</em>.</li> <li>Schema-on-read is similar to dynamic (runtime) type checking in programming languages, whereas schema-on-write is similar to static (compile-type) type checking.</li> <li>In cases where all records are expected to have the same structure, schemas are a useful mechanism for documenting and enforcing a data structure.</li> </ul> <h6>Data locality for queries</h6> <ul> <li>The locality advantage only applies if you need large parts of the document at the same time.</li> </ul> <h4>Query Languages for Data</h4> <ul> <li>An imperative language tells the computer to perform certain operations in a certain order.</li> <li>In a declarative query language like SQL or relational algebra, you specify the pattern of the data you want, but not <em>how</em> to achieve that goal.</li> <li>A declarative query language also hides implementation details of the database engine, allowing it to add performance improvements without requiring changes to queries.</li> <li>Declarative languages are more amenable to parallel execution because they specify the pattern of the results, but not the algorithm used to determine them.</li> </ul> <h5>MapReduce Querying</h5> <ul> <li>MapReduce is based on a <code>map</code> (or <code>collect</code>) and <code>reduce</code> (or <code>fold</code> or <code>inject</code>) functions that exist in many functional languages.</li> <li>The <code>map</code> and <code>reduce</code> functions must be pure, thereby allowing the database to run them anywhere, and to rerun them on failure.</li> </ul> <h4>Graph-Like Data Models</h4> <h5>Property Graphs</h5> <ul> <li>A property graph model, each vertex and edge has a collection of properties (i.e. key-value pairs), and each edge has a label to describe the relationship between its vertices.</li> <li>To get the set of incoming and outgoing edges for a vertex, you can query the <code>edges</code> table by <code>head_vertex</code> or <code>tail_vertex</code> respectively.</li> </ul> <h5>Graph Queries in SQL</h5> <ul> <li>In a relational database, you usually know in advance which joins you need in your query.</li> <li>In a graph query, you may need to traverse a variable number of edges before you find the vertex you're looking for –&nbsp;i.e. the number of joins is not fixed in advance.</li> </ul> <h5>Triple-Stores and SPARQL</h5> <ul> <li>In a triple-store, all information is stored in the form of very simple three-part statements: (<em>subject</em>, <em>predicate</em>, <em>object</em>).</li> <li>The object is one of two things:</li> <li>A value in a primitive data type, and so the predicate and object are equivalent to the key and value of a property on the subject vertex. (E.g. (<em>lucy</em>, <em>age</em>, <em>33</em>) is like a vertex <code>lucy</code> with properties <code>{"age": 33}</code>.)</li> <li>Another vertex in the graph, and so the predicate is an edge, the subject is the tail vertex, and the object is the head vertex. (E.g. (<em>alice</em>, <em>married_to</em>, <em>bob</em>) has an edge <em>married_to</em> between <em>alice</em> and <em>bob</em>.)</li> </ul> <h5>The Foundation: Datalog</h5> <ul> <li>Datalog generalizes the triple-store model, writing the triple as <em>predicate(subject, object)</em>.</li> <li>A Datalog rule applies if the system can find a match for <em>all</em> predicates on the right side of the <code>:-</code> operator.</li> <li>When a rule applies, it's as though the left side of the <code>:-</code> was added to the database, with variables replaced by the values they matched.</li> </ul> <h3>Chapter 3: Storage and Retrieval</h3> <ul> <li>There is a big difference between storage engines optimized for transactional workloads and those optimized for analytics.</li> </ul> <h4>Data Structures That Power Your Database</h4> <ul> <li><em>Log</em> is defined as an append-only sequence of records. It doesn't have to be human-readable, and instead might be binary.</li> <li>Well-chosen indexes speed up read queries, but every index slows down writes, because the index also needs to be updated every time data is written.</li> </ul> <h5>Hash Indexes</h5> <ul> <li>If new and updated values are blindly written to a log, that log can be divided into segments. Compacting a segment keeps only the most recent value for each key, discarding the older ones.</li> <li>Since compaction makes segments much smaller (assuming a key is updated several times in one segment), compaction can also merge several segments together.</li> <li>A background thread can merge and compact segments. While it happens, the database can still serve reads from old segment files, and write requests to the latest segment file.</li> <li>A <em>tombstone</em> marks a deleted key. When segments are merged, the tombstone tells the merging process to discard all previous values for the key.</li> <li>Data file segments are append-only and otherwise immutable, and so they can be read concurrently by multiple threads.</li> <li>Append-only design is advantageous in several ways:</li> <li>Appending and segment merging are sequential write operations, which are generally much faster than random writes.</li> <li>Crash recovery are much simpler if segment files are append-only or immutable, e.g. you avoid crashing while overwriting a value and leaving old and new data spliced together.</li> <li>Merging old segments avoids the problem of data files getting fragmented over time.</li> </ul> <h5>SSTables and LSM-Trees</h5> <ul> <li>Segment files sorted by key is a format called <em>Sorted String Table</em>, or <em>SSTable</em> for short.</li> <li>This has several advantages over log segments with hash indexes:</li> <li>Merging segments is efficient even if the files exceed available memory, by using an algorithm similar to mergesort.</li> <li>Your in-memory index of keys to file offsets can be sparse, and so the total size of your keys can exceed available memory.</li> <li>If your in-memory index of keys is sparse, reads may scan over several key-value pairs on disk anyway, which can be compressed. This reduces disk space and I/O bandwidth use.</li> </ul> <h6>Constructing and maintaining SSTables</h6> <ul> <li>Writes update a <em>memtable</em>, or an in-memory balanced tree. When its size exceeds some threshold, it can be written to disk as an SSTable file.</li> <li>Read requests first query the memtable and then query on-disk segments in order from newest to oldest.</li> <li>We can append every update to a log before updating the memtable. The log is not in sorted order, but can be used to restore the memtable after a crash.</li> </ul> <h6>Making an LSM-tree out of SSTables</h6> <ul> <li>Storage engines based on the principle of merging and compacting sorted files are called LSM (Log-Structured Merge) storage engines.</li> </ul> <h6>Performance optimizations</h6> <ul> <li>An LSM storage engine can use bloom filters to avoid querying the memtable and all segments for a key that does not exist.</li> <li>Since data is stored in sorted order, you can efficiently perform range queries, and because disk writes are essential the LSM-tree can support remarkably high write throughput.</li> </ul> <h5>B-Trees</h5> <ul> <li>B-trees remain the standard index implementation in almost all relational databases, and many non-relational databases use them too.</li> <li>B-trees break down the database into fixed-size <em>blocks</em> or <em>pages</em> (usually 4KB) and read or write one page at a time. This maps closely to the underlying disk hardware.</li> <li>The number of references to child pages in one page of the B-tree is called its <em>branching factor</em>, and is typically several hundred.</li> </ul> <h6>Making B-trees reliable</h6> <ul> <li>To make the database resilient to crashes, many B-tree implementations also persist to an append-only log called the <em>write-ahead log</em> (WAL, or <em>redo log</em>).</li> <li>When the database comes back up after a crash, the write-ahead log is used to restore the B-tree back to a consistent state.</li> </ul> <h6>B-tree optimizations</h6> <ul> <li>In pages in the interior of a tree, we do not need to store full keys, but only enough information to act as boundaries between key ranges.</li> <li>B-trees try to ensure leaf pages are in sequential order on disk. This is easier for LSM-trees, which rewrite large segments of the storage in one go during merging.</li> </ul> <h5>Comparing B-Trees and LSM-Trees</h5> <ul> <li>LSM-trees are typically faster for writes, whereas B-trees are thought to be faster for reads.</li> </ul> <h6>Advantages of LSM-trees</h6> <ul> <li><em>Write amplification</em> is where one write results in multiple writes over the database's lifetime, such as when compacting and merging SSTables.</li> <li>Write amplification is of concern with SSDs, which can only overwrite blocks a limited number of times before wearing out.</li> <li>LSM-trees typically sustain higher write throughput than B-trees because they sometimes have lower write-amplification, and because they sequentially write data.</li> </ul> <h6>Downsides of LSM-trees</h6> <ul> <li>The bigger the database gets, the more disk bandwidth is required for compaction. But this bandwidth is limited and can increase response times at higher percentiles.</li> <li>Relational databases implement transaction isolation using locks on ranges of keys. Because each key exists at exactly one place in a B-tree index, those locks can be attached directly to the tree.</li> </ul> <h5>Other Indexing Structures</h5> <h6>Storing values within the index</h6> <ul> <li>With secondary indexes, each index references a location in the <em>heap file</em> that belongs to the corresponding row.</li> <li>A <em>clustered index</em> stores the indexed row directly within an index, to avoid the performance penalty upon reads of hopping from the index to the heap file.</li> <li>A compromise between a clustered index and a non-clustered index is a <em>covering index</em>, which stores <em>some</em> of the table's columns within the index (the index <em>covers</em> the query).</li> </ul> <h6>Multi-column indexes</h6> <ul> <li>Multi-dimensional indexes allow querying several columns at once, which is important for geospatial data. But more commonly indexes like R-trees are used.</li> </ul> <h6>Full-text search and fuzzy indexes</h6> <ul> <li>In Lucene, the in-memory index is a finite state automaton over the characters in the keys, similar to a trie.</li> <li>This automaton can be transformed into a <em>Levenshtein automaton</em>, which supports efficient search for words within a given edit distance.</li> </ul> <h6>Keeping everything in memory</h6> <ul> <li>In-memory databases are faster because they avoid the overheads of encoding in-memory data structures in a form that can be written to disk.</li> <li>In-memory databases are not faster because they don't need to read from disk – the operating system caches recently used disk blocks in memory anyway.</li> </ul> <h4>Transaction Processing or Analytics?</h4> <ul> <li><em>Transaction processing</em> just means allowing clients to make low-latency reads and writes, as opposed to <em>batch-processing</em> jobs which only run periodically.</li> <li>OLTP, or <em>online transaction processing</em>, is the access pattern for "interactive applications" driven by the user's input.</li> <li>OLAP, or <em>online analytic processing</em>, scans over a huge number of records, reading only a few columns per record, and computes aggregates (e.g. sum, count, or average).</li> <li>The separate database on which OLAP queries are run is called the <em>data warehouse</em>.</li> </ul> <h5>Data Warehousing</h5> <ul> <li><em>Extract-Transform-Load</em> or ETL is the process of extracting data from the OLTP database, transforming it into an analysis-friendly schema, and loading it into the data warehouse.</li> </ul> <h6>The divergence between OLTP databases and data warehouses</h6> <ul> <li>OLTP and OLAP are increasingly becoming two separate storage and query engines, which happen to be accessible through a common SQL interface.</li> </ul> <h5>Stars and Snowflakes: Schemas for Analytics</h5> <ul> <li>Many data warehouses are used in a formulaic style known as <em>star schema</em> (also known as <em>dimensional modeling</em>).</li> <li>The center of the schema is a <em>fact table</em> where each row represents an event that occurred at a particular time.</li> <li>Columns in the fact table are attributes or foreign keys into <em>dimension tables</em>, which represent the <em>who</em>, <em>what</em>, <em>where</em>, <em>when</em>, <em>why</em>, and <em>how</em> of the event.</li> <li>The name <em>star schema</em> comes from visualizing the fact table in the middle, connected to its surrounding dimension tables like rays in a star.</li> <li>A variation of this is the <em>snowflake schema</em>, where dimensions are further broken down into subdimensions.</li> <li>In a typical data warehouse, tables are often very wide: fact tables have over 100 columns, and sometimes several hundred.</li> </ul> <h4>Column-Oriented Storage</h4> <ul> <li>Although fact tables are often over 100 columns wide, a typical data warehouse query only accesses 4 or 5 of them at one time.</li> <li><em>Column-oriented storage</em> stores all the values from each column together, instead of all the values from each row together.</li> <li>The column-oriented storage layout relies on each column file containing the rows in the same order.</li> </ul> <h5>Column Compression</h5> <ul> <li>We can convert a column with <em>n</em> distinct values into n separate bitmaps, with one bitmap for each distinct value, and one bit for each row.</li> <li>If <em>n</em> is large, then the bitmaps will be sparse, and the bitmaps can additionally be run-length encoded to make them very compact.</li> <li>An <code>IN</code> SQL query can be implemented by a bitwise <em>OR</em> of bitmaps, while an <code>AND</code> SQL query can be implemented by a bitwise <em>AND</em> of bitmaps.</li> </ul> <h6>Memory bandwidth and vectorized processing</h6> <ul> <li><em>Vectorized processing</em> is where operators like bitwise <em>AND</em> and <em>OR</em> operate on chunks of compressed column data directly.</li> </ul> <h5>Sort Order in Column Storage</h5> <ul> <li>In column store, the ordering of rows is irrelevant, and so inserting a new row requires simply appending to each of the column files.</li> <li>If after choosing an order, the primary sort column does not have many distinct values, then a simple run-length encoding can compress it efficiently.</li> <li>Run-length encoding compresses the first sort key best, as subsequent sort keys will not have such long runs of repeated values.</li> </ul> <h6>Several different sort orders</h6> <ul> <li>In a column store, there normally aren't pointers to data elsewhere, only columns containing values.</li> </ul> <h5>Writing to Column-Oriented Storage</h5> <ul> <li>Unlike B-trees, which use an update-in-place approach, LSM-trees allow inserting a row with compressed columns without rewriting all the column files.</li> <li>When using an LSM-tree, it doesn't matter whether the in-memory store is row-oriented or column-oriented.</li> </ul> <h5>Aggregation: Data Cubes and Materialized Views</h5> <ul> <li>A materialized view is an actual copy of the query results written to disk, while a virtual view is just a shortcut for writing queries.</li> <li>A <em>data cube</em> or <em>OLAP cube</em> is a materialized view. It is a grid of aggregates grouped by different dimensions.</li> <li>The advantage of a materialized data cube is that certain queries become very fast because they have effectively been precomputed.</li> <li>The disadvantage is that a data cube doesn't have the same flexibility as querying the raw data.</li> </ul> <h4>Summary</h4> <ul> <li>Analytic workloads require sequentially scanning across a large number of rows, and so indexes are much less relevant.</li> <li>Instead it is more important to encode data compactly, in order to minimize the amount of data that the query needs to read from disk.</li> </ul> <h3>Chapter 4: Encoding and Evolution</h3> <ul> <li>Backward compatibility means newer code can read data written by older code. Forward compatibility means older code can read data written by newer code.</li> <li>Forward compatibility is trickier because it requires older code to ignore additions made by a newer version of the code.</li> </ul> <h4>Formats for Encoding Data</h4> <h5>Language-Specific Formats</h5> <ul> <li>Using a language's built-in encoding functionality is bad from interoperability, security, efficiency, and versioning perspectives.</li> </ul> <h5>JSON, XML, and Binary Variants</h5> <ul> <li>JSON distinguishes strings and numbers, but it doesn't distinguish integers and floating-point numbers, and it doesn't specify a precision.</li> <li>Base64-encoding a binary string to use it in XML or JSON increases the data size by 33%.</li> <li>Simply having different organizations agree on a single format is far more important than the aesthetics or efficiency of a format.</li> </ul> <h6>Binary encoding</h6> <ul> <li>Because JSON and XML don't prescribe a schema, they need to include all the object field names within the encoded data.</li> <li>It's not clear whether the small space reduction by MessagePack, a binary encoding for JSON, is worth the loss of human-readability.</li> </ul> <h5>Thrift and Protocol Buffers</h5> <ul> <li>Confusingly, Thrift has two binary encoding formats, called <em>BinaryProtocol</em> and <em>CompactProtocol</em>.</li> <li>Field tags are like aliases for fields –&nbsp;they are a compact way of saying what field we're talking about without spelling out its name.</li> <li>Variable length integers use the top bit of each byte to specify whether there are still more bytes to come.</li> <li>With variable length integers, the numbers -64 to 63 are encoded in one byte, -8192 to 8191 are encoded in two bytes, etc. Bigger numbers use more bytes.</li> <li>The <code>required</code> keyword of Protocol Buffers enables a runtime check that fails if the field is not set, which is useful for catching bugs.</li> </ul> <h6>Field tags and schema evolution</h6> <ul> <li>With Thrift and Protocol Buffers, you can change a field name, but changing its tag renders all existing encoded data invalid.</li> <li>If old code reads data written by new code, including a new field with an unrecognized tag, it can simply ignore that field. This maintains forward compatibility.</li> </ul> <h6>Datatypes and schema evolution</h6> <ul> <li>With Protocol Buffers, it's okay to change an <code>optional</code> (single-valued) field into a <code>repeated</code> (multi-valued) field.</li> </ul> <h5>Avro</h5> <ul> <li>Avro has two schema languages: Avro IDL is intended for human editing, and a JSON equivalent that is more easily machine-readable.</li> <li>An encoded byte string specifies nothing to identify fields or their data types. It simply consists of values concatenated together.</li> <li>Consequently the binary data can only be decoded correctly if the code reading the data is using the <em>exact same schema</em> as the code that wrote the data.</li> </ul> <h6>The writer's schema and the reader's schema</h6> <ul> <li>While decoding data, Avro resolves the differences between the <em>writer's schema</em> and the <em>reader's schema</em>, and translates read data from the former to the latter. This enables schema evolution.</li> </ul> <h6>Schema evolution rules</h6> <ul> <li>Adding a field that has no default value breaks backward compatibility. Removing a field that has no default value breaks forward compatibility.</li> <li>Avro doesn't have <code>optional</code> or <code>required</code> keywords like Thrift and Protocol buffers, but instead allows defining union types with <code>null</code> as a value.</li> </ul> <h6>Dynamically generated schemas</h6> <ul> <li>Because an Avro schema doesn't define tag numbers, it is friendlier to dynamically generated schemas, e.g. creating an Avro schema from a relational database schema.</li> </ul> <h5>The Merits of Schemas</h5> <ul> <li>Database vendors provide a driver (e.g. using the ODBC or JDBC APIs) that decode responses from a database's network protocol into in-memory data structures.</li> <li>Schema evolution allows the same flexibility as schemaless/schema-on-read JSON databases, while also providing better guarantees about your data and better tooling.</li> </ul> <h4>Modes of Dataflow</h4> <h5>Dataflow through Databases</h5> <ul> <li>In an environment where the application is changing, it's likely that some processes accessing the database will be running newer code and some will be running older code.</li> <li>Consequently a value in the database might be written by a newer version of the code, and then later read by an older version. And so forward compatibility is required.</li> <li>If an application decodes a database value into model objects, and then later re-encodes those model objects, the unknown fields may be lost in that translation process.</li> </ul> <h6>Different values written at different times</h6> <ul> <li>Data outlives code: While deploys may replace older code with newer code within minutes, data in the database may have last been encoded and written years ago.</li> <li>Rewriting (migrating) data into a new schema is possible, but it is expensive on a large data set, and so most databases avoid it if possible.</li> <li>Schema evolution allows a database to appear as if it was encoded with a single schema, although the underlying storage may contain records encoded with various historical schema versions.</li> </ul> <h5>Dataflow Through Services: REST and RPC</h5> <ul> <li>The application-specific APIs of services provide encapsulation, by restricting what clients can and cannot do.</li> <li>A key design goal of service-oriented/microservices architecture is to make the application easier to change and maintain by making services independently deployable and evolvable.</li> </ul> <h6>Web services</h6> <ul> <li>The two popular approaches to web services are REST and SOAP.</li> <li>REST emphasizes data formats, using URLs for identifying resources and using HTTP features for cache-control, authentication, and content-type negotiation.</li> <li>SOAP is an XML protocol for API requests, and comes with a sprawling set of related standards (the <em>web standards framework</em>, or <em>WS-</em>) that add various features.</li> <li>The API of a SOAP web service is described using an XML-based language called the Web Services Descriptive Language, or WSDL. WSDL enables code generation so that a client can access a remote service using local classes and method calls.</li> <li>WSDL is not designed to be human-readable, and SOAP messages are often too complex to construct manually, so tooling, code generation, and IDEs fill in the gaps.</li> <li>Despite the ostensible standards, interoperability between different vendors' implementations often causes problems in practice.</li> </ul> <h6>The problems with remote procedure calls (RPCs)</h6> <ul> <li>The RPC model tries to make a request to a remote service look the same as calling a method within the same process –&nbsp;an abstraction called <em>location transparency</em>.</li> <li>Variable latency, timeouts, retries and idempotence semantics, and serialization all mean there's no point in location transparency. Calling a remote service is fundamentally different.</li> </ul> <h6>Current directions for RPC</h6> <ul> <li>The new generation of RPC frameworks is more explicit about the fact that a remote request is different from a local function call.</li> <li>RESTful APIs have the advantage of being good for experimentation and debugging, support by all mainstream programming languages and platforms, and a vast ecosystem of available tools.</li> <li>For these reasons, REST is used for public APIs, while RPC frameworks mostly focus on requests between services owned by the same organization.</li> </ul> <h6>Data encoding and evolution for RPC</h6> <ul> <li>Between services using RPC, we assume servers will be updated first, and clients second. You therefore need backward compatibility on requests, and forward compatibility on responses.</li> <li>The backward and forward compatibility properties of an RPC scheme are inherited from whatever encoding it uses, such as Thrift or gRPC.</li> </ul> <h5>Message-Passing Dataflow</h5> <ul> <li>In <em>asynchronous message-passing</em> systems, messages are delivered to an intermediary called a <em>message broker</em> which stores the message temporarily.</li> <li>Message brokers can act as a buffer if the recipient is unavailable or overloaded, redeliver messages to crashed processes, deliver messages to multiple recipients, and decouple producers and consumers.</li> </ul> <h6>Message brokers</h6> <ul> <li>A process sends a message to a named <em>queue</em> or <em>topic</em>, and the broker ensures delivery of the message to one or more <em>consumers</em> or <em>subscribers</em> of that queue or topic.</li> </ul> <h6>Distributed actor frameworks</h6> <ul> <li>In the <em>actor model</em>, logic is encapsulated by actors that communicate via asynchronous messages, where delivery of each message is not guaranteed (even within the same process).</li> <li>In a <em>distributed actor framework</em>, location transparency works better than with RPC, because the actor model already assumes that messages may be lost.</li> <li>A distributed actor framework essentially integrates a message broker and the actor programming model into a single framework.</li> </ul> <h3>Chapter 5: Replication</h3> <ul> <li>Reasons to replicate data include reducing access latency by moving data geographically close to users, increasing availability, and increasing read throughput.</li> <li>All the difficulty in replication lies in handling <em>changes</em> to replicated data.</li> </ul> <h4>Leaders and Followers</h4> <ul> <li><em>Leader-based replication</em>, or <em>active/passive replication</em>, is the most common solution to ensuring that data is persisted to all replicas.</li> <li>Whenever the leader writes new data to its local storage, it also sends the data change to each of its followers via a <em>replication log</em> or <em>change stream</em>.</li> </ul> <h5>Synchronous Versus Asynchronous Replication</h5> <ul> <li>With <em>synchronous</em> replication, the leader waits until a follower has confirmed it received a write before reporting success to the user and making it visible to other clients.</li> <li>With <em>asynchronous</em> replication, the leader sends the write to a follower, but doesn't wait for a response.</li> <li>If a synchronous follower does not respond because of a crash, network fault, etc. then no writes can be processed, and so fully synchronous replication is impractical.</li> <li>In practice replication may be <em>semi-synchronous</em>, where one of the followers is synchronous and the rest are asynchronous.</li> <li>Leader-based replication is often completely asynchronous:</li> <li>As an advantage, if the leader fails and is not recoverable, any writes that have not yet been replicated are lost.</li> <li>As a disadvantage, the leader can continue processing writes, even if all of its followers have fallen behind.</li> </ul> <h5>Setting Up New Followers</h5> <ul> <li>A new follower must ingest a consistent snapshot of the leader's database, and then consume all updates from the replication log since that time.</li> <li>A position in the leader's replication log is called the <em>log sequence number</em> in PostgreSQL, and the <em>binlog coordinates</em> in MySQL.</li> </ul> <h5>Handling Node Outages</h5> <h6>Follower failure: Catch-up recovery</h6> <ul> <li>The follower can connect to the leader and, via the replication log, consume all data changes that occurred while the follower was disconnected.</li> </ul> <h6>Leader failure: Failover</h6> <ul> <li>A <em>failover</em> requires promoting one follower as the new leader, reconfiguring clients to send data to the new leader, and reconfiguring other followers to consume data changes from the new leader.</li> <li>Most systems simply rely on a timeout to determine that the leader has failed.</li> <li>The best candidate for leadership is usually the replica with the most up-to-date changes from the old leader, to minimize any data loss.</li> <li>If asynchronous replication is used, the new leader may not have received all writes from the old leader before it failed. If the former leader later rejoins the cluster after the new leader has processed new writes, it usually discards the un-replicated writes.</li> <li>In a <em>split brain</em>, two nodes both believe they are the leader. If there is no process for resolving conflicting writes, this leads to data loss or corruption.</li> <li>Because failovers are fraught with things that can go wrong, some operations teams prefer to perform them manually.</li> </ul> <h5>Implementation of Replication Logs</h5> <h6>Statement-based replication</h6> <ul> <li>Replicating every write request (statement) from a leader to its followers has many edge cases (e.g. non-determinism from <code>NOW()</code> or <code>RAND()</code>) and is not preferred.</li> </ul> <h6>Write-ahead log (WAL) shipping</h6> <ul> <li>The WAL details which bytes were changed in which blocks. It's thus a poor choice for a replication log, e.g. you cannot run different versions of the database on leaders and followers.</li> </ul> <h6>Logical (row-based) log replication</h6> <ul> <li>A logical log is a sequence of records describing writes to database tables with row granularity. This is the approach MySQL's binlog uses.</li> <li>Logical logs are decoupled from storage engine internals and therefore backward compatible, and are also easier for external applications to parse.</li> </ul> <h6>Trigger-based replication</h6> <ul> <li>A trigger can log data changes into a separate table for reading by an external process.</li> <li>Trigger-based replication has greater overhead than other replication methods and is more error-prone, but offers increased flexibility.</li> </ul> <h4>Problems with Replication Lag</h4> <ul> <li>If an application reads from an asynchronous follower, it may see outdated information if the follower has fallen behind. This effect is known as <em>eventual consistency</em>.</li> <li>The <em>replication lag</em> to a follower may usually be only a fraction of a second, but given high load or network problems, it can increase to several minutes.</li> </ul> <h5>Reading Your Own Writes</h5> <ul> <li><em>Read-your-writes</em> consistency is also known as <em>read-after-write</em> consistency.</li> <li>If data is accessed from multiple devices, you may want to provide a stronger guarantee of <em>cross-device</em> read-after-write consistency.</li> <li>If your replicas are distributed across different data centers, there is no guarantee that connections from different devices will be routed to the same data center.</li> </ul> <h5>Monotonic Reads</h5> <ul> <li>If the first query by a user goes to an up-to-date replica, but the second query goes to a stale replica, then the user may observe data <em>moving backward in time</em>.</li> <li><em>Monotonic reads</em> ensures this anomaly does not happen. It is a weaker guarantee than strong consistency, but a stronger guarantee than eventual consistency.</li> <li>One way of achieving monotonic reads is to ensure that each user always queries data from the same replica, e.g. using the hash of the user ID.</li> </ul> <h5>Consistent Prefix Reads</h5> <ul> <li><em>Consistent prefix reads</em> guarantees that if a sequence of writes happen in a certain order, then anyone reading those writes will see them appear in the same order.</li> <li>In a sharded database, partitions operate independently, and so there is no global ordering of writes. A user's query might return some parts of the database in an older state, and some in a newer state.</li> </ul> <h5>Solutions for Replication Lag</h5> <ul> <li>When working with an eventually consistent system, it's worth thinking about how the application behaves if replication lag increases to several minutes or hours.</li> <li>Pretending that replication is synchronous when it is in fact asynchronous is a recipe for problems later.</li> </ul> <h4>Multi-Leader Replication</h4> <ul> <li>In a <em>multi-leader</em> configuration, or <em>active-active replication</em>, there are multiple leaders accepting writes, with each acting as a follower to the other leaders.</li> </ul> <h5>Use Cases for Multi-Leader Replication</h5> <h6>Multi-datacenter operation</h6> <ul> <li>You can have a leader in each data center. Within a datacenter, regular leader-follower replication is used; between datacenters, leaders replicate changes to each other.</li> <li>With multiple datacenters, the inter-datacenter network delay is hidden from users, which means the perceived performance may be better.</li> <li>The biggest downside of multi-leader replication is the need to resolve write conflicts if the same data is concurrently modified in two different datacenters.</li> <li>Auto-incrementing keys, triggers, and integrity constraints are problematic, and so multi-leader replication is considered dangerous territory to be avoided if possible.</li> </ul> <h6>Clients with offline operation</h6> <ul> <li>Clients with offline operation are like multi-leader replication between datacenters, but each device is a "datacenter" and network connections between them are highly unreliable.</li> </ul> <h5>Handling Write Conflicts</h5> <h6>Synchronous versus asynchronous conflict detection</h6> <ul> <li>With synchronous conflict detection, you lose the advantage of each replica accepting writes independently, and so you might as well just use single-leader replication.</li> </ul> <h6>Converging toward a consistent state</h6> <ul> <li>In a multi-leader configuration there is no defined ordering of writes. But the database must resolve each conflict in a <em>convergent way</em>, yielding a single final value across all replicas.</li> <li>Approaches to achieving convergent conflict resolution include:</li> <li><em>Last write wins</em> (LWW): Append the timestamp to each write, and apply the write with the largest value, discarding the others. This is dangerously prone to data loss.</li> <li>Somehow merge the values together.</li> <li>Record the conflict in an explicit data structure that preserves all information, and write application code to resolve the conflict later.</li> </ul> <h6>Custom conflict resolution logic</h6> <ul> <li><em>Conflict-free replicated data types</em> (CDRTs) are data structures (e.g. sets, maps, ordered lists, etc) that automatically resolve conflicts in sensible ways.</li> <li><em>Mergeable persistent data structures</em> track history explicitly and use a three-way merge function (similar to Git).</li> <li><em>Operational transformations</em> are for concurrent editing of an ordered list of items, and are used by Etherpad and Google Docs.</li> </ul> <h5>Multi-Leader Replication Topologies</h5> <ul> <li>The most general topology is all-to-all in which every leader sends its writes to every other leader.</li> <li>The fault tolerance of a more densely connected topology is better because it allows messages to travel along different paths, avoiding a single point of failure.</li> <li>One problem with all-to-all topologies is that if some network links are faster than others, some messages can overtake others, e.g. an update arriving before the preceding insert.</li> </ul> <h4>Leaderless Replication</h4> <ul> <li>Amazon's Dynamo has popularized abandoning the concept of a leader and allowing any replica to directly accept writes from clients.</li> </ul> <h5>Writing to the Database When a Node is Down</h5> <ul> <li>Writes are sent to all nodes in parallel. To account for a node being down and missing a write, clients read from nodes in parallel, and rely on version numbers to retain the latest value.</li> </ul> <h6>Read repair and anti-entropy</h6> <ul> <li>With <em>read repair</em>, a client reads from several nodes in parallel and then writes the latest value back to any replicas from which it received stale data. This works well for values that are frequently read.</li> <li>An <em>anti-entropy process</em> runs in the background, comparing data between replicas and fixing stale data. This does not copy writes in any particular order and may be slow to fix differences.</li> </ul> <h6>Quorums for reading and writing</h6> <ul> <li>If there are <em>n</em> replicas, every write must be confirmed by <em>w</em> nodes to be considered successful, and we must query at least <em>r</em> nodes for each read.</li> <li>As long as <em>w + r &gt; n</em>, we expect to get an up-to-date value when reading, because at least one of the <em>r</em> nodes we're reading from must be up-to-date.</li> <li>Reads and writes that obey these <em>r</em> and <em>w</em> parameters are called <em>quorum</em> reads and writes.</li> <li>Normally reads and writes are always sent to all <em>n</em> nodes in parallel, and <em>r</em> and <em>w</em> determine how many nodes we wait for.</li> </ul> <h5>Limitations of Quorum Consistency</h5> <ul> <li>Often <em>r</em> and <em>w</em> are chosen to be more than <em>n/2</em> nodes, because that ensures <em>w + r &gt; n</em> while tolerating up to <em>n/2</em> (rounded down) node failures.</li> <li>If you choose <em>w + r ≤ n</em> you are more likely to read stale values, but this configuration allows for lower latency and higher availability.</li> <li>Even with <em>w + r &gt; n</em> there are edge cases where stale values are returned:</li> <li>If two writes occur concurrently, it is not clear which one happened first. You must merge the concurrent writes.</li> <li>If a write happens concurrently with a read, the write may be reflected only on some of the replicas.</li> <li>If a write failed on some nodes and overall succeeded on fewer than <em>w</em> replicas, it is not rolled back on the replicas where it succeeded.</li> </ul> <h6>Monitoring staleness</h6> <ul> <li>In leader-based replication, the replication lag is computed by subtracting the follower's replication log position from that of the leader.</li> <li>In systems with leaderless replication, there is no fixed order in which writes are applied, which makes monitoring difficult.</li> </ul> <h5>Sloppy Quorums and Hinted Handoff</h5> <ul> <li>Network interruptions may cut off a client from a large number of database nodes, such that a quorum for reads and writes cannot be reached.</li> <li>In a <em>sloppy quorum</em>, writes and reads still require <em>w</em> and <em>r</em> successful responses, but those may include nodes not among the <em>n</em> "home" nodes for a value.</li> <li><em>Hinted handoff</em> is when, upon the network interruption being fixed, any writes that nodes temporarily accepted are sent to the appropriate "home" nodes.</li> <li>Even when <em>w + r &gt; n</em>, a client cannot be sure it read the latest value for a key, because the latest value may be temporarily written to nodes outside of <em>n</em>.</li> </ul> <h5>Detecting Concurrent Writes</h5> <ul> <li>When concurrently writing to a Dynamo-style database with the same key, events may arrive in a different order at different nodes due to variable network delays and partial failures.</li> </ul> <h6>Last write wins (discarding concurrent writes)</h6> <ul> <li>If we have some unambiguous way to determine which write is more "recent," and if every write is copied to every replica, then replicas will eventually converge to the same value.</li> <li>Concurrent writes do not have a natural ordering, and so <em>last write wins</em> (LWW) imposes an ordering by associating each write with a timestamp.</li> <li>LWW trades durability for convergence: Several concurrent writes to the same key may be reported as successful to the client, but only one of the writes will survive.</li> <li>If losing data is not acceptable, then LWW is a poor choice for conflict resolution.</li> </ul> <h6>The "happens-before" relationship and concurrency</h6> <ul> <li>An operation A <em>happens before</em> another operation B if B knows about A, or depends on A, or builds on A in some way.</li> <li>We can simply say that two operations are <em>concurrent</em> if neither happens before the other (i.e. neither knows about the other).</li> <li>Exact time does not matter: If the network is slow, two operations can occur some time apart but still appear to be concurrent, because the network prevented one operation from being able to know about the other.</li> </ul> <h6>Capturing the happens-before relationship</h6> <ul> <li>A server can determine whether two operations are concurrent by looking at version numbers&nbsp;–&nbsp;it does not need to interpret the value itself.</li> <li>When the server receives a write with a particular version number:</li> <li>It can overwrite all values with that version number or below, since the client must have merged them into the new value.</li> <li>It must keep all values with a higher version number, because those values are concurrent with the incoming write.</li> </ul> <h6>Merging concurrently written values</h6> <ul> <li>If several operations happen concurrently, clients must clean up afterward by merging the concurrently written <em>sibling</em> values.</li> <li>Merging sibling values is essentially the same problem as conflict resolution in multi-leader replication.</li> <li>Merging sibling values is complex and error prone, while CRDTs can automatically merge siblings in sensible ways.</li> </ul> <h6>Version vectors</h6> <ul> <li>With multiple replicas, we need to use a version number <em>per replica</em> as well as per key, so that we know which values to overwrite and which to preserve as siblings.</li> <li>The collection of version numbers from all the replicas is called a <em>version vector</em>. It is also sometimes called a <em>vector clock</em>, even though they are not the same.</li> </ul> <h3>Chapter 6: Partitioning</h3> <ul> <li>Partitioning enables scalability: By distributing data across many disks, the query load can be distributed across many processors.</li> </ul> <h4>Partitioning and Replication</h4> <ul> <li>Partitioning is usually combined with replication so that copies of each partition are stored on multiple nodes for fault tolerance.</li> </ul> <h4>Partitioning of Key-Value Data</h4> <ul> <li>If some partitions have more data or queries than others, then the partitioning is <em>skewed</em> and is less effective.</li> <li>A partition with a disproportionately high load is called a <em>hot spot</em>.</li> </ul> <h5>Partitioning by Key Range</h5> <ul> <li>When assigning a contiguous range of keys to each partition, the ranges are not necessarily evenly spaced, because your data may not be evenly distributed.</li> <li>Certain access patterns can lead to hot spots with key range partitioning, e.g. partitioning by timestamp and then always writing to the partition for today.</li> </ul> <h5>Partitioning by Hash of Key</h5> <ul> <li>When using a hash function to determine the partition for a given key, the hash function does not need to be cryptographically strong.</li> <li>Each partition is responsible for a range of hashes of keys. Assuming the hash function is uniform, the partition boundaries can be evenly spaced.</li> <li>Keys that were once adjacent are now scattered across all the partitions, so their sort order is lost.</li> <li>In Cassandra, the first part of the key is hashed to determine the partition, while the remaining columns are used as a concatenated index into Cassandra's SSTables.</li> <li>Cassandra achieves a compromise between the partitioning strategies: If a query specifies a fixed value for the first column, it can perform an efficient range scan over the other columns.</li> </ul> <h5>Skewed Workloads and Relieving Hot Spots</h5> <ul> <li>Most systems are not able to automatically compensate for a highly skewed workload, and so it's the responsibility of the application reduce the skew.</li> </ul> <h4>Partitioning and Secondary Indexes</h4> <ul> <li>The problem with secondary indexes is that they don't map neatly to partitions.</li> </ul> <h5>Partitioning Secondary Indexes by Document</h5> <ul> <li>Each partition can maintain its own secondary indexes, covering only the documents in the partition. Such a document-partitioned index is called a <em>local index</em>.</li> <li>Querying such a partitioned database requires a <em>scatter/gather</em> operation. Even if querying the partitions in parallel, scatter/gather is prone to tail latency amplification.</li> </ul> <h5>Partitioning Secondary Indexes by Term</h5> <ul> <li>Rather than each partition having its own secondary index, we can construct a <em>global index</em> that covers data in all partitions.</li> <li>Such an index is <em>term-partitioned</em> because the term you're looking for determines the partition of the index.</li> <li>A client needs to query only the partition containing the term it wants, but a write to a single document may update multiple terms and in turn update multiple partitions of the index.</li> <li>In practice, updates to global secondary indexes are often asynchronous.</li> </ul> <h4>Rebalancing Partitions</h4> <ul> <li>The process of moving load from one node to another is called <em>rebalancing</em>.</li> </ul> <h5>Strategies for Rebalancing</h5> <h6>How not to do it: hash mod N</h6> <ul> <li>The problem with the <em>mod N</em> approach is that if the number of nodes <em>N</em> changes, then most keys will need to be moved from one node to another.</li> </ul> <h6>Fixed number of partitions</h6> <ul> <li>We can create more partitions than nodes, and then assign multiple partitions to each node. As nodes are added or removed, this assignment changes.</li> <li>In this configuration, the number of partitions does not change, nor does the assignment of keys to partitions.</li> <li>By assigning more partitions to nodes that are more powerful, you can force these nodes to take a greater share of the load.</li> </ul> <h6>Dynamic partitioning</h6> <ul> <li>With <em>dynamic partitioning</em>, partitions are split and merged dynamically based on their size, similar to interior nodes in a B-tree.</li> <li>An advantage of dynamic partitioning is that the number of partitions adapts to the total data volume.</li> <li>Dynamic partitioning can be used with both key range-partitioned data, as well as with hash-partitioned data.</li> </ul> <h6>Partitioning proportionally to nodes</h6> <ul> <li>With dynamic partitioning, the number of partitions is proportional to the size of the data set. With a fixed number of partitions, the size of each partition is.</li> <li>A third option is a fixed number of partitions per node. Since a larger data volume generally requires a larger number of nodes to store, this also keeps the size of each partition stable.</li> </ul> <h5>Operations: Automatic or Manual Rebalancing</h5> <ul> <li>If not done correctly, rebalancing can overload the network or the nodes and harm the performance of other requests while it is happening.</li> <li>If a node is overloaded, then automated rebalancing may lead other nodes to conclude the node is dead and move load away from it. This increases load on the other nodes and the network, potentially causing a cascading failure.</li> <li>While fully automated rebalancing can be convenient, it's good to have a human in the loop to prevent operational surprises.</li> </ul> <h4>Request Routing</h4> <ul> <li><em>Service discovery</em> addresses which node a client should connect to, and is a critical requirement for software accessible over a network with high availability.</li> <li>Any component making the routing decision (a random node, a routing tier, or a client) must learn about changes in the assignment of partitions to nodes.</li> <li>Many distributed systems rely on a separate coordination service such as ZooKeeper to track this cluster metadata.</li> <li>Cassandra and Riak use a <em>gossip protocol</em> among the nodes to disseminate any changes in cluster state.</li> <li>When a random node or a routing tier makes the routing decision, clients can use DNS to find their IP addresses as their assignment changes slowly.</li> </ul> <h3>Chapter 7: Transactions</h3> <ul> <li>Transactions <em>simplify the programming model</em> for applications accessing a database.</li> <li>A <em>safety guarantee</em> is when the database prevents a potential error and concurrency issue, and so the application does not have to compensate for it.</li> <li>Sometimes there are advantages to weakening transaction guarantees or abandoning them entirely, e.g. for higher performance or higher availability.</li> </ul> <h4>The Slippery Concept of a Transaction</h4> <ul> <li>Transactions are not the antithesis of stability. But like every other technical design choice, they come with trade-offs.</li> </ul> <h5>The Meaning of ACID</h5> <ul> <li>In practice, one database's implementation of <em>ACID</em> does not equal that of another. ACID has become mostly a marketing term.</li> </ul> <h6>Atomicity</h6> <ul> <li>Atomicity is not about concurrency, but about ensuring that when a transaction is aborted, that no data was changed and the operation can be safely retried.</li> </ul> <h6>Consistency</h6> <ul> <li>Consistency means that there exist invariants about your data, but it is the application's responsibility to define its transactions so that they preserve consistency.</li> <li>Atomicity, isolation, and durability are properties of the database, whereas consistency is actually a property of the application.</li> </ul> <h6>Isolation</h6> <ul> <li>Isolation means that concurrently executing transactions do not step on each other's toes.</li> <li>Isolation is sometimes formalized as <em>serializability</em>, where each transaction can pretend like it's the only transaction running, but this carries a large performance penalty.</li> </ul> <h6>Durability</h6> <ul> <li>Durability means that once a transaction has committed successfully, its written data will not be forgotten, even if there is a hardware fault or the database crashes.</li> <li>To provide durability, a database must wait until writing to non-volatile storage or replication has completed before reporting a transaction as committed.</li> </ul> <h5>Single-Object and Multi-Object Operations</h5> <ul> <li>Atomicity gives all-or-nothing guarantees when writing data in a transaction, and isolation ensures concurrently running transactions don't interfere with each other.</li> </ul> <h6>Single-object writes</h6> <ul> <li>For a single object in a storage engine, atomicity can be implemented using a log for crash recovery, and isolation can be implemented using a lock on each object.</li> <li>Compare-and-set and other single-object operations are not transactions, because they do not allow grouping multiple operations on multiple objects.</li> </ul> <h6>The need for multi-object transactions</h6> <ul> <li>Use cases where writes to several different objects need to be coordinated include:</li> <li>Inserting several records that refer to one another and ensuring that their foreign keys are valid.</li> <li>Updating denormalized data across several documents together.</li> <li>Updating secondary indexes to reflect a changed value.</li> </ul> <h6>Handling errors and aborts</h6> <ul> <li>If the database is in danger of violating its guarantee of atomicity, isolation, or durability, then it would rather abandon the transaction than half-finish.</li> </ul> <h4>Weak Isolation Levels</h4> <ul> <li>Concurrency issues happen when one transaction reads data that is concurrently modified by another, or when two transactions try to concurrently modify the same data.</li> <li>Databases try to hide concurrency issues from application developers by providing <em>transaction isolation</em>.</li> <li><em>Serializable</em> isolation means the database guarantees that transactions have the same effect as if they ran <em>serially</em>.</li> </ul> <h5>Read Committed</h5> <ul> <li>The most basic level of transaction isolation is <em>read committed</em>, which guarantees <em>no dirty reads</em> and <em>no dirty writes</em>.</li> </ul> <h6>No dirty reads</h6> <ul> <li><em>No dirty reads</em> means when reading from the database, you will only see data that has been committed.</li> <li>Any writes by a transaction only become visible to others when that transaction commits, and then all its writes become visible at once.</li> <li>No dirty reads means transactions won't see data in a partially updated state, and that a transaction won't see data that is later rolled back.</li> </ul> <h6>No dirty writes</h6> <ul> <li><em>No dirty writes</em> means when writing to the database, you will only overwrite data that has been committed.</li> <li>Instead of allowing overwriting of uncommitted data, databases usually delay the second write until the first write's transaction has committed or aborted.</li> <li>If transactions update multiple objects, dirty writes allow interleaving the writes to the multiple objects.</li> </ul> <h6>Implementing read committed</h6> <ul> <li>Read committed is the default setting in Oracle 11g, PostgreSQL, SQL Server 2012, MemSQL, and many other databases.</li> <li>To prevent dirty reads, a transaction must acquire a row-level lock before writing and then relinquish it upon committing or aborting.</li> <li>When a transaction updates a object, other transactions read the old value until the transaction commits, at which point they begin reading the updated value.</li> </ul> <h5>Snapshot Isolation and Repeatable Read</h5> <ul> <li><em>Read skew</em> is an example of a <em>non-repeatable read</em>, where a transaction querying an object at different times sees different values because of concurrent updates.</li> <li><em>Read skew</em> is considered acceptable under read committed isolation, as the client is indeed reading only committed data.</li> <li><em>Snapshot isolation</em> provides a consistent snapshot of the database, so the transaction sees all data that was committed at the start of the transaction.</li> <li>Snapshot isolation is a boon for long-running read-only queries such as backups and analytics.</li> <li>Snapshot isolation is supported by PostgreSQL, MySQL with the InnoDB storage engine, Oracle, SQL Server, and others.</li> </ul> <h6>Implementing snapshot isolation</h6> <ul> <li>A key principle of snapshot isolation is that readers never block writers, and writers never block readers. Reads do not require any locks.</li> <li>With <em>multi-version concurrency control</em>, the database keeps several different versions of a committed object so that various in-progress transactions can see the database state at different points in time.</li> <li>When a transaction deletes a row, a garbage collector deletes it later, only when it is certain no transaction can any longer access the deleted data.</li> </ul> <h6>Visibility rules for observing a consistent snapshot</h6> <ul> <li>When a transaction starts, the database enumerates all the IDs of in-progress transactions, and their writes are ignored even if they subsequently commit.</li> <li>Any writes made by transactions with a later transaction ID – which are strictly increasing, and therefore started after the current transaction –&nbsp;are ignored.</li> <li>An object is visible if both of the following conditions are true:</li> <li>When the reader's transaction started, the transaction that created the object had already committed.</li> <li>The object is not marked for deletion, or if it is, the transaction that requested deletion had not yet committed when the reader's transaction started.</li> </ul> <h6>Indexes and snapshot isolation</h6> <ul> <li>With a <em>copy-on-write</em> strategy for B-tree indexes, parent pages&nbsp;– up to the root of the tree&nbsp;– are copied and updated to point to the new versions of their child pages.</li> <li>With this strategy, any pages that are not affected by a write do not need to be copied, and remain immutable.</li> </ul> <h6>Repeatable read and naming confusion</h6> <ul> <li>The SQL standard's definition of isolation levels is flawed – it is ambiguous, imprecise, and not an implementation-independent standard.</li> <li>No one really knows what repeatable read means: Databases implementing it provide very different guarantees, and most don't satisfy a formal definition.</li> </ul> <h5>Preventing Lost Updates</h5> <ul> <li>The <em>lost update</em> can occur if an application reads some value, modifies it, and writes back the modified value (a <em>read-modify-write</em> operation).</li> </ul> <h6>Atomic write operations</h6> <ul> <li>Atomic operations are usually implemented by acquiring an exclusive lock on the object to update. This technique is known as <em>cursor stability</em>.</li> <li>Object-relational mapping frameworks make it easy to accidentally perform unsafe read-modify-write operations instead of relying on atomic operations provided by the database.</li> </ul> <h6>Explicit locking</h6> <ul> <li>The <code>FOR UPDATE</code> clause in SQL indicates that the database should take a lock on all rows returned by the query.</li> </ul> <h6>Automatically detecting lost updates</h6> <ul> <li>PostgreSQL's repeatable read, Oracle's serializable, and SQL Server's snapshot isolation levels automatically detect a lost update and abort the corresponding transaction.</li> </ul> <h6>Compare-and-set</h6> <ul> <li>An update statement with a <code>WHERE</code> clause specifying the old value may not stop a lost update if it reads from an old snapshot while a concurrent write occurs.</li> </ul> <h6>Conflict resolution and replication</h6> <ul> <li>Locks and compare-and-set operations assume a single up-to-date copy of the data, but multi-leader or leaderless replication does not guarantee this.</li> <li>But atomic operations can work well in a replicated context, especially if they are commutative.</li> <li>While the <em>last write wins</em> (LWW) conflict resolution strategy is prone to lost updates, it is the default in many replicated databases.</li> </ul> <h5>Write Skew and Phantoms</h5> <ul> <li><em>Write skew</em> generalizes the lost update problem: Two transactions read the same objects, and then concurrently update some of those objects.</li> <li>In the special case where different transactions update the same object, you a lost update or a dirty write anomaly, depending on the timing.</li> <li>If you cannot use a serializable isolation level to prevent write skew, the second-best option is to explicitly lock rows that the transaction depends on.</li> </ul> <h6>Phantoms causing write skew</h6> <ul> <li>In all cases of write skew, a <code>SELECT</code> query finds rows that match some condition, and then an <code>UPDATE</code> statement changes the set of rows matching that condition.</li> <li>If the <code>SELECT</code> query checks for the <em>absence</em> of rows matching a search condition, and the write adds a matching row, then <code>SELECT FOR UPDATE</code> returning no rows can't attach locks to anything.</li> <li>A <em>phantom</em> is when a write in one transaction changes the result of a query in another transaction.</li> <li>Snapshot isolation avoids phantoms for read-only queries, but in read-write transactions, phantoms can lead to tricky cases of write skew.</li> </ul> <h6>Materializing conflicts</h6> <ul> <li><em>Materializing conflicts</em> takes a phantom and turns it into a lock conflict on a concrete set of rows that exist in the database.</li> </ul> <h4>Serializability</h4> <ul> <li>Serializable isolation is the strongest isolation level, guaranteeing that transactions executing in parallel yield the same result as if they ran serially.</li> <li>This means that the database prevents <em>all</em> possible race conditions.</li> </ul> <h5>Actual Serial Execution</h5> <ul> <li>Redis, VoltDB, and more use a single-threaded execution model with all data in RAM, which allows transactions to execute much faster than if they had to load data from disk.</li> </ul> <h6>Encapsulating transactions in stored procedures</h6> <ul> <li>Almost all OLTP applications keep transactions short by avoiding interactively waiting for a user within a transaction, because we're slow at making up our minds.</li> <li>Even after removing the user, an interactive style between the application and database would yield dreadful throughput in a database without concurrency.</li> <li>Instead, single-threaded transaction processing requires the application to submit the entire transaction code to the database as a <em>stored procedure</em>.</li> </ul> <h6>Pros and cons of stored procedures</h6> <ul> <li>With stored procedures and in-memory data, executing all transactions on a single thread become feasible.</li> </ul> <h6>Partitioning</h6> <ul> <li>For applications with high write throughput, the single-threaded transaction processor can become a serious bottleneck.</li> <li>If you can ensure that each transaction only reads and writes data within some partition, then each partition to have its own independent transaction processing thread.</li> <li>Since cross-partition transactions have additional coordination overhead, they are much slower than single-partition transactions.</li> </ul> <h5>Two-Phase Locking (2PL)</h5> <ul> <li>Two-phase <em>locking</em> sounds very similar to two-phase <em>commit</em>, but they are completely different things.</li> <li>In 2PL, writers don't just block other writers, but they also block readers. Additionally, readers can block writers.</li> </ul> <h6>Implementation of two-phase locking</h6> <ul> <li>2PL is used by the serializable isolation level in MySQL (InnoDB) and SQL Server, and is the repeatable read isolation level in DB2.</li> <li>Each object in the database has a lock that can be acquired in <em>shared mode</em> or <em>exclusive mode</em>:</li> <li>Multiple readers can hold the lock in shared mode simultaneously, but they must wait for any transaction with an exclusive lock.</li> <li>A writer must first acquire the lock in exclusive mode. A transaction that first reads and then writes can upgrade its shared lock to an exclusive lock.</li> <li>The "two-phase" name comes from a transaction acquiring locks in the first phase, and then releasing its locks in the second phase (at the end of the transaction).</li> <li>The database automatically detects deadlocks betweens transactions and aborts one so that the others can make progress.</li> </ul> <h6>Performance of two-phase locking</h6> <ul> <li>Transaction throughput and query latency with 2PL is worse than under weak isolation because the locking and exclusive access reduces concurrency.</li> <li>Databases running 2PL have unstable latencies and can be very slow at high percentiles if there is contention in the workload.</li> <li>Deadlocks happen much more frequently with 2PL, and so clients can waste significant effort retrying transactions repeatedly.</li> </ul> <h6>Predicate locks</h6> <ul> <li>To prevent phantoms, a <em>predicate lock</em> allows locking all objects that match some search condition.</li> <li>A predicate lock applies even to objects that don't yet exist in the database, but which might be added in the future (phantoms).</li> </ul> <h6>Index-range locks</h6> <ul> <li>Checking for matching locks is time consuming, so most databases with 2PL implement a simplified approximation of predicate locking called <em>index-range locking</em>.</li> <li>It's safe to simplify a predicate by making it match a larger set of objects, as any write that matches the original predicate will also match the approximations.</li> <li>If there is no suitable index where a range lock can be attached, the database can fall back to a shared lock on the entire table.</li> </ul> <h5>Serializable Snapshot Isolation (SSI)</h5> <ul> <li><em>Serializable snapshot isolation</em> (SSI) provides full serializability, but has only a small performance penalty compared to snapshot isolation.</li> <li>SSI is the serializable isolation level in PostgreSQL since 9.1, and FoundationDB uses a similar algorithm.</li> </ul> <h6>Pessimistic versus optimistic concurrency control</h6> <ul> <li>2PL uses <em>pessimistic</em> concurrency control: if anything can go wrong (as indicated by another transaction holding a lock), then a transaction must wait.</li> <li>SSI uses <em>optimistic</em> concurrency control: a transaction will always forge ahead, but the database will abort it before committing if anything bad happened (i.e. isolation was violated).</li> <li>Given enough capacity, and if contention between transactions is not too high, optimistic concurrency control techniques perform better than pessimistic ones.</li> </ul> <h6>Decisions based on an outdated premise</h6> <ul> <li>Under snapshot isolation, the result of a query before updating may no longer be up-to-date when the transaction commits, because it may have been concurrently modified.</li> <li>We say the transaction was based on a <em>premise</em>, or a fact that was true at the start of the transaction, but may no longer be true by the end.</li> <li>To provide serializable isolation, the database must detect when a transaction may have acted on an outdated premise and consequently abort it.</li> </ul> <h6>Detecting stale MVCC reads</h6> <ul> <li>When a transaction wants to commit, the database checks whether other transactions have committed writes that it ignored earlier because of MVCC visibility rules. If so, the database aborts the transaction.</li> </ul> <h6>Detecting writes that affect prior reads</h6> <ul> <li>When a transaction writes to the database, it notifies transactions that recently read the affected data that what they read may no longer be up to date.</li> </ul> <h6>Performance of serializable snapshot isolation</h6> <ul> <li>Less detailed tracking of reads and writes is faster but less precise, and consequently may lead to more transactions being aborted than necessary.</li> <li>The big advantage of SSI over 2PL is that a transaction doesn't need to block waiting for locks held by another transaction, thereby making query latency more predictable and less variable.</li> </ul> <h4>Summary</h4> <ul> <li>Transactions are an abstraction layer allowing applications to pretend that certain concurrency problems and certain kinds of faults don't exist.</li> <li>Only serializable isolation protects against all types of anomalies and race conditions in a database.</li> </ul> <h3>Chapter 8: The Trouble with Distributed Systems</h3> <h4>Faults and Partial Failures</h4> <ul> <li>An individual computer with good software is either fully functional or completely broken, but not something in between.</li> <li>If an internal fault occurs, we prefer to crash completely rather than returning a wrong result, which are confusing and difficult to debug.</li> <li>A <em>partial failure</em> in a distributed system is where some parts break in an unpredictable way, even though other parts of the system are working fine.</li> <li>The possibility of partial failures and their non-determinism is what makes distributed systems hard to work with.</li> </ul> <h5>Cloud Computing and Supercomputing</h5> <ul> <li>In a system with thousands of nodes, it's reasonable to assume that <em>something</em> is always broken.</li> <li>We must accept the possibility of partial failure and incorporate fault-tolerance, thereby building a reliable system from unreliable components.</li> <li>Suspicion, pessimism, and paranoia pay off in distributed systems, so consider and test for a wide range of possible faults in your environment.</li> </ul> <h4>Unreliable Networks</h4> <ul> <li>We focus on <em>shard-nothing distributed systems</em>, which are a collection of machines connected by a network.</li> <li>In this kind of network, one node can send a message to another node, but the node makes no guarantees when it will arrive, or if it will arrive at all.</li> <li>If you send a request to another node and don't receive a response, it is impossible to tell why.</li> <li>When a timeout occurs, you still don't know whether the remote node got your request or not.</li> </ul> <h5>Network Faults in Practice</h5> <ul> <li>A <em>network partition</em> or <em>netsplit</em> is when one part of the network is cut off from the rest due to a network fault.</li> <li>Whenever any communication happens over a network, it may fail – there is no avoiding it.</li> </ul> <h5>Detecting Faults</h5> <ul> <li>Many systems need to automatically detect faulty nodes, but the uncertainty about the network makes it difficult to tell whether a node is working or not.</li> <li>Even if TCP acknowledges that a packet was delivered, the application may have crashed before processing it. The client needs a positive response from the application itself to confirm its processing.</li> <li>Conversely, if something goes wrong, you have to assume that you will get no response at all.</li> </ul> <h5>Timeouts and Unbounded Delays</h5> <ul> <li>Prematurely declaring a node as dead is problematic because if it's actually alive and in the middle of performing some action, having another node take over may perform the action twice.</li> <li>If the system is struggling with high load, declaring nodes dead prematurely can transfer their load to other nodes, causing a cascading failure.</li> <li>Asynchronous networks have unbounded delays and most server implementations cannot guarantee handling a request within some maximum time.</li> </ul> <h6>Network congestion and queueing</h6> <ul> <li><em>Network congestion</em> is when a packet experiences delay because it is queued up for sending onto a busy network link.</li> <li>When a VM is paused while another VM runs in a virtualized environment, it cannot consume data from the network, and so any incoming data is queued by the virtual machine monitor.</li> <li>Because UDP does not perform control flow and does not retransmit lost packets, it is a better choice than TCP when delayed data is worthless.</li> <li>Queueing delays have an especially wide range when a system is close to its maximum capacity, as long queues build up quickly.</li> <li>In multi-tenant environments, a <em>noisy neighbor</em> can use a lot of the resources without your knowledge and introduce highly variable network delays.</li> <li>Rather than using fixed timeouts, systems can continually measure the distribution of response times and their variability and adjust their timeouts accordingly.</li> </ul> <h5>Synchronous Versus Asynchronous Networks</h5> <ul> <li>A <em>synchronous</em> network with reserved resources for communication has no queueing, and so the maximum end-to-end latency is fixed. We call this a <em>bounded delay</em>.</li> </ul> <h6>Can we not simply make network delays predictable?</h6> <ul> <li>Ethernet an IP are packet-switched protocols, which suffer from queueing and thus unbounded delays in the network.</li> <li>Protocols using packet switching is optimized for <em>bursty traffic</em>, and dynamically adapt the rate of data transfer to the available network capacity.</li> <li>With careful use of <em>quality of service</em> (prioritization of packets) and <em>admission control</em> (rate limiting senders) we can provide statistically bounded delay.</li> <li>There is no "correct" value for timeouts - they need to be determined experimentally.</li> </ul> <h4>Unreliable Clocks</h4> <ul> <li>The quartz crystal oscillator powering a clock on a node is not perfectly accurate, and may be faster or slower than those of other machines.</li> </ul> <h5>Monotonic Versus Time-of-Day Clocks</h5> <ul> <li>Modern computers have a <em>time-of-day clock</em> and a <em>monotonic clock</em>.</li> </ul> <h6>Time-of-day clocks</h6> <ul> <li>A time-of-day clock returns the current date and time according to some calendar, also known as the <em>wall-clock time</em>.</li> <li>If such a clock is too-far ahead of the NTP server it's synchronizing with, it can be forcibly reset and jump back to a previous time. This makes it unsuitable for measuring elapsed time.</li> </ul> <h6>Monotonic clocks</h6> <ul> <li>The name <em>monotonic clock</em> comes from the fact that it is guaranteed to always move forward.</li> <li>The difference between two monotonic clock values tells you how much time has elapsed, but the <em>absolute</em> value of the clock is meaningless.</li> <li>NTP allows the monotonic clock to be sped up or slowed down by up to 0.05%, but it cannot cause the clock to jump forward or backward.</li> <li>The resolution of monotonic clocks is quite good, as they can measure time intervals in microseconds or less.</li> </ul> <h5>Clock Synchronization and Accuracy</h5> <ul> <li>Google assumes a drift of 6 ms for a clock that is synchronized every 30 seconds, or 17 seconds for a server that is synchronized once per day.</li> <li>If a computer's clock differs too much from an NTP server, it may refuse to synchronize, or the local clock will be forcibly reset.</li> <li>The best way for NTP servers to handle leap seconds is to perform the leap second adjustment gradually over the course of the day, a technique known as <em>smearing</em>.</li> <li>When a VM is paused so that another VM can run, the application experiences the pause as the clock suddenly jumping forward.</li> </ul> <h5>Relying on Synchronized Clocks</h5> <ul> <li>Incorrect clocks easily go unnoticed. Any bug that results is more likely to be silent and subtle data loss than a dramatic crash.</li> <li>Any node who's clock drifts too far from the others should be declared dead and removed from the cluster.</li> </ul> <h6>Timestamps for ordering events</h6> <ul> <li>Fundamental problems with last write wins given inaccurate clocks include:</li> <li>A node with a lagging clock cannot overwrite values previously written by a node with a fast clock until the clock skew between them has elapsed.</li> <li>Two nodes can independently generate writes with the same timestamp, especially when the clock has only millisecond precision.</li> <li>NTP's synchronization accuracy is limited by the network round-trip time, as well as quartz drift.</li> <li><em>Logical clocks</em> are based on incrementing counters rather than oscillating quartz crystal and are a safer alternative for ordering events.</li> <li><em>Physical clocks</em> are time-of-day and monotonic clocks, which measure actual elapsed time.</li> </ul> <h6>Clock readings have a confidence interval</h6> <ul> <li>Given quartz drift, don't think of a clock reading as a point in time, but as a range of times within a confidence interval.</li> <li>Google's <em>TrueTime</em> API in Spanner explicitly returns a confidence interval with the earliest possible and latest possible timestamp.</li> </ul> <h6>Synchronized clocks for global snapshots</h6> <ul> <li>The most common implementation of snapshot isolation requires a monotonically increasing transaction ID.</li> <li>With lots of small, rapid transactions, creating transaction IDs in a distributed system becomes an untenable bottleneck.</li> <li>If you have two non-overlapping TrueTime confidence intervals in Spanner, then one definitely happened before the other.</li> <li>To ensure transaction timestamps reflect reality, Spanner waits for the confidence interval length before committing a read-write transaction. This ensures that any transaction that may read the data has a non-overlapping confidence interval.</li> <li>Google deploys a GPS receiver or atomic clock in each data center to synchronize clocks within 7 ms.</li> </ul> <h5>Process Pauses</h5> <ul> <li>Reasons for <em>preempting</em> a running thread for a long time include:</li> <li>The programming language time might run garbage collection and "stop the world."</li> <li>The hypervisor can switch to a different virtual machine. The CPU time spent in other virtual machines is known as <em>steal time</em>.</li> <li>Unexpected disk access, such as the Java class loader lazily loading class files on first use.</li> <li>The server did not disable paging, and so a simple memory access results in a page fault that loads a page from disk into memory.</li> <li>The Unix process being paused by a <code>SIGSTOP</code> signal, and so it yields all CPU cycles until it receives a <code>SIGCONT</code> signal.</li> <li>A node in a distributed system must assume its execution can be paused, during which the rest of the world keeps moving and may even declare the node dead.</li> </ul> <h6>Response time guarantees</h6> <ul> <li>A <em>hard real-time</em> system is one where if the software does not respond by some <em>deadline</em>, the entire system may fail.</li> <li>In embedded systems, <em>real-time</em> means that a system is carefully designed and tested to meet specified timing guarantees in all circumstances.</li> <li>Real-time systems may have lower throughput, since they have to prioritize timely responses above all else.</li> </ul> <h6>Limiting the impact of garbage collection</h6> <ul> <li>If the runtime can warn the application that a node soon requires a GC pause, it can shift traffic to other nodes while the garbage collection runs.</li> </ul> <h4>Knowledge, Truth, and Lies</h4> <ul> <li>A node in the network cannot <em>know</em> anything for sure –&nbsp;it can only make guesses based on messages it does or does not receive via the network.</li> <li>In a distributed system, we can state the assumptions we are making about the behavior (the system model) and design the system in a way to meet those assumptions.</li> </ul> <h5>The Truth is Defined by the Majority</h5> <ul> <li>Many distributed algorithms rely on a <em>quorum</em>, where decisions require some minimum number of votes from several nodes in order to reduce the dependence on any single node.</li> <li>Most commonly the quorum is an absolute majority of more than half the nodes, as you cannot have two such majorities with conflicting decisions.</li> </ul> <h6>The leader and the clock</h6> <ul> <li>There can be problems if the majority of nodes have declared some node dead, but that node still believes it is the single node responsible for some thing.</li> </ul> <h6>Fencing tokens</h6> <ul> <li>With <em>fencing</em>, when a lock server grants a lock or lease, it can also return a <em>fencing token</em>, or a number that increases every time.</li> <li>A resource protected by the lock service can check tokens and reject any requests with an older token than one that has already been processed.</li> <li>It's unwise for a service to assume that clients are well behaved, as the people running the clients and those running the service likely have very different priorities.</li> </ul> <h5>Byzantine Faults</h5> <ul> <li>A <em>Byzantine fault</em> is when there is the risk that nodes may "lie," or send arbitrarily faulty or corrupted responses.</li> <li>The <em>Byzantine Generals Problem</em> imagines there are <em>n</em> generals who need to agree, but this is hampered by generals who are traitors.</li> <li>A system is <em>Byzantine fault-tolerant</em> if it operates correctly even if some nodes are not obeying the protocol, or if attackers are interfering with the network.</li> <li>In most server-side data systems, the cost of deploying Byzantine fault-tolerant solutions makes the impracticable.</li> </ul> <h5>System Model and Reality</h5> <ul> <li>A <em>system model</em> is an abstraction that describes what things an algorithm may assume.</li> <li>There are three system models for timing assumptions:</li> <li>A <em>synchronous model</em> assumes bounded network delay, bounded process pauses, and bounded clock error.</li> <li>A <em>partially asynchronous model</em> mostly behaves like the synchronous model, but can sometimes exceed its bounds for network delay, process pauses, and clock drift.</li> <li>An <em>asynchronous model</em> does not allow the model to make any timing assumptions, and in fact does not even have a clock.</li> <li>There are three system models for node failures:</li> <li>A <em>crash-stop faults model</em> assumes a node can fail only by crashing, and once it has crashed it is gone forever.</li> <li>A <em>crash-recovery faults model</em> assumes the node can crash but respond at some later time, and that stable storage is preserved while in-memory state is lost.</li> <li>A <em>Byzantine faults model</em> assumes nodes can do absolutely anything.</li> <li>For modeling real systems, the most useful model is partially asynchronous with crash-recovery faults.</li> </ul> <h6>Correctness of an algorithm</h6> <ul> <li>An algorithm is <em>correct</em> if it always satisfies its <em>properties</em> in all situations that we assume may occur in that system model.</li> </ul> <h6>Safety and liveness</h6> <ul> <li>Safety is often informally defined as <em>nothing bad happens</em>, and liveness as <em>something good eventually happens</em>.</li> <li>If a safety property is violated, we can point to the exact time at which it was broken. At this time the damage is done, and cannot be undone.</li> <li>A liveness property may not hold at some point in time, but there is always hope that it may be satisfied in the future.</li> <li>For distributed algorithms, it's common to require that safety properties <em>always</em> hold. Meaning even if all nodes crash, the algorithm must not return a wrong result.</li> <li>With liveness properties we can make caveats. The definition of the partially synchronous model requires that eventually the system returns to a synchronous state.</li> </ul> <h6>Mapping system models to the real world</h6> <ul> <li>System models help distill the complexity of real systems to a set of faults that we can reason about, so that we can understand the problem and try to solve it systematically.</li> </ul> <h4>Summary</h4> <ul> <li>If you can avoid opening Pandora's box and simply keep things on a single machine, it's generally worth doing so.</li> <li>Distributed sequence number generators like Twitter's Snowflake cannot guarantee that ordering is consistent with causality, because the timescale at which blocks of IDs are assigned is longer than the timescale of database reads and writes.</li> </ul> <h3>Chapter 9: Consistency and Consensus</h3> <ul> <li>The best way of building fault-tolerant systems is to find some general-purpose abstractions with useful guarantees, implement them once, and then let applications rely on them.</li> <li><em>Split brain</em> is when nodes believe they are the leader, and it often leads to data loss.</li> </ul> <h4>Consistency Guarantees</h4> <ul> <li>A better name for eventual consistency may be <em>convergence</em>, as we expect all replicas to eventually converge to the same value.</li> <li>The edge cases of eventual consistency only become apparent when there is a fault in the system or at high concurrency.</li> <li>Transaction isolation is about avoiding race conditions due to concurrently executing transactions. Distributed consistency is about coordinating the state of replicas in the face of delays and faults.</li> </ul> <h4>Linearizability</h4> <ul> <li><em>Linearizability</em> (or <em>strong consistency</em>) is to make a system appear as if there were only one copy of the data, and all operations on it are atomic.</li> <li>Linearizability is a <em>recency guarantee</em>: As soon as a client completes a write, all clients reading from the database must be able to see the value just written.</li> </ul> <h5>What Makes a System Linearizable?</h5> <ul> <li>A linearizable system appears as if it has only a single copy of the data.</li> <li>A <em>concurrent read</em> is a read that overlaps in time with the write operation. Because we don't know whether the write has taken effect when the read operation is processed, it may return either the old value or the new value.</li> <li>In a linearizable system there must be some point in time at which a register is updated to its new value, and so all subsequent reads must return the new value even if the write operation has not yet completed.</li> <li>It is possible to test whether a system is linearizable by recording the timings of all requests and responses, and checking whether they can be arranged in a valid sequential order.</li> <li>Serializability is a isolation property of transactions which guarantees that the transactions behave as if they had executed in some serial order.</li> <li>Implementations of serializability based on two-phase locking or actual serial execution are typically linearizable.</li> <li>Snapshot isolation is not linearizable because a consistent snapshot does not include writes that are more recent than the snapshot.</li> </ul> <h5>Relying on Linearizability</h5> <h6>Locking and leader election</h6> <ul> <li>Electing a leader requires one node successfully acquiring a lock. This system must be linearizable, as all nodes must agree on which node owns the lock.</li> <li>Consensus algorithms implement linearizability in a fault-tolerant way, but linearizable storage is the foundation for these coordination tasks.</li> </ul> <h6>Constraints and uniqueness guarantees</h6> <ul> <li>A hard uniqueness constraint requires linearizability, while constraints like foreign key or attribute constraints do not require linearizability.</li> </ul> <h6>Cross-channel timing dependencies</h6> <ul> <li>Without the recency guarantees of linearizability, you must be concerned with race conditions between two communication channels.</li> </ul> <h5>Implementing Linearizable Systems</h5> <ul> <li>The simplest way to implement linearizability is to rely on only a single copy of the data, but that does not tolerate faults.</li> <li>Single-leader replication is potentially linearizable if you make all reads from the leader&nbsp;– which assumes you know for sure who the leader is – or from synchronously updated followers.</li> <li>Multi-leader replication are generally not linearizable, because they concurrently process writes on multiple nodes and asynchronously replicate them to other nodes.</li> <li>Leaderless replication systems with LWW conflict resolution or sloppy quorums are not linearizable.</li> <li>Quorums where <em>w + r &gt; n</em> can be linearizable if readers perform read repair synchronously before returning results, and if writers read the latest quorum state before sending writes.</li> <li>Only linearizable read and write operations can be performed in this way; a linearizable compare-and-set operation cannot, because it requires a consensus algorithm.</li> </ul> <h5>The Cost of Linearizability</h5> <ul> <li>The <em>CAP theorem</em> states that applications that don't require linearizability can be more tolerant of failures:</li> <li>If your application requires linearizability, then a disconnected replica is <em>unavailable</em> and must either wait until the network is fixed, or return an error.</li> <li>If your application doesn't require linearizability, then a replica can remain <em>available</em> and process requests even if disconnected.</li> <li>The CAP theorem encouraged database engineers to explore a wider design space of distributed shared-nothing systems, which were more suitable for implementing large-scale web services.</li> <li>CAP is sometimes presented as <em>Consistent, Available, Partitioned – pick 2</em> but partitions are a kind of fault and so they will happen whether you like it or not.</li> <li>A better way of presenting CAP is <em>Consistent or Available when Partitioned</em>, as you must choose linearizability or total availability when a network fault occurs.</li> <li>CAP has little practical value for designing systems as it doesn't say anything about network delays, dead nodes, or other trade-offs.</li> </ul> <h6>Linearizability and network delays</h6> <ul> <li>The response times of linearizable reads and writes is very high in a network with highly variable network delays, and so weaker consistency models can be much faster.</li> </ul> <h4>Ordering Guarantees</h4> <ul> <li>The leader in single-leader replication determines the <em>order of writes</em> in the replication log, while serializability ensures that transactions behave as if executed in <em>some sequential order</em>.</li> </ul> <h5>Ordering and Causality</h5> <ul> <li>Ordering is recurring because it helps preserve <em>causality</em>. Cases where causality is important include:</li> <li>Snapshot isolation provides a snapshot consistent with causality: The effects of all operations that happened causally before that point in time are visible, but no operations that happened causally afterward are seen.</li> <li>Serializable snapshot isolation detects write skew by tracking the causal dependencies between transactions.</li> <li>A <em>causally consistent</em> system obeys the ordering imposed by causality, where chains of causally dependent operations define the causal order in the system.</li> </ul> <h6>The causal order is not a total order</h6> <ul> <li>In a linearizable system, we have a <em>total order</em> of operations: For any two operations, we can always say which happened first.</li> <li>Other consistency models offer a <em>partial order</em>: Two events are ordered if they are causally related, but they are incomparable if they are concurrent.</li> <li>There are no concurrent operations in a linearizable datastore, as there must be a single timeline along which all operations are totally ordered.</li> <li>Concurrency means that the timeline branches and merges again, and that operations on different branches are incomparable (i.e. concurrent).</li> </ul> <h6>Linearizability is stronger than causal consistency</h6> <ul> <li>Linearizability ensures causality, which is what makes linearizable systems simple to understand and appealing.</li> <li>Causal consistency is the strongest consistency model that does not slow down due to network delays, and remains available in the face of network failures.</li> </ul> <h6>Capturing causal dependencies</h6> <ul> <li>Concurrent operations may be processed in any order, but if one operation happened before another, then they must be processed in that order on every replica.</li> <li>Causal consistency must track causal dependencies across the entire database. And to determine the causal ordering, the database must know which version of the data was read by the application.</li> </ul> <h5>Sequence Number Ordering</h5> <ul> <li>Causality is an important theoretical concept, but actually keeping track of all causal dependencies can become impracticable.</li> <li>Monotonically increasing <em>sequence numbers</em> can create a total ordering that is consistent with causality: If operation A happened before operation B, then A occurs before B in the total order.</li> </ul> <h6>Noncausal sequence number generators</h6> <ul> <li>Multiple leaders independently generating sequence numbers will not correctly capture the ordering of operations across different nodes, and are not consistent with causality.</li> </ul> <h6>Lamport timestamps</h6> <ul> <li>If each node has a unique identifier and a count of the number of operations it has processed, its Lamport timestamp is the pair <em>(counter, node ID)</em>.</li> <li>Each node and client maintains the maximum counter value it has seen so far and includes it on every request. When a node receives a request or response with a maximum counter value greater than its own counter value, it immediately increases its own counter to that maximum.</li> <li>The ordering from the Lamport timestamps is consistent with causality, because every causal dependency results in an increased timestamp.</li> <li>Version vectors can distinguish whether two operations are concurrent or whether one is casually dependent on the other, whereas Lamport timestamps enforce a total ordering.</li> </ul> <h6>Timestamp ordering is not sufficient</h6> <ul> <li>To implement something like a uniqueness constraint for usernames, it's not sufficient to have a total ordering of operations -&nbsp;you also need to know when that ordering is finalized.</li> </ul> <h5>Total Order Broadcast</h5> <ul> <li><em>Total order broadcast</em> or <em>atomic broadcast</em> is the problem of how to scale the system if the throughput is greater than a single leader can handle, and how to handle failover if the leader fails.</li> <li>Total ordering across all partitions in a partitioned database is possible, but it requires additional coordination.</li> <li>Total order broadcast is a protocol for exchanging messages between nodes and requires two safety guarantees: reliable delivery and totally ordered delivery.</li> </ul> <h6>Using total order broadcast</h6> <ul> <li>Consensus services such as ZooKeeper and etcd actually implement total order broadcast.</li> <li><em>State machine replication</em> is the use of total order broadcast for database replication: Every replica processes every write in the same order.</li> <li>Total order broadcast can be framed as creating a <em>log</em>, where delivering a message is like appending to the log.</li> <li>ZooKeeper uses total order broadcast to implement a lock service: Every request to acquire the lock is appended as a message to the log, and all messages in the log are sequentially ordered. The sequence number, or <code>zxid</code>, serves as a fencing token.</li> </ul> <h6>Implementing linearizable storage using total order broadcast</h6> <ul> <li>You can build linearizable storage on top of total order broadcast by appending a message for each write or read, and then processing the write or read upon reading the message from the log.</li> </ul> <h6>Implementing total order broadcast using linearizable storage</h6> <ul> <li>For every message you want to send through total order broadcast, increment-and-get the linearizable integer, and then attach the register value as a sequence number to the message.</li> <li>It can be proved that a linearizable compare-and-set (or increment-and-get) register and total order broadcast are both <em>equivalent to consensus</em>.</li> </ul> <h4>Distributed Transactions and Consensus</h4> <ul> <li>Consensus is needed for leader election, as all nodes need to agree on which node is the leader.</li> <li>In the <em>atomic commit problem</em>, all nodes participating in a distributed transaction need to agree on the outcome of the transaction (either all aborting or all committing).</li> </ul> <h5>Atomic Commit and Two-Phase Commit (2PC)</h5> <ul> <li>Atomicity ensures that a secondary index stays consistent with the primary data.</li> </ul> <h6>From single-node to distributed atomic commit</h6> <ul> <li>The deciding moment for whether a transaction commits or aborts is the moment at which the disk finishes writing the commit record: After this moment, the transaction is committed.</li> <li>Once a transaction has been committed on one node, it cannot be retracted if it later turns out that it was aborted on another node.</li> <li>If a transaction was allowed to abort after committing, it would violate <em>read committed isolation</em>, and any transactions that read the committed data would be based on data that was retroactively declared not to have existed.</li> </ul> <h6>Introduction to two-phase commit</h6> <ul> <li>2PC (two-phase commit) provides atomic commit in a distributed database, whereas 2PL (two-phase locking) provides serializable isolation.</li> <li>2PC requires a <em>coordinator</em>, or <em>transaction manager</em>, which is often implemented as a library within the same application process that is requesting the transaction.</li> <li>When an application is ready to commit, the coordinator begins phase 1, sending a <em>prepare</em> request to every node asking if it is able to commit.</li> <li>If all nodes reply "yes" then the coordinator sends a <em>commit</em> request in phase 2 and the commit actually takes place.</li> </ul> <h6>A system of promises</h6> <ul> <li>By replying "yes" to the coordinator, a node promises to commit the transaction without error if requested.</li> <li>The <em>commit point</em> is when the coordinator writes to its transaction log on disk whether it will go ahead with phase 2 or abort, so that its decision is preserved in case it subsequently crashes.</li> <li>If the decision was to commit, then that decision must be enforced, no matter how many retires it takes.</li> </ul> <h6>Coordinator failure</h6> <ul> <li>After a node has voted "yes" to a prepare request, it must wait to hear from the coordinator whether to commit or abort.</li> <li>If a node replies to a prepare request and then the coordinator crashes or the network fails, the node's transaction state is called <em>in doubt</em> or <em>uncertain</em>.</li> <li>Upon the coordinator recovering from a crash, it computes the status of all in-doubt transactions by reading its transaction log, and aborts any without a commit record.</li> </ul> <h5>Distributed Transactions in Practice</h5> <ul> <li>Distributed transactions are criticized for causing operational problems, killing performance, and promising more than they can deliver.</li> <li>The performance cost inherent in 2PC is due to the additional disk forcing (<code>fsync</code>) that is required for crash recovery, and the additional network round-trips.</li> <li>Database-internal distributed transactions can often work quite well, while transactions spanning heterogeneous technologies are a lot more challenging.</li> </ul> <h6>XA transactions</h6> <ul> <li>X/Open XA (for eXtended Architecture) is a C API for interfacing with a transaction coordinator. It is supported by many traditional databases and message brokers.</li> <li>In practice the transaction coordinator implementing the XA API is a library loaded into the same process as the application issuing the transaction.</li> </ul> <h6>Holding locks while in doubt</h6> <ul> <li>Database transactions take a row-level exclusive lock on any rows they modify, and serializable isolation requires acquiring a shared lock on any rows read by the transaction.</li> <li>Using 2PC, a transaction must hold onto locks throughout the entire time it is in doubt, which can cause large parts of your application to be unavailable until the in-doubt transaction is resolved.</li> </ul> <h6>Recovering from coordinator failure</h6> <ul> <li>Orphaned in-doubt transactions do occur, which hold locks and block other transactions until they are either manually committed or rolled back by an administrator.</li> </ul> <h6>Limitations of distributed transactions</h6> <ul> <li>When the transaction coordinator is part of the application server, the application servers are no longer stateless because the coordinator logs are a critical part of the durable system state.</li> <li>Distributed transactions tend to <em>amplify failures</em>, because if any part of the system is broken then the distributed transaction also fails.</li> </ul> <h5>Fault-Tolerant Consensus</h5> <ul> <li>Consensus is where one or more nodes may <em>propose</em> values, and a consensus algorithm <em>decides</em> on one of those values.</li> <li>The algorithm must have uniform agreement (no two nodes decide differently), integrity (no node decides twice), validity (a node can decide only on a proposed value), and termination (every node that does not crash eventually decides a value).</li> <li>The termination property formalizes the idea of fault tolerance, as the consensus algorithm must make progress.</li> <li>Termination is a liveness property, whereas uniform agreement, integrity, and validity are all safety properties.</li> <li>Any consensus algorithm requires at least a majority of nodes to be functioning correctly in order to assure termination. This majority can safely form a quorum.</li> <li>A large-scale outage can stop the consensus system from being able to process requests, but it cannot corrupt the system by causing it to make invalid decisions.</li> </ul> <h6>Consensus algorithms and total order broadcast</h6> <ul> <li>Total order is equivalent to repeated rounds of consensus: In each round, nodes propose the message that they want to send next, and then decide on the next message to be delivered in the total order.</li> </ul> <h6>Epoch numbering and quorums</h6> <ul> <li>Consensus protocols like Raft and Paxos define an <em>epoch number</em>, and guarantee that within each epoch the leader is unique.</li> <li>Every time the leader is thought to be dead, a vote is started among the nodes to elect a new leader. This election is given an incremented epoch number.</li> <li>The quorums for voting for a leader and for voting on a leader's proposal must overlap. So if a vote on a proposal succeeds, then at least one of the nodes that voted for it must have also participated in the most recent leader election.</li> <li>If the vote on a proposal doesn't reveal any higher-numbered epoch, the current leader can conclude that no leader election with a higher epoch number has happened, and so it is still the leader.</li> </ul> <h6>Limitations of consensus</h6> <ul> <li>The process by which nodes vote on proposals before they are decided is a kind of synchronous replication.</li> <li>Most consensus algorithms assume a fixed set of nodes that participate in voting, meaning you cannot add or remove nodes in the cluster.</li> <li>In geographically distributed systems, nodes can falsely believe the leader to have failed because of a transient network issue, leading to frequently leader elections that degrade performance.</li> </ul> <h5>Membership and Coordination Services</h5> <ul> <li>ZooKeeper and etc are designed to hold small amounts of data that fit entirely in memory, which is replicated across all nodes using a fault-tolerant total order broadcast algorithm.</li> <li>Using an atomic compare-and-set operation, you can implement a lock in ZooKeeper: If several nodes concurrently try to perform the same operation, only one will succeed.</li> <li>ZooKeeper provides a monotonically increasing fencing token by totally ordering all operations and giving each one a monotonically increasing transaction ID (<code>zxid</code>) and version number (<code>cversion</code>).</li> </ul> <h6>Allocating work to nodes</h6> <ul> <li>Leader election and assigning partitioned resources to nodes can be achieved by judicious use of atomic operations, ephemeral nodes, and notifications in ZooKeeper.</li> <li>ZooKeeper provides a way of "outsourcing" some of the work of coordinating nodes (consensus, operation ordering, and failure detection) to an external service.</li> </ul> <h6>Service discovery</h6> <ul> <li>ZooKeeper, etc, and consul are often used for <em>service discovery</em>, which returns the set of IP addresses that are running a given service.</li> <li>Replicas that asynchronously receive the log of all decisions of the consensus algorithm but don't participate in the voting can serve read requests that don't need to be linearizable.</li> </ul> <h3>Chapter 10: Batch Processing</h3> <ul> <li>There are three types of systems:</li> <li>Services (online systems), where servers wait for a request to arrive and then attempt to send a response back as quickly as possible. Availability and response time are most important metrics.</li> <li>Bach processing (offline systems), where a job processes some large amount of input data to create output data. Throughput is the most important metric.</li> <li>Stream processing (near-realtime systems), where a stream job operates on events shortly after they happen.</li> </ul> <h4>Batch Processing with Unix Tools</h4> <h5>Simple Log Analysis</h5> <h6>Sorting versus in-memory aggregation</h6> <ul> <li>If data to sort exceeds available memory, chunks can be sored in memory and written to disk as segment files, and then multiple sorted segments can be merged together into a larger sorted file.</li> <li>The <code>sort</code> utility handles larger-than-memory datasets by spilling to disk, and automatically parallelizes sorting across multiple CPU cores.</li> </ul> <h5>The Unix Philosophy</h5> <ul> <li>The philosophy described in 1978 says: Make each program do one thing well, and expect the output of every program to be the input to another, as yet unknown, program.</li> <li>A Unix shell like <code>bash</code> allows us to easily <em>compose</em> small programs into surprisingly powerful data processing jobs.</li> </ul> <h6>A uniform interface</h6> <ul> <li>In Unix, all programs use a file descriptor as the input/output interface. That file is just an ordered sequence of bytes.</li> </ul> <h6>Separation of logic and wiring</h6> <ul> <li>Pipes allow you to attach the <code>stdout</code> of one process to the <code>stdin</code> of another process with an in-memory buffer, without writing all intermediate data to disk.</li> <li>Separating the input/output wiring from the program logic makes it easier to compose small tools into bigger systems.</li> </ul> <h6>Transparency and experimentation</h6> <ul> <li>Unix tools are successful in part because it is easy to see what is going on:</li> <li>The input files are normally treated as immutable, and so you can run the commands repeatedly.</li> <li>You can pipe the output into <code>less</code> and look to see if it has the expected form at any point.</li> <li>By writing the output of one stage to a file and using that file as the input to the next stage, you can restart the later stage without re-running the entire pipeline.</li> </ul> <h4>MapReduce and Distributed Filesystems</h4> <ul> <li>Hadoop Distributed File System (HDFS) is based on the shard-nothing principle requiring no special hardware, only computers connected by a conventional network.</li> <li>HDFS consists of a daemon process running on each machine for I/O, and a central server called NameNode to track which file blocks are stored on which machine.</li> </ul> <h5>MapReduce Job Execution</h5> <ul> <li>The <em>mapper</em> of a MapReduce job can generate any number of key-value pairs for a given input.</li> <li>The <em>reducer</em> processes the collection of values for a given key and can produce output records.</li> <li>The role of the mapper is to prepare the data by putting it in a form that is suitable for sorting, and the role of the reducer is to process the data that has been sorted.</li> </ul> <h6>Distributed execution of MapReduce</h6> <ul> <li>The mapper and reducer do not know where each input record is coming from or output record is going to, so the framework can handle those complexities.</li> <li>The MapReduce framework <em>puts computation near the data</em> by attempting to run each mapper on one of the machines that stores the record of the input file.</li> <li>While the number of map tasks is determined by the number of input file blocks, the number of reduce tasks is configured by the job author.</li> <li>Each map task partitions its output by reducer, based on the hash of the key. Each of these partitions is written to a sorted file on the mapper's local disk.</li> <li>The reducers download the files of sorted key-value pairs for their partition. The process of partitioning by reducer, sorting, and copying data partitions from mappers to reducers is known as the <em>shuffle</em>.</li> <li>When the reduce task merges the files from the mapper it preserves their sorted order, and so the values are iterated over in sorted order.</li> </ul> <h6>MapReduce workflows</h6> <ul> <li>It's common for MapReduce jobs to be chained together into <em>workflows</em>, where the output of one job becomes the input to the next job.</li> <li>Chained MapReduce jobs are less like pipelines of Unix commands and more like a sequence of commands where each command's output is written to a temporary file, which the next command reads from.</li> </ul> <h5>Reduce-Side Joins and Grouping</h5> <ul> <li>Associations between records include a <em>foreign key</em> in a relational model, <em>document reference</em> in a document model, or an <em>edge</em> in a graph model.</li> <li>A join is necessary whenever you have some code that needs to access records on both sides of that association.</li> <li>In the context of batch processing, a join means resolving <em>all</em> occurrences of some association within a data set, and is not limited to a subset of records.</li> </ul> <h6>Example: analysis of user activity events</h6> <ul> <li>To achieve good throughput in a batch process, computation must be (as much as possible) local to one machine. Requests over the network on a per-record basis is too slow.</li> <li>When data must be joined together, it should be co-located on the same machine.</li> </ul> <h6>Sort-merge joins</h6> <ul> <li>A <em>secondary sort</em> ensures that a reducer first processes all records on one side of the association first, and then all records on the other side.</li> <li>A sort-merge join has its name because mapper output is sorted by key, and the reducers then merge together the sorted lists of records from both sides of the join.</li> </ul> <h6>Bringing related data together in the same place</h6> <ul> <li>Having lined up all the data in advance, the reducer can be a simple, single-threaded piece of code that churns through records with high throughput and low memory overhead.</li> <li>Because MapReduce separates the physical network communication aspects of computation from the application logic, it can transparently retry failed tasks without affecting the application logic.</li> </ul> <h6>GROUP BY</h6> <ul> <li>The simplest way of implementing <code>GROUP BY</code> in MapReduce is to configure the mappers so that all produced key-value pairs use the desired grouping key, and then aggregate them in the reducer.</li> </ul> <h6>Handling skew</h6> <ul> <li>Keys with a disproportionate number of values, or <em>hot keys</em>, can lead to <em>skew</em> or <em>hot spots</em> where one reducer must process significantly more records than the others.</li> <li>This is problematic because any subsequent jobs must wait for the slowest reducer to complete before they can start.</li> <li>To aggregate values for a hot key, a first stage of aggregation can be distributed across multiple reducers, and then a second stage combines those values to create a single value for the key.</li> </ul> <h5>Map-Side Joins</h5> <ul> <li>A <em>reduce-side join</em> where join logic belongs to the reducers is flexible, but all that sorting, copying to reducers, and merging of reducer inputs can be expensive.</li> <li>A <em>map-side join</em> makes assumptions about the size, sorting, and partitioning of input data, but removes the needs for reducers and sorting.</li> </ul> <h6>Broadcast hash joins</h6> <ul> <li>In a <em>broadcast hash join</em>, all the data on one side of the association is loaded into memory and is used directly by the mapper.</li> <li>The term <em>broadcast</em> refers to how the small input is "broadcast" to all partitions of the large input, and <em>hash</em> refers to its use of a hash table.</li> </ul> <h6>Partitioned hash joins</h6> <ul> <li>In a <em>partitioned hash join</em> all records you might want to join belong to the same partition, and so each mapper can read only one partition from each of the input data sets.</li> <li>This only works if both of the join's inputs have the same number of partitions, with records assigned to partitions based on the same key and the same hash function.</li> </ul> <h6>Map-side merge joins</h6> <ul> <li>If the input data sets are not only partitioned in the same way but also sorted by the same key, the mapper can perform the merging operation normally done by a reducer.</li> </ul> <h6>MapReduce workflows with map-side joins</h6> <ul> <li>The output of a reduce-side join is partitioned and sorted by the join key, whereas the output of a map-side join is partitioned and sorted in the same way as the large input.</li> </ul> <h5>The Output of Batch Workflows</h5> <ul> <li>Batch processing is closer to analytics in that it typically scans over large portions of an input data set, but its output is often not a report but some other structure.</li> </ul> <h6>Key-value stores as batch process output</h6> <ul> <li>To persist the output of a batch process into a database for querying, writing one record at a time to the database from the mapper or reducer is a bad idea:</li> <li>Making a network request for every single record is orders of magnitude slower than the normal throughput of a batch task.</li> <li>If all mappers or reducers concurrently write to the same database, then that database can become overwhelmed.</li> <li>Partially completed jobs may be visible to other systems, while MapReduce provides a clean all-or-nothing guarantee for job output.</li> <li>A better solution is to create a new database inside the batch job and write it as files to the job's output directory in the distributed filesystem.</li> </ul> <h6>Philosophy of batch process outputs</h6> <ul> <li>By treating inputs as immutable and avoiding side-effects, batch jobs not only achieve good performance but also become easier to maintain.</li> <li>The idea of being able to recover from buggy code has been called <em>human fault tolerance</em>.</li> <li>The automatic retry of a map or reduce task is safe only because inputs are immutable and outputs from failed tasks are discarded by the MapReduce framework.</li> </ul> <h5>Comparing Hadoop to Distributed Databases</h5> <ul> <li>Hadoop is somewhat like a distributed version of Unix, where HDFS is the filesystem and MapReduce is a quirky implementation of a Unix process.</li> </ul> <h6>Diversity of storage</h6> <ul> <li>In practice, making data available quickly – even if in a difficult-to-use, raw format –&nbsp;is more valuable than trying to decide on the ideal data model up front.</li> <li>A "data lake" or "enterprise data hub" collects data in its raw form, allowing its collection to be expedited.</li> <li>Instead of forcing the producer of a dataset to bring it into a standardized format, the interpretation of the data becomes the consumer's problem.</li> <li>The <em>sushi principle</em> says raw data is better, as dumping the raw data allows a consumer to transform the data into whatever form is ideal for its purposes.</li> </ul> <h6>Diversity of processing models</h6> <ul> <li>MapReduce is too limiting or performs too badly for some types of processing, and so various other processing models have been developed on top of Hadoop.</li> <li>These various processing models can all be run on a single shared-use cluster of machines, all accessing the same files on the distributed system.</li> <li>The Hadoop ecosystem includes random-access OLTP databases such as HBase, and MPP (Massively Parallel Processing) databases such as Impala.</li> </ul> <h6>Designing for frequent faults</h6> <ul> <li>Unlike MPP databases, MapReduce can tolerate the failure of a MapReduce map or reduce task by retrying it, and it eagerly writes data to disk for fault tolerance and in case the data cannot fit entirely in memory.</li> <li>MapReduce was designed to tolerate frequent faults because MapReduce jobs at Google were run at low priority and could be preempted at any time by higher-priority processes requiring their resources.</li> </ul> <h4>Beyond MapReduce</h4> <ul> <li>MapReduce is robust, but other tools are sometimes orders of magnitude faster for other types of processing.</li> </ul> <h5>Materialization of Intermediate State</h5> <ul> <li>The process of writing out intermediate state to files is called <em>materialization</em>.</li> <li>Pipes in Unix do not fully materialize the intermediate state, but instead <em>stream</em> one command's output to the next command's input using a small in-memory buffer.</li> <li>Unlike Unix pipes, MapReduce fully materializing intermediate state has downsides:</li> <li>Having to wait until all of the preceding job's tasks have completed slows down the execution of the workflow as a whole.</li> <li>Mappers are often redundant because they just read back the same file that was just written by a reducer, and prepare it for the next stage of partitioning and sorting.</li> <li>Storing intermediate state in a distributed filesystem replicates it across several nodes, which is overkill for temporary data.</li> </ul> <h6>Dataflow engines</h6> <ul> <li><em>Dataflow engines</em> model the flow of data through several processing stages. The user-defined functions for operating on the data are called <em>operators</em>.</li> <li>Advantages over the MapReduce model include:</li> <li>Expensive work like sorting is only done in places where it is required, rather than always happening between every map and reduce stage.</li> <li>There are no unnecessary map tasks, since the work done by a mapper can often be incorporated into the preceding reduce operator.</li> <li>Because all joins and data dependencies are explicitly declared, the scheduler can make locality optimizations.</li> <li>Intermediate state between operators can be kept in memory or written to local disk, which requires less I/O than writing it to HDFS.</li> <li>Operators can start executing as soon as their input is ready instead of waiting for the entire preceding stage to finish.</li> </ul> <h6>Fault tolerance</h6> <ul> <li>If a machine fails and the intermediate state on a machine is lost, the dataflow engine can recompute it from the preceding stages.</li> <li>If a computation is not deterministic, then the original data is not the same as the lost data, and downstream operators must reconcile the contradictions between the old and new data.</li> <li>The solution to such nondeterministic operators is to kill the downstream operators as well, and run them again on the new data.</li> <li>If the intermediate data is much smaller than the source data, or if the computation is very CPU-intensive, it's cheaper to materialize the intermediate data than to recompute it.</li> </ul> <h6>Discussion of materialization</h6> <ul> <li>While materialized datasets on HDFS are still usually the inputs and outputs of a job, the dataflow engine saves the job from writing all intermediate state to the filesystem.</li> </ul> <h5>Graphs and Iterative Processing</h5> <ul> <li>Many graph algorithms traverse one edge at a time, joining one vertex with an adjacent one to propagate some information, and repeating until some condition is met like no more edges to follow, or some metric converging.</li> </ul> <h6>The Pregel processing model</h6> <ul> <li>The <em>Pregel</em> model, or <em>bulk synchronous processing model</em> (BSP), is a popular optimization for batch processing graphs.</li> <li>One vertex can "send a message" to another vertex, usually along an edge in a graph – similar to how mappers can conceptually send messages to reducers by key.</li> <li>In each iteration, a function is called for each vertex, passing the function all the messages that were sent to that vertex – again like a reducer.</li> </ul> <h6>Fault tolerance</h6> <ul> <li>Pregel implementations guarantee that messages are processed exactly once at their destination vertex in the following iteration.</li> </ul> <h6>Parallel execution</h6> <ul> <li>Graph algorithms often have a lot of cross-machine communication overhead, as the intermediate state is often bigger than the original graph.</li> <li>If your graph can fit in memory on a single computer, it's likely that a single-machine or even single-threaded algorithm will outperform a distributed batch process.</li> </ul> <h5>High Level APIs and Languages</h5> <ul> <li>Now that MapReduce is robust at scale, attention has turned to improving the programming model, improving efficiency, and broadening the set of problems that these technologies can solve.</li> </ul> <h6>The move toward declarative query languages</h6> <ul> <li>The freedom to run arbitrary code is what has long distinguished batch processing systems of MapReduce heritage from MPP databases.</li> <li>By incorporating declarative aspects into high-level APIs, batch processing frameworks can use query optimizers to achieve comparable performance with MPP databases.</li> </ul> <h6>Specialization for different domains</h6> <ul> <li>As batch processing systems gain high-level declarative operators, and MPP databases become more programmable, the two are beginning to look more alike.</li> </ul> <h4>Summary</h4> <ul> <li>Distributed batch processing systems need to solve partitioning, where all related data (e.g. all records with the same key) are brought together in the same place, and fault tolerance.</li> <li>Callback functions like mappers and reducers are assumed to be stateless and without side-effects, allowing the framework to hide some of the hard distributed systems problems behind its abstraction.</li> </ul> <h3>Chapter 11: Stream Processing</h3> <ul> <li>A lot of data is unbounded because it changes gradually over time. Batch processors just artificially divide the data into chunks of fixed duration.</li> <li>The problem with daily batch processes is that changes in the input are only reflected in the output a day later, which is too slow for many impatient users.</li> <li>A "stream" refers to data that is incrementally made available over time.</li> </ul> <h4>Transmitting Event Streams</h4> <ul> <li>In a stream processing context, a record is commonly referred to as an <em>event</em>.</li> <li>An event is generated once by a <em>producer</em> or <em>publisher</em>, and is processed by multiple <em>subscribers</em> or <em>recipients</em>. Related events are grouped together into a <em>topic</em> or <em>stream</em>.</li> <li>When moving toward continual processing with low delays, polling can become expensive; instead it's better for consumers to be notified when new events appear.</li> </ul> <h5>Messaging Systems</h5> <ul> <li>Two questions are worth asking in a publish/subscribe model:</li> <li>What happens if producers send messages faster than consumers can process them? There are three options: dropping messages, buffering messages, and applying backpressure.</li> <li>What happens if nodes crash or temporarily go offline? If you can lose messages then you can probably get higher throughput and lower latency on the same hardware.</li> </ul> <h6>Direct messaging from producers to consumers</h6> <ul> <li>StatsD uses unreliable UDP messaging for collecting metrics from all machines on the network and monitoring them.</li> <li>Direct messaging systems generally require the application code to be aware of the possibility of message loss.</li> <li>If a consumer is offline, it may miss messages that were sent while it is unreachable.</li> </ul> <h6>Message brokers</h6> <ul> <li>A <em>message broker</em> or <em>message queue</em> is a database optimized for handling message streams.</li> <li>Centralizing data in the broker allows clients to come and go, and the question of durability is moved to the broker instead.</li> </ul> <h6>Message brokers compared to databases</h6> <ul> <li>Databases usually keep data until it is explicitly deleted, whereas most message brokers automatically delete a message when it has been successfully delivered to consumers.</li> <li>Since message brokers quickly delete messages, it's assumed that their queues are short.</li> </ul> <h6>Multiple consumers</h6> <ul> <li>Given multiple consumers reading messages in the same topic, two patterns of messaging are used:</li> <li>Load balancing: Each message is delivered to one of the consumers, and you can add consumers to parallelize the processing.</li> <li>Fan-out: Each message is delivered to <em>all</em> of the consumers, allowing several independent consumers to consume the same topic of messages without affecting each other.</li> <li>The two patterns can be combined, e.g. two separate groups of consumers can subscribe to the same topic.</li> </ul> <h6>Acknowledgments and redelivery</h6> <ul> <li>Message brokers use <em>acknowledgments</em> where a client must explicitly tell the broker when it has finished processing a message so that it can be removed from the queue.</li> <li>The combination of load balancing and redelivery inevitably leads messages to be reordered. To avoid the issue, you can disable load balancing by enforcing a separate queue per consumer.</li> </ul> <h5>Partitioned Logs</h5> <ul> <li>Receiving a message is destructive if the acknowledgment causes it to be deleted from the broker, so you cannot run the same consumer again and get the same result.</li> <li>When adding a new consumer, it typically only starts receiving messages sent after the time it was registered, and so any prior messages cannot be recovered.</li> <li>A <em>log-based message broker</em> combines the durable storage approach of databases with the low-latency notification facilities of messaging.</li> </ul> <h6>Using logs for message storage</h6> <ul> <li>A topic is a group of partitions that all carry messages of the same type.</li> <li>Within each partition, the broker assigns a monotonically increasing sequence number, or <em>offset</em>, to every message. There is no total ordering of messages across partitions.</li> </ul> <h6>Logs compared to traditional messaging</h6> <ul> <li>To load balance across a group of consumers, the broker can assign entire partitions to each consumer in a consumer group, as opposed to assigning individual messages.</li> <li>Each consumer consumes <em>all</em> the messages in the partitions it has been assigned. This has some downsides:</li> <li>The number of consumers in a group can be at most the number of log partitions in that topic.</li> <li>If a single message is slow to process, it holds up the processing of subsequent messages in that partition.</li> </ul> <h6>Consumer offsets</h6> <ul> <li>If a consumer node fails, another node in the consumer group is assigned the failed consumer's partitions, and it starts consuming messages at the last recorded offset.</li> </ul> <h6>Disk space usage</h6> <ul> <li>To reclaim disk space, the log is divided into segments, and periodically the oldest segments are deleted or moved to archive storage.</li> <li>The log therefore implements a bounded-size buffer, also known as a <em>circular buffer</em> or <em>ring buffer</em>, that discards old messages when it gets full.</li> </ul> <h6>When consumers cannot keep up with producers</h6> <ul> <li>If a consumer falls so far behind that the messages it requires are older than what is retained on disk, then it cannot read those messages and they are effectively dropped.</li> <li>You can monitor how far a consumer is behind the head of the log, or its lag, and raise an alert if it falls behind significantly.</li> </ul> <h4>Databases and Streams</h4> <ul> <li>The fact that something was written to the database is itself an event that can be captured, stored, and processed.</li> <li>A replication log is a stream of database write events, produced by the leader as it processes transactions, and consumed by replicas.</li> </ul> <h5>Keeping Streams in Sync</h5> <ul> <li>Often data is persisted somewhere and then needs to be replicated to several different places, such as a search index or cache.</li> <li>Dual writes, where the application explicitly writes to each of the systems, suffer from two problems:</li> <li>A <em>race condition</em> where upon two clients dual writing a value, one client may write to the first system first and the second system second.</li> <li>One of the writes may fail while the other succeeds, thereby leaving the data in the two systems in an inconsistent state.</li> </ul> <h5>Change Data Capture</h5> <ul> <li><em>Change data capture</em> (CDC) is the process of observing all data changes to a database and extracting them in a form in which they can be replicated to other systems.</li> </ul> <h6>Implementing change data capture</h6> <ul> <li>CDC ensures that all changes made to the system of record are also reflected in the derived data systems so that they too have an accurate copy of the data.</li> <li>A log-based message broker is well suited for transporting the change events from the source database to the derived systems, because it preserves the ordering of messages.</li> <li>Parsing the replication log of a database is a robust approach to CDC, but it comes with challenges such as handling schema changes.</li> </ul> <h6>Initial snapshot</h6> <ul> <li>If you do not have the entire log history, you must start with a consistent snapshot.</li> <li>The snapshot must correspond to a known position or offset in the change log, so that you know at which point to start applying changes after the snapshot has been processed.</li> </ul> <h6>Log compaction</h6> <ul> <li>A compaction or merging process running in the background looks for messages with the same key, and keeps only the most recent update for each key.</li> <li>The disk usage required by a compacted log depends only on the current contents of the database, and not the number of writes that have ever occurred in the database.</li> <li>To rebuild a derived data system, start a new consumer from offset 0 of the log-compacted topic, and sequentially scan over all messages in the log.</li> </ul> <h5>Event Sourcing</h5> <ul> <li><em>Event sourcing</em> stores all changes to the application state as a log of change events, but applies the idea at a different level of abstraction:</li> <li>In CDC, the application uses the database in a mutable way, updating and deleting records. The application layer can be unaware that CDC is occurring.</li> <li>In event sourcing, the application logic is built on the basis of immutable events written to an event log. Events reflect things that happened at the application level, rather than low-level state changes.</li> <li>From an application point of view it is more meaningful to record the user's actions as immutable events rather than recording the effects of those actions on a mutable database.</li> </ul> <h6>Deriving current state from the event log</h6> <ul> <li>Applications using event sourcing must transform the log of events (representing data <em>written</em> to the system) into application state that is suitable for display to the user.</li> <li>Log compaction is not possible with event sourcing, as later events do not override prior events, and so you need the full history of events to construct the final state.</li> <li>Applications using event sourcing can restore from a snapshot of the current state, but the intention is that the system can reprocess the full event log whenever required.</li> </ul> <h6>Commands and events</h6> <ul> <li>When a request from a user first arrives, it is initially a <em>command</em>: At this point it may still fail, for example because some integrity constraint is violated.</li> <li>If the validation is successful and the command is accepted, it becomes an <em>event</em>, which is durable and immutable. It becomes a <em>fact</em>.</li> <li>A consumer of the event stream cannot reject an event, as it's already an immutable part of the log. So validation of the command must happen synchronously before it becomes an event.</li> </ul> <h5>State, Streams, and Immutability</h5> <ul> <li>Mutable state and an append-only log are two sides of the same coin: The log of all changes, or the <em>changelog</em>, represents the evolution of state over time.</li> <li>You might say that the application state is what you get when you integrate an event stream over time, and a change stream is what you get when you differentiate the state by time.</li> <li>You can consider the log as truth. The database is a cache of the subset of the log, namely the contents of the latest record values in the logs.</li> </ul> <h6>Advantages of immutable events</h6> <ul> <li>If you accidentally deploy buggy code that writes bad data to a database, recovery is much harder if the code is able to destructively overwrite data.</li> </ul> <h6>Deriving several views from the same event log</h6> <ul> <li>By separating mutable state from the immutable event log, you can derive several different read-oriented representations from the same log of events.</li> <li>You can use the event log to build a separate read-optimized view for some new feature, and run it alongside the existing system until transitioning over and decommissioning the old system.</li> <li>Storing data is normally quite straightforward if you don't have to worry about how it is going to be queried, as most schema complexity comes from supporting querying the data.</li> <li>Separating the form in which data is written from the form it is read is called <em>command query responsibility segregation</em>, or CQRS.</li> </ul> <h6>Concurrency control</h6> <ul> <li>Because event sourcing and CDC is asynchronous, the user may write to a log, and then read from a log-derived view and find that their write has not yet been reflected in the read view.</li> <li>Because event sourcing requires only a single, atomic write&nbsp;– namely appending an event to a log&nbsp;– which is a simpler concurrency model than using multi-object transactions to update data in multiple places.</li> </ul> <h6>Limitations to immutability</h6> <ul> <li>Workloads with many updates and deletes on comparatively few entities may cause the immutable history to grow prohibitively large.</li> <li>Given a large history of changes, the performance of compaction and garbage collection becomes crucial for operational robustness.</li> <li>Immutability is also at odds with having to delete the data for administrative or legal reasons, such as GDPR.</li> </ul> <h4>Processing Streams</h4> <ul> <li>There are three options for processing a stream:</li> <li>You can take the data in the events and write it to a database, cache, search index, or similar storage system, from where it can then be queried by other clients.</li> <li>You can push the events to the users in some way, so that a human is the ultimate consumer of a stream.</li> <li>You can process one or more input streams to produce one or more output streams.</li> <li>A piece of code that processes one or more streams to produce other, derived streams is called an <em>operator</em> or a <em>job</em>.</li> </ul> <h5>Uses of Stream Processing</h5> <h6>Complex event processing</h6> <ul> <li><em>Complex event processing</em> (CEP) allows specifying rules to search for certain patterns of events in a stream, similar to a regular expression operating on a string.</li> <li>CEP systems often use a high-level declarative query language like SQL to describe patterns of events for detection.</li> <li>A processing engine maintains a state machine that performs the required matching on the input streams, and emits a <em>complex event</em> with the details of a matching event pattern.</li> <li>CEP engines function in an manner opposite to normal databases: The queries are stored long-term, and the data that they operate on is transient.</li> </ul> <h6>Stream analytics</h6> <ul> <li>Analytics is less interested in finding specific event sequences, and is more oriented toward aggregations and statistical metrics over a large number of events.</li> <li>The time interval over which you aggregate is known as a <em>window</em>.</li> <li>Probabilistic algorithms produce approximate results, but have the advantage of requiring significantly less memory in the stream processor than exact algorithms.</li> </ul> <h6>Search on streams</h6> <ul> <li>Running in a manner opposite to normal search engines, searching a stream requires storing the queries, and the documents they search over are transient.</li> </ul> <h5>Reasoning About Time</h5> <ul> <li>Many stream processing frameworks use the local system clock on the processing machine (or <em>processing time</em>) to determine windowing, but this breaks down if there is significant processing lag.</li> </ul> <h6>Event time versus processing time</h6> <ul> <li>Processing may be delayed from queueing, network faults, contention in the broker or processor, a restart of the stream consumer, or reprocessing of past events.</li> <li>Confusing event time and processing time leads to bad data.</li> </ul> <h6>Knowing when you're ready</h6> <ul> <li>You can never be sure when you've received all of the events for a given window, or whether there are some events still to come.</li> <li>You have two options for handling <em>straggler</em> events that arrive after the window has already been declared complete: Ignore them, or publish a corrective value with stragglers included.</li> </ul> <h6>Whose clock are you using, anyway?</h6> <ul> <li>Assigning timestamps to events is even more difficult when events can be buffered at several points in the system, such as on a mobile device that is offline and later reconnects.</li> <li>To estimate the offset between the server and device clocks, subtract the timestamp at which the event was sent to the server according to the device clock from the timestamp at which the event was received by the server according to the server clock.</li> <li>Applying this computed offset to the event timestamp allows you to estimate the true time at which the event occurred.</li> </ul> <h6>Types of windows</h6> <ul> <li>A <em>tumbling window</em> has a fixed length, and every event belongs to exactly one window.</li> <li>A <em>hopping window</em> has a fixed length, but windows may overlap to provide smoothing. You can implement this by computing 1-minute tumbling windows, and then aggregating over several adjacent windows.</li> <li>A <em>sliding window</em> contains all events that occur within some interval of each other. You can implement this by buffering events sorted by time, and removing old events that have expired from the window.</li> <li>A <em>session window</em> has no fixed duration, but instead groups together all events for the same user that occur closely together in time. The window ends when the user is inactive for some time.</li> </ul> <h5>Stream Joins</h5> <ul> <li>The fact that new events can appear anytime on a stream makes joins on streams more challenging than in batch jobs.</li> </ul> <h6>Stream-stream join (window join)</h6> <ul> <li>The stream processor maintains state that is updated whenever an event occurs, which is referenced upon processing subsequent events.</li> </ul> <h6>Stream-table join (stream enrichment)</h6> <ul> <li><em>Enriching</em> a stream means augmenting its events with additional data queried from a database.</li> <li>If the stream processor has a local copy of the database (similar to map-side joins), then that database can be kept up to date using change data capture.</li> <li>Unlike a stream-stream join, a stream-table join uses a window that reaches back to the beginning of time (a conceptually infinite window) with newer versions of records overwriting old ones.</li> </ul> <h6>Table-table join (materialized view maintenance)</h6> <ul> <li>A table-table join is a stream process that maintains a materialized join between two tables.</li> </ul> <h6>Time-dependence of joins</h6> <ul> <li>All three joins above require the stream processor to maintain some state based on one join input, and to query that state on messages from the other input.</li> <li>The ordering of the events that maintain the state is important, and in a partitioned log, there is no ordering guarantee of events across different streams or partitions.</li> <li>Because state changes over time, and you join with some state, you must ask yourself what point in time you are using for the join.</li> <li>If the ordering of events across streams is undetermined, then the join is nondeterministic, and the events may be interleaved differently when you run the job again.</li> <li>This issue is known as a <em>slowly changing dimension</em> (SCD) in data warehouses and is often addressed by using a unique identifier for a particular version of a joined record.</li> <li>A unique identifier makes the join deterministic, but log compaction is no longer possible because all versions of the records in the table must be retained.</li> </ul> <h5>Fault Tolerance</h5> <ul> <li>The batch processing approach to fault tolerance exposes <em>exactly-once semantics</em>, where it appears as though every record was processed exactly once.</li> <li>In stream processing, waiting until a task is finished before making its output visible is not an option, because a stream is infinite.</li> </ul> <h6>Microbatching and checkpointing</h6> <ul> <li><em>Microbatching</em> breaks the stream into small blocks and treats each block like a miniature batch process.</li> <li>With microbatching, small batches incur greater scheduling and coordination overhead, while larger batches mean a longer delay before results of the stream processor become available.</li> <li>An alternative is to periodically generate rolling checkpoints of state and write them to durable storage.</li> <li>Once the output leaves the stream processor, the framework cannot discard the output of a failed batch. Restarting a failed task causes its external side-effects to happen twice, regardless of checkpointing or microbatching.</li> </ul> <h6>Atomic commit revisited</h6> <ul> <li>To give the appearance of exactly-once processing in the presence of faults, all outputs and side-effects of processing an event must happen if and only if the processing is successful.</li> <li>More restricted environments, which do not attempt to persist data across heterogeneous environments, can implement an atomic commit facility efficiently.</li> </ul> <h6>Idempotence</h6> <ul> <li>An idempotent operation is one that you can perform multiple times, and it has the same effect as if you had performed it only once.</li> <li>Using idempotence relies on restarting a task to replay the same messages in the same order, the processing must be deterministic, and no other node may concurrently update the same value.</li> </ul> <h6>Rebuilding state after a failure</h6> <ul> <li>Any stream process that requires state (e.g. any windowed aggregations or tables for joins) must ensure that the state can be recovered after a failure.</li> <li>A performant solution is to keep the state local to the stream processor, and then to replicate it periodically.</li> <li>If the state is a local replica of a database, maintained by CDC, then the database can be rebuilt from the log-compacted change stream.</li> </ul> <h4>Summary</h4> <ul> <li>There are three types of joins that may appear in stream processes:</li> <li><em>Stream-stream joins</em>: Both input streams consist of activity events, and the join operator searches for related events that occur in some window of time.</li> <li><em>Stream-table joins</em>: One input stream consists of activity events, while the other is a database changelog that keeps a local copy of a database up to date. This emits enriched activity events.</li> <li><em>Table-table joins</em>: Both input streams are database changelogs. The result is a stream of changes to the materialized view of the join between the two tables.</li> </ul> <h3>Chapter 12: The Future of Data Systems</h3> <h4>Data Integration</h4> <ul> <li>There is unlikely one piece of software that is suitable for all circumstances in which data is used, and so we must cobble together several pieces of software to provide the application's functionality.</li> </ul> <h5>Combining Specialized Tools by Deriving Data</h5> <ul> <li>As the number of different representations of the data increases, the integration problem becomes harder.</li> <li>The need for data integration often only becomes apparent when you zoom out and consider the dataflows across an entire organization.</li> </ul> <h6>Reasoning about dataflows</h6> <ul> <li>You must have clarity on the inputs and outputs, namely where data is written first, and which representations are derived from which sources.</li> <li>By funneling all user input through a single system that decides on an ordering for all writes, it's easier to derive other representations of the data by processing the writes in the same order.</li> <li>Whether to use change data capture or an event sourcing log is less important than simply the principle of deciding on a total order.</li> </ul> <h6>Derived data versus distributed transactions</h6> <ul> <li>Distributed transactions decide on an ordering of writes by using locks for mutual exclusion, while CDC and event sourcing use a log for ordering.</li> <li>Distributed transactions use atomic commit to ensure that changes take effect only once, while log-based systems are often based on deterministic retry and idempotence.</li> <li>Transactions systems usually provide linearizability and reading your own writes, while derived data systems are asynchronous and do not offer such guarantees.</li> </ul> <h6>The limits of total ordering</h6> <ul> <li>As systems scale, limitations of total ordering begin to emerge:</li> <li>To scale you may need to partition the log across multiple machines, but the order of events across two different partitions is ambiguous.</li> <li>If servers are are spread across multiple geographically distributed datacenters, there is an undefined ordering of events between events in two different datacenters.</li> <li>When two events originate in different micro-services, then there is no defined order for those events.</li> <li>Deciding on a total order of events is known as a <em>total order broadcast</em>, which is equivalent to consensus. It's an open research problem to design consensus algorithms that work well beyond a single node in a geographically distributed setting.</li> </ul> <h6>Ordering events to capture causality</h6> <ul> <li>Where there is no causal link between events, the lack of a total order is not a big problem since concurrent events can be ordered arbitrarily.</li> <li>Logical timestamps can provide total ordering without coordination, so they may help in cases where total order broadcast is not feasible.</li> <li>If you can log an event to record the state of the system the user saw before making a decision, and give that event a unique identifier, then any later events can refer to that event identifier to record the causal dependency.</li> </ul> <h5>Batch and Stream Processing</h5> <ul> <li>The goal of data integration is to ensure that data ends up in the right form in all the right places.</li> </ul> <h6>Maintaining derived state</h6> <ul> <li>No matter what the derived data is, it's helpful to think in terms of data pipelines that derive one thing from another, pushing state changes in one system through functional application code and applying the effects to derived systems.</li> <li>Asynchrony is what makes systems based on event logs robust, as a fault in one part of the system is contained locally.</li> </ul> <h6>Reprocessing data for application evolution</h6> <ul> <li>With reprocessing it is possible to re-structure a dataset into a completely different model in order to better serve new requirements.</li> <li>Derived views allow <em>gradual</em> evolution. A gradual migration has little risk as you always have a working system to go back to.</li> </ul> <h6>The lambda architecture</h6> <ul> <li>The lambda architecture records incoming data by appending immutable events to an always-growing dataset, from which read-optimized views are derived.</li> <li>A stream processor consumes the events and quickly produces an approximate update to a view. A batch processor later consumes the <em>same</em> set of events and produces a corrected version of the derived view.</li> <li>The stream processor can use fast approximate algorithms while the batch process uses slower exact algorithms.</li> <li>The lambda architecture has several practical problems:</li> <li>The same logic must be implemented in both a batch and in a stream processing framework.</li> <li>The batch pipeline and the stream pipeline outputs must be merged in order to respond to user requests.</li> <li>Batch processing on large datasets is expensive, and so the batch pipeline often needs to process incremental batches rather than reprocessing everything, which adds complexity.</li> </ul> <h4>Unbundling Databases</h4> <ul> <li>Relational databases want to give programmers a high-level abstraction that hide the complexities of data structures on disk, concurrency, crash recovery, and so on.</li> <li>The NoSQL movement wants to apply a Unix-esque approach of low-level abstractions to the domain of distributed OLTP data storage.</li> </ul> <h5>Composing Data Storage Technologies</h5> <h6>The meta-database of everything</h6> <ul> <li>The dataflow across an entire organization looks like one huge database: Each batch, stream, or ETL process acts like a database subsystem that keeps indexes or materialized views up to date.</li> <li>There are two avenues by which different storage and processing tools can be composed into a cohesive system:</li> <li>Federated databases (unifying reads): A <em>federated database</em> or <em>polystore</em> provides a unified query interface to a wide variety of storage engines and processing methods.</li> <li>Unbundled databases (unifying writes): Data capture and event logs allow <em>unbundling</em> a database's index-maintenance features in a way that can synchronize writes across disparate technologies.</li> </ul> <h6>Making unbundling work</h6> <ul> <li>Asynchronous event logs with idempotent retries is a more robust and practical approach than distributed transactions across heterogeneous storage systems.</li> <li>Log-based integration is <em>loose coupling</em> between storage components:</li> <li>Asynchronous event streams make the system as a whole more robust to outages or performance degradation of individual components.</li> <li>Unbundling data systems allows different software components and services to be developed, improved, and maintained independently from each other by different teams.</li> </ul> <h6>Unbundling versus integrated systems</h6> <ul> <li>Unbundling allows you to combine several different databases in order to achieve good performance for a much wider range of workflows than is possible with a single piece of software.</li> <li>The advantages of unbundling and composition only come into the picture when there is no single piece of software that satisfies all your requirements.</li> </ul> <h5>Designing Applications Around Dataflow</h5> <ul> <li>The approach of unbundling databases by composing specialized storage and processing systems with application code is becoming known as the "database inside-out" approach.</li> </ul> <h6>Application code as a derivation function</h6> <ul> <li>A dataset derived from another must go through some transformation function. Examples include a secondary index, a full-text search index, a machine learning model (derived from the training data), or a cache (aggregating data in a form for rendering in the UI).</li> </ul> <h6>Separation of application code and state</h6> <ul> <li>In the typical web application model, the database acts as a kind of mutable shared variable that can be accessed synchronously over the network.</li> <li>Subscribing to changes in a database is only now emerging as a feature; historically databases have inherited a passive approach to mutable data that requires polling for changes.</li> </ul> <h6>Dataflow: Interplay between state changes and application code</h6> <ul> <li>Instead of thinking of a database as a passive variable that is updated only by the application, we must think of application code responding to state changes in one place by updating state in another place.</li> <li>Unbundling the database applies this idea to derived datasets from the primary database, including caches, full-text search indexes, machine learning, or analytics systems.</li> <li>This requires stable message ordering and fault-tolerant message processing, but those are less stringent demands than those imposed by distributed transactions.</li> </ul> <h6>Stream processors and services</h6> <ul> <li>The advantage of a service-oriented architecture over a single monolithic application is primarily organizational scalability through loose coupling.</li> <li>Instead of one service querying another service for data, it can subscribe to to state changes from the other service and query a local database for improved speed and reliability.</li> </ul> <h5>Observed Derived State</h5> <ul> <li>The write path between a primary database and its derived dataset and the read path between that derived dataset and the point it is consumed represents the whole journey of the data.</li> <li>Framing the journey in terms of functional programming languages, the write path is similar to eager evaluation, while the read path is similar to lazy evaluation.</li> <li>The derived dataset is where the write path and read path meet, and represents a tradeoff between the amount of work that must be done at write time and the amount that must be done at read time.</li> </ul> <h6>Materialized views and caching</h6> <ul> <li>Caches, indexes, and materialized views allow us to do more work on the write path and less work on the read path, thereby shifting the boundary between the write path and read path.</li> </ul> <h6>Stateful, offline-capable clients</h6> <ul> <li>When we move from stateless clients to a model where end-user devices maintain state, we can think of the on-device state as a <em>cache of state on the server</em>.</li> </ul> <h6>Pushing state changes to clients</h6> <ul> <li>Actively pushing state changes all the way to client devices means extending the write path all the way to the end user.</li> <li>When a client is first initialized, it would still need to use a read path to get its initial state, but thereafter it could rely on a stream of state changes sent by the server.</li> </ul> <h6>End-to-end event streams</h6> <ul> <li>Extending the write path all the way to the end user requires moving away from request/response interaction and toward a publish/subscribe dataflow.</li> </ul> <h6>Reads are events too</h6> <ul> <li>We can treat reads as events, and send them to the same stream processor as writes; the processor can respond to a read event by emitting the result of the read on the output stream.</li> <li>This performs a stream-table join between the stream of read queries and the database.</li> <li>Recording a log of read events has benefits with regard to tracking causal dependencies and data provenances, as you can reconstruct what a user saw before making some decision.</li> </ul> <h6>Multi-partition data processing</h6> <ul> <li>Representing queries as events and collecting responses from streams enables executing queries across several partitions, leveraging the infrastructure for message routing, partitioning, and joining already provided by stream processors.</li> </ul> <h4>Aiming for Correctness</h4> <ul> <li>We want to build applications that are reliable and <em>correct</em>, i.e. programs whose semantics are well defined and understood, even in the presence of faults.</li> <li>Serializability and atomic commits provide strong assurances of correctness, but typically only work in a single datacenter, and they limit the scale of fault-tolerance properties you can achieve.</li> </ul> <h5>The End-to-End Argument for Databases</h5> <h6>Exactly-once execution of an operation</h6> <ul> <li><em>Exactly-once</em> means arranging the computation such that the final effect is the same as if no faults had occurred, even if the operation was retried due to some fault.</li> <li>One of the most effective approaches is to make the operation <em>idempotent</em>.</li> </ul> <h6>Duplicate suppression</h6> <ul> <li>Even if you suppress duplicate transactions between the database client and server, you must still worry about the network between the end-user device and the application server.</li> </ul> <h6>Uniquely identifying requests</h6> <ul> <li>Relational databases can generally maintain a uniqueness constraint correctly (such as on an idempotence token), even at weak isolation levels.</li> </ul> <h6>The end-to-end argument</h6> <ul> <li>The <em>end-to-end argument</em> says some problems can be completely and correctly implemented only with the help of the endpoints of the communication system; providing a solution as a feature of the communication system is not possible.</li> <li>In such circumstances, low-level reliability features are not by themselves sufficient to ensure end-to-end correctness.</li> </ul> <h6>Applying end-to-end thinking in data systems</h6> <ul> <li>Reasoning about concurrency and partial failure is difficult and counterintuitive, and so most application-level mechanisms likely do not work correctly, thereby resulting in lost or corrupted data.</li> </ul> <h5>Enforcing Constraints</h5> <h6>Uniqueness constraints require consensus</h6> <ul> <li>Enforcing a uniqueness constraint requires consensus, as the system must decide which one of the conflicting operations is accepted, and reject the others as violating the constraint.</li> <li>Uniqueness constraints can be scaled out by partitioning based on the value that needs to be unique.</li> <li>Asynchronous multi-master replication cannot be used in this case, as two different masters could concurrently accept conflicting writes.</li> </ul> <h6>Uniqueness in log-based messaging</h6> <ul> <li>If a log is partitioned based on the value that needs to be unique, then a stream processor can unambiguously and deterministically decide which of several conflicting operations comes first in the log.</li> <li>To scale the number of handled requests, simply increase the number of partitions.</li> <li>Because all writes that may conflict are routed to the same partition and processed sequentially, this works for many constraints other than uniqueness constraints.</li> </ul> <h6>Multi-partition request processing</h6> <ul> <li>Single-object writes are atomic in almost all data systems, and so a request appended to a log either appears in the log or it doesn't.</li> <li>By breaking down a multi-partition transaction into partitioned stages using the same end-to-end request ID, you can achieve correctness even in the presence of faults without using an atomic commit protocol.</li> </ul> <h5>Timeliness and Integrity</h5> <ul> <li>The term <em>consistency</em> conflates two requirements that are worth considering separately:</li> <li><em>Timeliness</em> means ensuring the data is in an up-to-date state. Read data may be stale but that inconsistency is only temporary.</li> <li><em>Integrity</em> means absence of corruption, and no contradictory or false data. If integrity is violated, then the inconsistency is permanent.</li> <li>Violations of timeliness are "eventual consistency" whereas violations of integrity are "permanent inconsistency."</li> </ul> <h6>Correctness of dataflow systems</h6> <ul> <li>ACID transactions usually provide both timeliness (e.g. linearizability) and integrity (e.g. atomic commit) guarantees.</li> <li>Event-based dataflow systems decouple timeliness and integrity: You forfeit timeliness because of asynchrony, but <em>exactly-once</em> or <em>effectively-once</em> semantics preserve integrity.</li> </ul> <h6>Loosely interpreted constraints</h6> <ul> <li>If an application can get away with weaker notions of uniqueness, then persisting a <em>compensating transaction</em> may undo a constraint violation.</li> <li>In many business contexts, it may be acceptable to temporarily violate a constraint and fix it up later by apologizing. The cost of the apology is a business decision.</li> <li>Such applications that tolerate optimistic writes and checking the constraint afterward do require integrity, but don't require timeliness on the enforcement of the constraint.</li> </ul> <h6>Coordinating-avoiding data systems</h6> <ul> <li>Systems that synchronize cross-partition coordination or have strict constraints reduce the number of apologies you must make due to inconsistencies, but potentially also reduce your performance and availability.</li> </ul> <h5>Trust, but Verify</h5> <ul> <li>The <em>system model</em> is the set of assumptions that certain things might go wrong, but other things won't.</li> <li>Faults are really modeled by probabilities. The question is whether violations of our assumptions happen often enough that we may encounter them in practice.</li> <li>If you have enough devices running your software, then even very unlikely things do happen.</li> </ul> <h6>Maintaining integrity in the face of software bugs</h6> <ul> <li>If the application uses the database incorrectly in some way, for example using a weak isolation level unsafely, then the integrity of the database cannot be guaranteed.</li> </ul> <h6>Don't blindly trust what they promise</h6> <ul> <li>Checking the integrity of data is known as <em>auditing</em>.</li> <li>HDFS and Amazon S3 run background checks that continually read back files, compare them to other replicas, and move files between disks, in order to mitigate the risk of silent corruption.</li> </ul> <h6>Designing for auditability</h6> <ul> <li>Mutations to database tables do not detail <em>why</em> those mutations were performed. The invocation of the application logic that decided on those mutations is transient and cannot be reproduced.</li> <li>Being explicit about dataflow makes the <em>provenance</em> of data much clearer, which makes integrity checking much more feasible.</li> </ul> <h6>The end-to-end argument again</h6> <ul> <li>Checking the integrity of data systems is best done in an end-to-end fashion: The more systems we can include in an integrity check, the fewer opportunities there are for corruption to go unnoticed at some stage of the process.</li> </ul> <h6>Tools for auditable data systems</h6> <ul> <li>Blockchains and distributed ledgers are essentially distributed databases run by mutually untrusting organizations. These replicas check each other's work and use a consensus protocol to agree on the transactions that should be executed.</li> </ul> <h4>Doing the Right Thing</h4> <ul> <li>Data may be an abstract thing, but many datasets are about people: their behavior, their interests, their identity. We must treat such data with humanity and respect.</li> </ul> <h5>Predictive Analytics</h5> <ul> <li>The criminal justice system may presume innocence until proven guilty, but automated systems can systematically and arbitrarily exclude a person from participating in society without any proof of guilt, and with little chance of appeal.</li> </ul> <h6>Bits and discrimination</h6> <ul> <li>Predictive analytics systems do not merely automate a human's decision by using software to specify the rules for when to say yes or no; instead we leave the rules themselves to be inferred from the data.</li> <li>If there is systematic bias in the input to an algorithm, the system will most likely learn and amplify that bias in its output.</li> <li>Predictive analytics systems merely extrapolate from the past; if the past is discriminatory, then they codify that discrimination.</li> </ul> <h6>Responsibility and accountability</h6> <ul> <li>As data-driven decision making becomes more widespread, we must discover how to make algorithms accountable and transparent, how to avoid reinforcing existing biases, and how to fix them when they inevitably make mistakes.</li> </ul> <h6>Feedback loops</h6> <ul> <li>Consequences such as feedback loops can be predicted by thinking about the entire system, including the people interacting with it – an approach known as <em>systems thinking</em>.</li> </ul> <h5>Privacy and Tracking</h5> <ul> <li>When a system only stores data that has been explicitly entered, then the system is performing a service for the user: The user is the customer.</li> <li>Tracking the user serves not the individual but the needs of advertisers who are funding the service. This relationship is appropriately described as <em>surveillance</em>.</li> </ul> <h6>Surveillance</h6> <ul> <li>In our attempts to make software "eat the world" we have built the greatest mass surveillance infrastructure that the world has ever seen.</li> </ul> <h6>Consent and freedom of choice</h6> <ul> <li>Most privacy policies do more to obscure than to illuminate. But without understanding what happens to their data, users cannot give any meaningful consent.</li> <li>Data is extracted from users through a one-way process, not a relationship with true reciprocity, and not a fair value exchange.</li> <li>For the user who does not consent to surveillance, the only real alternative is simply to not use the service, which may have real social cost.</li> <li>For people in a less privileged position, there is no meaningful freedom of choice: surveillance becomes inescapable.</li> </ul> <h6>Privacy and use of data</h6> <ul> <li>Privacy does not mean keeping everything secret, but having the freedom to choose which things to reveal to whom, what to make public, and what to keep secret.</li> <li>When data is extracted from people through surveillance infrastructure, privacy rights are usually not eroded but rather transferred to the data collector.</li> <li>Even if particular users cannot be personally identified from the group targeted by a particular ad, they have lost their agency about the disclosure of some intimate information.</li> <li>Whether something is "undesirable" or "inappropriate" is down to human judgment; algorithms are oblivious to such notions unless we explicitly program them to respect human needs.</li> <li>Surveillance has always existed, but it used to be expensive and manual. Trust relationships have always existed, but the are mostly governed by ethical, legal, and regulatory constraints.</li> </ul> <h6>Data as assets and power</h6> <ul> <li>Behavioral data is sometimes called "data exhaust" because it is a byproduct of users interacting with a service.</li> <li>From an economic point of view, if targeted advertising is what pays for a service, then behavioral data about people is the service's core asset.</li> <li>Because data can be abused so easily, critics have said that data is not just an asset, but a "toxic asset" or at least a "hazardous material."</li> </ul> <h6>Remembering the Industrial Revolution</h6> <ul> <li>Just as the Industrial Revolution had a dark side that needed to be managed, our transition to the information age has major problems that we must confront and solve.</li> <li>As Bruce Schneier said, data is the pollution problem of the information age, and protecting privacy is the environmental challenge.</li> </ul> https://tedneward.github.io/Research/reading/development/dynamic-instrumentation/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/dynamic-instrumentation/index.html <p><a href="http://www.dyninst.org/">Website</a> | <a href="https://github.com/dyninst/dyninst">Source</a></p> <p><a href="https://github.com/rimsa/dumpcfgs">dumpcfgs</a>: Dump (extract) CFGs statically from binary programs using DynInst API</p> <p><a href="https://arxiv.org/abs/2001.10621">Parallelizing Binary Code Analysis</a>: by Xiaozhu Meng, Jonathon M. Anderson, John Mellor-Crummey, Mark W. Krentel, Barton P. Miller, Srđan Milaković, 2020 arXiv</p> https://tedneward.github.io/Research/reading/development/ffi/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/ffi/index.html <p>The concept of an FFI is fairly straightforward: How does a language/platform "bind" (in other words, be able to call) to underlying APIs of the host environment or other libraries/languages/platforms? Examples include Java code, running on top of the JVM, being able to "call out to" native C code, such as operating system APIs or native libraries; game engines being able to call out to libraries that aren't a part of the game engine itself; and so on.</p> <p>Most of the time this requires several pieces of knowledge:</p> <ul> <li>How to dynamically load a library at runtime on the host environment (operating system)</li> <li>How to understand the "signature" and calling convention of the host/OS library or API</li> <li>How to "marshal" parameters and return values from one environment to the other, which requires knowing information about both the caller and the target's choice of binary representation of values</li> </ul> <h2>Dynamically loading libraries</h2> <p>Most FFIs (particularly those of VM-based languages) use dynamic loading of libraries, so as to reduce the need to statically-link new executables each time a new FFI invocation is needed/desired.</p> <p>Lots of this material is covered in <a href="../linking-loading">linking and loading</a> reading. Typically the two-step process is to:</p> <ol> <li>Load the library. Asking the OS to bring the library up into the process' space and make it available.</li> <li>Request the address of a given exported symbol by name. In other words, look up the function by its name, returning either the address (which we can then coerce somehow into a callable reference--function pointer, if you will) or NULL if no such name is found.</li> </ol> <h2>Call signatures</h2> <p>How do we name the exported entry point? Does the entry point have any metadata describing the parameters and/or return type?</p> <h4><a href="/languages/c">C</a></h4> <p>C generally mapped the name of a function directly to the exported ABI name, usually with a prefixed <code>_</code>. (Not sure of the reason for that prefix, to be honest.) "an identifier beginning with an underscore followed by a capital letter is a <a href="https://en.wikipedia.org/wiki/Reserved_word">reserved identifier</a> in C, so conflict with user identifiers is avoided"</p> <table> <thead> <tr> <th>Element </th> <th> Implementation</th> </tr> </thead> <tbody> <tr> <td>Argument-passing order </td> <td> Right to left.</td> </tr> <tr> <td>Stack-maintenance responsibility </td> <td> Calling function pops the arguments from the stack.</td> </tr> <tr> <td>Name-decoration convention </td> <td> Underscore character (_) is prefixed to names, except when __cdecl functions that use C linkage are exported.</td> </tr> <tr> <td>Case-translation convention </td> <td> No case translation performed.</td> </tr> </tbody> </table> <h4><a href="/languages/cplusplus">C++</a></h4> <p>... was where things got <em>really</em> interesting. (<em>Inside the C++ Object Model</em> had/has a lot of details on this.) In order to support function overloading (same name, different parameters), C++ generated C-style function names with the parameters encoded as part of the name; example:</p> <pre><code>int f () { return 1; } int f (int) { return 0; } void g () { int i = f(), j = f(0); } </code></pre> <p>... could produce ...</p> <pre><code>int __f_v () { return 1; } int __f_i (int) { return 0; } void __g_v () { int i = __f_v(), j = __f_i(0); } </code></pre> <p>and it got even more interesting for classes:</p> <pre><code>namespace wikipedia { class article { public: std::string format (); // = _ZN9wikipedia7article6formatEv bool print_to (std::ostream&amp;); // = _ZN9wikipedia7article8print_toERSo class wikilink { public: wikilink (std::string const&amp; name); // = _ZN9wikipedia7article8wikilinkC1ERKSs }; }; } </code></pre> <p>... depending on the precise name-mangling convention for that C++ compiler:</p> <p>Compiler | <code>void h(int)</code> | <code>void h(int, char)</code> | <code>void h(void)</code><br> -------- + ----------- + ----------------- + ------------<br> Intel C++ 8.0 for Linux, HP aC++ A.05.55 IA-64, IAR EWARM C++, GCC 3.x and higher, Clang 1.x and higher | <code>_Z1hi</code> | <code>_Z1hic</code> | <code>_Z1hv</code><br> GCC 2.9.x, HP aC++ A.03.45 PA-RISC | <code>h__Fi</code> | <code>h__Fic</code> | <code>h__Fv</code><br> Microsoft Visual C++ v6-v10 (<a href="https://en.wikiversity.org/wiki/Visual_C%2B%2B_name_mangling">mangling details</a>), Digital Mars C++ | <code>?h@@YAXH@Z</code> | <code>?h@@YAXHD@Z</code> | <code>?h@@YAXXZ</code><br> Borland C++ v3.1 | <code>@h$qi</code> | <code>@h$qizc</code> | <code>@h$qv</code><br> OpenVMS C++ v6.5 (ARM mode) | <code>H__XI</code> | <code>H__XIC</code> | <code>H__XV</code><br> OpenVMS C++ v6.5 (ANSI mode) | | <code>CXX$__7H__FIC26CDH77</code> | <code>CXX$__7H__FV2CB06E8</code><br> OpenVMS C++ X7.1 IA-64 | <code>CXX$_Z1HI2DSQ26A</code> | <code>CXX$_Z1HIC2NP3LI4</code> | <code>CXX$_Z1HV0BCA19V</code><br> SunPro CC | <code>__1cBh6Fi_v_</code> | <code>__1cBh6Fic_v_</code> | <code>__1cBh6F_v_</code><br> Tru64 C++ v6.5 (ARM mode) | <code>h__Xi</code> | <code>h__Xic</code> | <code>h__Xv</code><br> Tru64 C++ v6.5 (ANSI mode) | <code>__7h__Fi</code> | <code>__7h__Fic</code> | <code>__7h__Fv</code><br> Watcom C++ 10.6 | <code>W?h$n(i)v</code> | <code>W?h$n(ia)v</code> | <code>W?h$n()v</code></p> <h4><a href="/content/platforms/windows/index.md">Windows</a></h4> <p>Windows developed a number of different calling conventions.</p> <h5><a href=""><code>__cdecl</code></a></h5> <p>The C calling convention, above. Generally used solely for C-compiled libraries that weren't a core part of the Windows OS (a la the C standard library).</p> <h5><a href="https://learn.microsoft.com/en-us/cpp/cpp/thiscall"><code>__thiscall</code></a></h5> <p>The (Microsoft-specific) <code>__thiscall</code> calling convention is used on C++ class member functions on the x86 architecture. It's the default calling convention used by member functions that don't use variable arguments (vararg functions).</p> <p>Under <code>__thiscall</code>, the callee cleans the stack, which is impossible for vararg functions. Arguments are pushed on the stack from right to left. The this pointer is passed via register ECX, and not on the stack.</p> <p>On ARM, ARM64, and x64 machines, <code>__thiscall</code> is accepted and ignored by the compiler. That's because they use a register-based calling convention by default.</p> <p>One reason to use <code>__thiscall</code> is in classes whose member functions use __clrcall by default. In that case, you can use <code>__thiscall</code> to make individual member functions callable from native code.</p> <p>When compiling with /clr:pure, all functions and function pointers are __clrcall unless specified otherwise. The /clr:pure and /clr:safe compiler options are deprecated in Visual Studio 2015 and unsupported in Visual Studio 2017.</p> <p>vararg member functions use the __cdecl calling convention. All function arguments are pushed on the stack, with the this pointer placed on the stack last.</p> <p>Because this calling convention applies only to C++, it doesn't have a C name decoration scheme.</p> <p>When you define a non-static class member function out-of-line, specify the calling convention modifier only in the declaration. You don't have to specify it again on the out-of-line definition. The compiler uses the calling convention specified during declaration at the point of definition.</p> <h5><a href="https://learn.microsoft.com/en-us/cpp/cpp/stdcall"><code>__stdcall</code></a></h5> <p>This is the original Win32 API calling convention. The callee cleans the stack, so the compiler makes vararg functions __cdecl.</p> <table> <thead> <tr> <th>Element </th> <th> Implementation</th> </tr> </thead> <tbody> <tr> <td>Argument-passing order </td> <td> Right to left.</td> </tr> <tr> <td>Argument-passing convention </td> <td> By value, unless a pointer or reference type is passed.</td> </tr> <tr> <td>Stack-maintenance responsibility </td> <td> Called function pops its own arguments from the stack.</td> </tr> <tr> <td>Name-decoration convention </td> <td> An underscore (_) is prefixed to the name. The name is followed by the at sign (@) followed by the number of bytes (in decimal) in the argument list. Therefore, the function declared as int func( int a, double b ) is decorated as follows: _func@12</td> </tr> </tbody> </table> <p>Case-translation convention None</p> <h5><a href="https://learn.microsoft.com/en-us/cpp/cpp/fastcall"><code>_fastcall</code></a></h5> <p>The __fastcall calling convention specifies that arguments to functions are to be passed in registers, when possible. This calling convention only applies to the x86 architecture. The following list shows the implementation of this calling convention.</p> <table> <thead> <tr> <th>Element </th> <th> Implementation</th> </tr> </thead> <tbody> <tr> <td>Argument-passing order </td> <td> The first two DWORD or smaller arguments that are found in the argument list from left to right are passed in ECX and EDX registers; all other arguments are passed on the stack from right to left.</td> </tr> <tr> <td>Stack-maintenance responsibility </td> <td> Called function pops the arguments from the stack.</td> </tr> <tr> <td>Name-decoration convention </td> <td> At sign (@) is prefixed to names; an at sign followed by the number of bytes (in decimal) in the parameter list is suffixed to names.</td> </tr> <tr> <td>Case-translation convention </td> <td> No case translation performed.</td> </tr> <tr> <td>Classes, structs, and unions </td> <td> Treated as "multibyte" types (regardless of size) and passed on the stack.</td> </tr> <tr> <td>Enums and enum classes </td> <td> Passed by register if their underlying type is passed by register. For example, if the underlying type is int or unsigned int of size 8, 16, or 32 bits.</td> </tr> </tbody> </table> <h5><a href="https://learn.microsoft.com/en-us/cpp/cpp/clrcall"><code>_clrcall</code></a></h5> <p>Specifies that a function can only be called from <a href="/content/vms/clr/">managed code</a>. Use __clrcall for all virtual functions that will only be called from managed code. However this calling convention cannot be used for functions that will be called from native code.</p> <p>Use __clrcall to improve performance when calling from a managed function to a virtual managed function or from managed function to managed function through pointer.</p> <p>Entry points are separate, compiler-generated functions. If a function has both native and managed entry points, one of them will be the actual function with the function implementation. The other function will be a separate function (a thunk) that calls into the actual function and lets the common language runtime perform PInvoke. When marking a function as __clrcall, you indicate the function implementation must be MSIL and that the native entry point function will not be generated.</p> <p>When taking the address of a native function if __clrcall is not specified, the compiler uses the native entry point. __clrcall indicates that the function is managed and there is no need to go through the transition from managed to native. In that case the compiler uses the managed entry point.</p> <p>When /clr (not /clr:pure or /clr:safe) is used and __clrcall is not used, taking the address of a function always returns the address of the native entry point function. When __clrcall is used, the native entry point function is not created, so you get the address of the managed function, not an entry point thunk function. For more information, see Double Thunking. The /clr:pure and /clr:safe compiler options are deprecated in Visual Studio 2015 and unsupported in Visual Studio 2017.</p> <p>/clr (Common Language Runtime Compilation) implies that all functions and function pointers are __clrcall and the compiler will not permit a function inside the compiland to be marked anything other than __clrcall. When /clr:pure is used, __clrcall can only be specified on function pointers and external declarations.</p> <p>You can directly call __clrcall functions from existing C++ code that was compiled by using /clr as long as that function has an MSIL implementation. __clrcall functions cannot be called directly from functions that have inline asm and call CPU-specific intrinsics, for example, even if those functions are compiled with /clr.</p> <p>__clrcall function pointers are only meant to be used in the application domain in which they were created.</p> <p><strong>Readings</strong></p> <ul> <li>Measuring Mangled Name Ambiguity in Large C / C++ Projects <ul> <li>SQAMIA 2017</li> <li>Richárd Szalay, Zoltán Porkoláb, Dániel Krupp</li> <li><a href="https://www.researchgate.net/publication/320923382_Measuring_Mangled_Name_Ambiguity_in_Large_C_C_Projects">https://www.researchgate.net/publication/320923382_Measuring_Mangled_Name_Ambiguity_in_Large_C_C_Projects</a></li> <li><a href="http://ceur-ws.org/Vol-1938/paper-sza.pdf">http://ceur-ws.org/Vol-1938/paper-sza.pdf</a></li> </ul> </li> <li>Towards Better Symbol Resolution for C/C++ Programs: A Cluster-Based Solution <ul> <li>Source Code Analysis and Manipulation (SCAM) 2017</li> <li>Richárd Szalay, Zoltán Porkoláb, Dániel Krupp</li> <li><a href="http://ieeexplore.ieee.org/document/8090143/">http://ieeexplore.ieee.org/document/8090143/</a></li> <li><a href="https://www.researchgate.net/publication/320832497_Towards_Better_Symbol_Resolution_for_CC_Programs_A_Cluster-Based_Solution">https://www.researchgate.net/publication/320832497_Towards_Better_Symbol_Resolution_for_CC_Programs_A_Cluster-Based_Solution</a></li> </ul> </li> </ul> <p><strong>Software</strong></p> <ul> <li><a href="https://d.fuqu.jp/c++filtjs/">c++filtjs</a> (<a href="https://github.com/nattofriends/c-filtjs">Source</a>): c++filt in JavaScript with Emscripten</li> <li><a href="https://github.com/whitequark/binja_itanium_cxx_abi/blob/master/demangler.py">C++ Itanium ABI demangler</a>: C++ demangler in Python that converts the mangled name into an AST</li> <li><a href="https://github.com/AVGTechnologies/cppmangle">cppmangle</a>: A library for demangling and mangling Visual Studio C++ names</li> <li><a href="https://github.com/gimli-rs/cpp_demangle">cpp_demangle</a>: A crate for demangling C++ symbols</li> <li><a href="https://github.com/avast-tl/demangler">Demangler</a>: A C++ library and tools for demangling mangled C++ names <ul> <li><a href="https://github.com/avast-tl/retdec/tree/master/src/demangler">https://github.com/avast-tl/retdec/tree/master/src/demangler</a></li> <li><a href="https://github.com/avast-tl/retdec/tree/master/tests/demangler">https://github.com/avast-tl/retdec/tree/master/tests/demangler</a></li> </ul> </li> <li><a href="https://github.com/nico/demumble">demumble</a>: A better c++filt and a better undname.exe, in one binary. demumble demangles both POSIX and Visual Studio symbols. It runs on both POSIX and Windows.</li> <li><a href="http://demangler.com/">GCC and MSVC C++ Demangler</a></li> </ul> <p><strong>Talks</strong></p> <ul> <li>C++ Weekly - Ep 8 C++ Name Demangling - <a href="https://www.youtube.com/watch?v=uX99t7GmuDc">https://www.youtube.com/watch?v=uX99t7GmuDc</a></li> </ul> <h4><a href="/languages/objc">Obj-C</a></h4> <p>Two forms of method in Objective-C, the class ("static") method, and the instance method. A method declaration in Objective-C is of the following form:</p> <pre><code>+ (return-type) name0:parameter0 name1:parameter1 ... – (return-type) name0:parameter0 name1:parameter1 ... </code></pre> <p>Class methods are signified by +, instance methods use -. A typical class method declaration may then look like:</p> <pre><code>+ (id) initWithX: (int) number andY: (int) number; + (id) new; </code></pre> <p>With instance methods looking like this:</p> <pre><code>- (id) value; - (id) setValue: (id) new_value; </code></pre> <p>Each of these method declarations have a specific internal representation. When compiled, each method is named according to the following scheme for class methods:</p> <pre><code>_c_Class_name0_name1_ ... </code></pre> <p>and this for instance methods:</p> <pre><code>_i_Class_name0_name1_ ... </code></pre> <p>The colons in the Objective-C syntax are translated to underscores. So, the Objective-C class method <code>+ (id) initWithX: (int) number andY: (int) number;</code>, if belonging to the Point class would translate as <code>_c_Point_initWithX_andY_</code>, and the instance method (belonging to the same class) <code>- (id) value;</code> would translate to <code>_i_Point_value</code>.</p> <p>Each of the methods of a class are labeled in this way. However, in order to look up a method that a class may respond to would be tedious if all methods are represented in this fashion. Each of the methods is assigned a unique symbol (such as an integer). Such a symbol is known as a selector. In Objective-C, one can manage selectors directly — they have a specific type in Objective-C — <code>SEL</code>.</p> <p>During compilation, a table is built that maps the textual representation, such as <code>_i_Point_value</code>, to selectors. Managing selectors is more efficient than manipulating the textual representation of a method. Note that a selector only matches a method's name, not the class it belongs to — different classes can have different implementations of a method with the same name. Because of this, implementations of a method are given a specific identifier too, these are known as implementation pointers, and are also given a type, <code>IMP</code>.</p> <p>Message sends are encoded by the compiler as calls to the <code>id objc_msgSend (id receiver, SEL selector, ...)</code> function, or one of its cousins, where receiver is the receiver of the message, and SEL determines the method to call. Each class has its own table that maps selectors to their implementations — the implementation pointer specifies where in memory the actual implementation of the method resides. There are separate tables for class and instance methods. Apart from being stored in the <code>SEL</code> to <code>IMP</code> lookup tables, the functions are essentially anonymous.</p> <p>The SEL value for a selector does not vary between classes. This enables polymorphism.</p> <p>The Objective-C runtime maintains information about the argument and return types of methods. However, this information is not part of the name of the method, and can vary from class to class.</p> <p>Since Objective-C does not support namespaces, there is no need for the mangling of class names (that do appear as symbols in generated binaries).</p> <h4>Swift</h4> <p>Swift keeps metadata about functions (and more) in the mangled symbols referring to them. This metadata includes the function's name, attributes, module name, parameter types, return type, and more. For example:</p> <p>The mangled name for a method <code>func calculate(x: int) -&gt; int</code> of a <code>MyClass</code> class in module <code>test</code> is <code>_TFC4test7MyClass9calculatefS0_FT1xSi_Si</code>, for 2014 Swift. <a href="https://mikeash.com/pyblog/friday-qa-2014-08-15-swift-name-mangling.html">The components and their meanings</a> are as follows:</p> <ul> <li>_T: The prefix for all Swift symbols. Everything will start with this.</li> <li>F: Non-curried function.</li> <li>C: Function of a class, i.e. a method</li> <li>4test: Module name, prefixed with its length.</li> <li>7MyClass: Name of class the function belongs to, prefixed with its length.</li> <li>9calculate: Function name, prefixed with its length.</li> <li>f: The function attribute. In this case ‘f’, which means a normal function.</li> <li>S0: Designates the type of the first parameter (namely the class instance) as the first in the type stack (here MyClass is not nested and thus has index 0).</li> <li>_FT: This begins the type list for the parameter tuple of the function.</li> <li>1x: External name of first parameter of the function.</li> <li>Si: Indicates builtin Swift type Swift.Int for the first parameter.</li> <li>_Si: The return type: again Swift.Int.</li> </ul> <p>Mangling for versions since Swift 4.0 is <a href="https://github.com/apple/swift/blob/main/docs/ABI/Mangling.rst">documented officially</a>.</p> <h2>Calling conventions</h2> <p>An implementation-level (low-level) scheme for how subroutines receive parameters from their caller and how they return a result. Differences in various implementations include where parameters, return values, return addresses and scope links are placed (registers, stack or memory etc.), and how the tasks of preparing for a function call and restoring the environment afterwards are divided between the caller and the callee.</p> <p>Calling conventions may differ in:</p> <ul> <li>Where parameters, return values and return addresses are placed (in registers, on the call stack, a mix of both, or in other memory structures)</li> <li>For parameters passed in memory, the order in which actual arguments for formal parameters are passed (or the parts of a large or complex argument)</li> <li>How a (possibly long or complex) return value is delivered from the callee back to the caller (on the stack, in a register, or within the heap)</li> <li>How the task of setting up for and cleaning up after a function call is divided between the caller and the callee</li> <li>Whether and how metadata describing the arguments is passed</li> <li>Where the previous value of the frame pointer is stored, which is used to restore the frame pointer when the routine ends (in the stack frame, or in some register)</li> <li>Where any static scope links for the routine's non-local data access are placed (typically at one or more positions in the stack frame, but sometimes in a general register, or, for some architectures, in special-purpose registers)</li> <li>How local variables are allocated can sometimes also be part of the calling convention (when the caller allocates for the callee)</li> </ul> <p>In some cases, differences also include the following:</p> <ul> <li>Conventions on which registers may be directly used by the callee, without being preserved</li> <li>Which registers are considered to be volatile and, if volatile, need not be restored by the callee</li> </ul> <h4>x86</h4> <p>Due to the small number of architectural registers, and historical focus on simplicity and small code-size, many x86 calling conventions pass arguments on the stack. The return value (or a pointer to it) is returned in a register. Some conventions use registers for the first few parameters which may improve performance, especially for short and simple leaf-routines very frequently invoked (i.e. routines that do not call other routines).</p> <p>Example caller:</p> <pre><code>push EAX ; pass some register result push dword [EBP+20] ; pass some memory variable (FASM/TASM syntax) push 3 ; pass some constant call calc ; the returned result is now in EAX </code></pre> <p>Example callee (<code>calc</code>):</p> <pre><code>calc: push EBP ; save old frame pointer mov EBP,ESP ; get new frame pointer sub ESP,localsize ; reserve stack space for locals ; perform calculations, leave result in EAX mov ESP,EBP ; free space for locals pop EBP ; restore old frame pointer ret paramsize ; free parameter space and return. </code></pre> <p>Some conventions leave the parameter space allocated, using plain <code>ret</code> instead of <code>ret imm16</code>. In that case, the caller could <code>add esp,12</code> in this example, or otherwise deal with the change to ESP.</p> <h4>ARM (A32)</h4> <p>The standard 32-bit ARM calling convention allocates the 15 general-purpose registers as:</p> <ul> <li>r15: Program counter (as per the instruction set specification).</li> <li>r14: Link register. The BL instruction, used in a subroutine call, stores the return address in this register.</li> <li>r13: Stack pointer. The Push/Pop instructions in "Thumb" operating mode use this register only.</li> <li>r12: Intra-Procedure-call scratch register.</li> <li>r4 to r11: Local variables.</li> <li>r0 to r3: Argument values passed to a subroutine and results returned from a subroutine.</li> </ul> <p>If the type of value returned is too large to fit in r0 to r3, or whose size cannot be determined statically at compile time, then the caller must allocate space for that value at run time, and pass a pointer to that space in r0.</p> <p>Subroutines must preserve the contents of r4 to r11 and the stack pointer (perhaps by saving them to the stack in the function prologue, then using them as scratch space, then restoring them from the stack in the function epilogue). In particular, subroutines that call other subroutines must save the return address in the link register r14 to the stack before calling those other subroutines. However, such subroutines do not need to return that value to r14—they merely need to load that value into r15, the program counter, to return.</p> <p>The ARM calling convention mandates using a full-descending stack. (<a href="https://github.com/ARM-software/abi-aa/blob/2bcab1e3b22d55170c563c3c7940134089176746/aapcs32/aapcs32.rst">Reference</a>)</p> <p>This calling convention causes a "typical" ARM subroutine to:</p> <ul> <li>In the prologue, push r4 to r11 to the stack, and push the return address in r14 to the stack (this can be done with a single STM instruction);</li> <li>Copy any passed arguments (in r0 to r3) to the local scratch registers (r4 to r11);</li> <li>Allocate other local variables to the remaining local scratch registers (r4 to r11);</li> <li>Do calculations and call other subroutines as necessary using BL, assuming r0 to r3, r12 and r14 will not be preserved;</li> <li>Put the result in r0;</li> <li>In the epilogue, pull r4 to r11 from the stack, and pull the return address to the program counter r15. This can be done with a single LDM instruction.</li> </ul> <h4>ARM (A64)</h4> <p>The <a href="https://developer.arm.com/documentation/den0024/a/The-ABI-for-ARM-64-bit-Architecture/Register-use-in-the-AArch64-Procedure-Call-Standard/Parameters-in-general-purpose-registers">AArch 64 calling convention</a> allocates the 31 general-purpose registers as:</p> <ul> <li>x31 (SP): Stack pointer or a zero register, depending on context.</li> <li>x30 (LR): Procedure link register, used to return from subroutines.</li> <li>x29 (FP): Frame pointer.</li> <li>x19 to x29: Callee-saved.</li> <li>x18 (PR): Platform register. Used for some operating-system-specific special purpose, or an additional caller-saved register.</li> <li>x16 (IP0) and x17 (IP1): Intra-Procedure-call scratch registers.</li> <li>x9 to x15: Local variables, caller saved.</li> <li>x8 (XR): Indirect return value address.</li> <li>x0 to x7: Argument values passed to and results returned from a subroutine.</li> </ul> <p>All registers starting with x have a corresponding 32-bit register prefixed with w. Thus, a 32-bit x0 is called w0.</p> <p>Similarly, <a href="https://developer.arm.com/documentation/den0024/a/The-ABI-for-ARM-64-bit-Architecture/Register-use-in-the-AArch64-Procedure-Call-Standard/Parameters-in-NEON-and-floating-point-registers">the 32 floating-point registers are allocated</a> as:</p> <ul> <li>v0 to v7: Argument values passed to and results returned from a subroutine.</li> <li>v8 to v15: callee-saved, but only the bottom 64 bits need to be preserved.</li> <li>v16 to v31: Local variables, caller saved.</li> </ul> <h2>Known calling conventions</h2> <p>References:</p> <h4>"cdecl"</h4> <h4>"Pascal" style</h4> <h4>Windows "stdcall"</h4> <h4>macOS ABI</h4> <p><a href="https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/LowLevelABI/000-Introduction/introduction.html">Outdated: OS X ABI Function Call Guide</a> covers 32-/64-bit PowerPC, and <a href="https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/LowLevelABI/130-IA-32_Function_Calling_Conventions/IA32.html">IA-32</a> / <a href="https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/LowLevelABI/140-x86-64_Function_Calling_Conventions/x86_64.html">x86-64</a> calling conventions</p> <h2>Application Binary Interfaces (ABI) in general</h2> <ul> <li><code>[[trivial_abi]]</code> 101: <a href="https://quuxplusone.github.io/blog/2018/05/02/trivial-abi-101/">https://quuxplusone.github.io/blog/2018/05/02/trivial-abi-101/</a></li> <li>ABI Breaks: Not just about rebuilding: <a href="https://www.reddit.com/r/cpp/comments/fc2qqv/abi_breaks_not_just_about_rebuilding/">https://www.reddit.com/r/cpp/comments/fc2qqv/abi_breaks_not_just_about_rebuilding/</a></li> <li>ABI Policy and Guidelines - The GNU C++ Library Manual: <a href="https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html">https://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html</a></li> <li>ABIs, linkers and other animals - Stephen Kell (2014): <a href="https://www.cl.cam.ac.uk/~srk31/research/talks/kell14abis-slides.pdf">https://www.cl.cam.ac.uk/~srk31/research/talks/kell14abis-slides.pdf</a></li> <li>Binary Compatibility Examples: <a href="https://community.kde.org/Policies/Binary_Compatibility_Examples">https://community.kde.org/Policies/Binary_Compatibility_Examples</a></li> <li>Binary Compatibility Issues With C++: <a href="https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B">https://community.kde.org/Policies/Binary_Compatibility_Issues_With_C%2B%2B</a></li> <li>Binary Compatibility of Shared Libraries Implemented in C++ on GNU/Linux Systems <ul> <li>SYRCoSE 2009; Pavel Shved, Denis Silakov</li> <li><a href="http://syrcose.ispras.ru/2009/files/02_paper.pdf">http://syrcose.ispras.ru/2009/files/02_paper.pdf</a></li> <li><a href="http://static.coldattic.info/restricted/science/syrcose09/cppbincomp.pdf">http://static.coldattic.info/restricted/science/syrcose09/cppbincomp.pdf</a></li> </ul> </li> <li>Binary-compatible C++ Interfaces - <a href="https://chadaustin.me/cppinterface.html">https://chadaustin.me/cppinterface.html</a></li> <li><a href="https://www.bell-labs.com/usr/dmr/www/clcs.html">"The C Language Calling Sequence"</a></li> <li>C++ Standards Committee Papers <ul> <li>ABI breakage - summary of initial comments <ul> <li>2019; Roger Orr</li> <li><a href="http://wg21.link/P1654">http://wg21.link/P1654</a></li> </ul> </li> <li>What is ABI, and What Should WG21 Do About It? <ul> <li>2020; Titus Winters</li> <li><a href="http://wg21.link/P2028">http://wg21.link/P2028</a></li> </ul> </li> <li>Extending the Type System to Provide API and ABI Flexibility <ul> <li>P2123R0; 2020-03-04; Hal Finkel, Tom Scogland</li> <li><a href="http://wg21.link/p2123">http://wg21.link/p2123</a></li> </ul> </li> </ul> </li> <li>C++: Under the Hood (March 1994) by Jan Gray <ul> <li><a href="http://files.rsdn.org/4539/cud94.htm">http://files.rsdn.org/4539/cud94.htm</a></li> <li><a href="https://blogs.msdn.microsoft.com/xiangfan/2012/02/06/c-under-the-hood/">https://blogs.msdn.microsoft.com/xiangfan/2012/02/06/c-under-the-hood/</a></li> <li><a href="https://www.openrce.org/articles/files/jangrayhood.pdf">https://www.openrce.org/articles/files/jangrayhood.pdf</a></li> </ul> </li> <li>Calling conventions for different C++ compilers and operating systems: <a href="http://www.agner.org/optimize/calling_conventions.pdf">http://www.agner.org/optimize/calling_conventions.pdf</a></li> <li>Describing the MSVC ABI for Structure Return Types: <a href="http://blog.aaronballman.com/2012/02/describing-the-msvc-abi-for-structure-return-types/">http://blog.aaronballman.com/2012/02/describing-the-msvc-abi-for-structure-return-types/</a></li> <li>How C array sizes become part of the binary interface of a library: <a href="https://developers.redhat.com/blog/2019/05/06/how-c-array-sizes-become-part-of-the-binary-interface-of-a-library/">https://developers.redhat.com/blog/2019/05/06/how-c-array-sizes-become-part-of-the-binary-interface-of-a-library/</a></li> <li>Itanium C++ ABI <ul> <li><a href="https://itanium-cxx-abi.github.io/cxx-abi/">https://itanium-cxx-abi.github.io/cxx-abi/</a></li> <li><a href="https://github.com/itanium-cxx-abi/cxx-abi">https://github.com/itanium-cxx-abi/cxx-abi</a></li> </ul> </li> <li>Lessons from the Unix stdio ABI: 40 Years Later <ul> <li><a href="https://fingolfin.org/blog/20200327/stdio-abi.html">https://fingolfin.org/blog/20200327/stdio-abi.html</a></li> </ul> </li> <li>Some thoughts on binary compatibility - <a href="http://blog.qt.io/blog/2009/08/12/some-thoughts-on-binary-compatibility/">http://blog.qt.io/blog/2009/08/12/some-thoughts-on-binary-compatibility/</a></li> <li>Some thoughts on calling convention - <a href="http://blog.qt.io/blog/2009/08/15/some-thoughts-on-calling-convention/">http://blog.qt.io/blog/2009/08/15/some-thoughts-on-calling-convention/</a></li> <li>The Importance of Calling Conventions - <a href="http://blog.aaronballman.com/2011/04/the-importance-of-calling-conventions/">http://blog.aaronballman.com/2011/04/the-importance-of-calling-conventions/</a></li> <li>The value of passing by value - <a href="https://www.macieira.org/blog/2012/02/the-value-of-passing-by-value/">https://www.macieira.org/blog/2012/02/the-value-of-passing-by-value/</a></li> <li>X86-64 System V Application Binary Interface: <a href="https://github.com/hjl-tools/x86-psABI/wiki/X86-psABI">https://github.com/hjl-tools/x86-psABI/wiki/X86-psABI</a></li> </ul> <h4>Software</h4> <ul> <li>ABI Compliance Checker (ABICC) <ul> <li>A tool for checking backward API/ABI compatibility of a C/C++ library</li> <li><a href="https://lvc.github.io/abi-compliance-checker/">https://lvc.github.io/abi-compliance-checker/</a></li> <li><a href="https://github.com/lvc/abi-compliance-checker">https://github.com/lvc/abi-compliance-checker</a></li> <li><a href="http://ispras.linuxbase.org/index.php/ABI_compliance_checker">http://ispras.linuxbase.org/index.php/ABI_compliance_checker</a></li> <li>How to check for ABI changes with abi compliance checker - <a href="https://fedoraproject.org/wiki/How_to_check_for_ABI_changes_with_abi_compliance_checker">https://fedoraproject.org/wiki/How_to_check_for_ABI_changes_with_abi_compliance_checker</a></li> </ul> </li> <li>ABI Dumper <ul> <li>A tool to dump ABI of an ELF object containing DWARF debug info</li> <li><a href="https://github.com/lvc/abi-dumper">https://github.com/lvc/abi-dumper</a></li> </ul> </li> <li>ABIGAIL: Application Binary Interface Generic Analysis and Instrumentation Library <ul> <li>abidiff - compares the Application Binary Interfaces (ABI) of two shared libraries in ELF format <ul> <li><a href="https://sourceware.org/libabigail/manual/abidiff.html">https://sourceware.org/libabigail/manual/abidiff.html</a></li> </ul> </li> <li>abidw - reads a shared library in ELF format and emits an XML representation of its ABI to standard output <ul> <li><a href="https://sourceware.org/libabigail/manual/abidw.html">https://sourceware.org/libabigail/manual/abidw.html</a></li> </ul> </li> <li>abicompat - checks that an application that links against a given shared library is still ABI compatible with a subsequent version of that library <ul> <li><a href="https://sourceware.org/libabigail/manual/abicompat.html">https://sourceware.org/libabigail/manual/abicompat.html</a></li> </ul> </li> <li>Comparing ABIs for Compatibility with libabigail – Part 1 - <a href="https://developers.redhat.com/blog/2014/10/23/comparing-abis-for-compatibility-with-libabigail-part-1/">https://developers.redhat.com/blog/2014/10/23/comparing-abis-for-compatibility-with-libabigail-part-1/</a></li> <li>Comparing ABIs for Compatibility with libabigail – Part 2 - <a href="https://developers.redhat.com/blog/2014/10/28/comparing-abis-for-compatibility-libabigail-part-2/">https://developers.redhat.com/blog/2014/10/28/comparing-abis-for-compatibility-libabigail-part-2/</a></li> <li>Pruning Dynamic Rebuilds With libabigail <ul> <li><a href="https://engineering.mongodb.com/post/pruning-dynamic-rebuilds-with-libabigail">https://engineering.mongodb.com/post/pruning-dynamic-rebuilds-with-libabigail</a></li> <li><a href="https://github.com/acmorrow/abilink-demo">https://github.com/acmorrow/abilink-demo</a></li> </ul> </li> <li>Talk: Libabigail: How semantic analysis of C and C++ ELF binaries can be used to analyze ABI changes (openSUSE Conference 2017) <ul> <li><a href="https://media.ccc.de/v/1234-libabigail-how-semantic-analysis-of-c-and-c-elf-binaries-can-be-used-to-analyze-abi-changes">https://media.ccc.de/v/1234-libabigail-how-semantic-analysis-of-c-and-c-elf-binaries-can-be-used-to-analyze-abi-changes</a></li> <li><a href="https://www.youtube.com/watch?v=wxVBuZK8Dl0">https://www.youtube.com/watch?v=wxVBuZK8Dl0</a></li> </ul> </li> </ul> </li> <li>abimap: A helper for library maintainers to use symbol versioning <ul> <li><a href="https://github.com/ansasaki/abimap">https://github.com/ansasaki/abimap</a></li> <li>Don't break your users: keep your API/ABI stable! <ul> <li>DevConf.CZ 2020; Anderson Sasaki</li> <li><a href="https://www.youtube.com/watch?v=tFuFO_bDke0">https://www.youtube.com/watch?v=tFuFO_bDke0</a></li> </ul> </li> </ul> </li> <li>pexcheck: Pexcheck is a command-line tool for checking the binary compatibility of public interfaces. <ul> <li><a href="https://github.com/AVGTechnologies/pexcheck">https://github.com/AVGTechnologies/pexcheck</a></li> </ul> </li> </ul> <h4>Talks</h4> <ul> <li>Analyzing changes to the binary interface exposed by the Kernel to its modules <ul> <li>Kernel Recipes 2019; Dodji Seketeli, Matthias Männich, Jessica Yu</li> <li><a href="https://kernel-recipes.org/en/2019/talks/analyzing-changes-to-the-binary-interface-exposed-by-the-kernel-to-its-modules/">https://kernel-recipes.org/en/2019/talks/analyzing-changes-to-the-binary-interface-exposed-by-the-kernel-to-its-modules/</a></li> </ul> </li> <li>API &amp; ABI versioning <ul> <li>Meeting C++ 2017; Mathieu Ropert</li> <li><a href="https://www.youtube.com/watch?v=k9PLRAnnEmE">https://www.youtube.com/watch?v=k9PLRAnnEmE</a></li> </ul> </li> <li>Binary compatibility for library developers <ul> <li>C++Now 2013; Thiago Macieira</li> <li><a href="https://www.youtube.com/watch?v=PHrXGHDd9no">https://www.youtube.com/watch?v=PHrXGHDd9no</a></li> <li><a href="https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/binary_compat_for_cpp_devs.pdf?raw=true">https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/binary_compat_for_cpp_devs.pdf?raw=true</a></li> </ul> </li> <li>Easy Binary Compatible C++ Interfaces Across Compilers <ul> <li>C++Now 2013; John Bandela</li> <li><a href="https://www.youtube.com/watch?v=BbbqBJ94-_E">https://www.youtube.com/watch?v=BbbqBJ94-_E</a></li> <li>PDF: <a href="https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/easy_binary_compat.pdf?raw=true">https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/easy_binary_compat.pdf?raw=true</a></li> <li>PPT: <a href="https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/easy_binary_compat.ppt?raw=true">https://github.com/boostcon/cppnow_presentations_2013/blob/master/tue/easy_binary_compat.ppt?raw=true</a></li> <li><a href="https://jrb-programming.blogspot.com/2012/12/easy-binary-compatible-interfaces.html">https://jrb-programming.blogspot.com/2012/12/easy-binary-compatible-interfaces.html</a></li> <li><a href="https://github.com/jbandela/cppcomponents">https://github.com/jbandela/cppcomponents</a></li> <li><a href="https://github.com/jbandela/cross_compiler_call">https://github.com/jbandela/cross_compiler_call</a></li> </ul> </li> <li>How to break an ABI and keep your users happy <ul> <li>CppCon 2017; Gennadiy Rozental</li> <li><a href="https://www.youtube.com/watch?v=NzaYUlAw93k">https://www.youtube.com/watch?v=NzaYUlAw93k</a></li> <li><a href="https://abseil.io/blog/20171023-cppcon-breaking-abi">https://abseil.io/blog/20171023-cppcon-breaking-abi</a></li> </ul> </li> <li>Linux User/Kernel ABI: the realities of how C and C++ programs really talk to the OS <ul> <li>ACCU 2018; Greg Law</li> <li><a href="https://www.youtube.com/watch?v=4CdmGxc5BpU">https://www.youtube.com/watch?v=4CdmGxc5BpU</a></li> </ul> </li> <li>Reversing C++ <ul> <li>Black Hat USA 2007; Paul Vincent Sabanal, Mark Vincent Yason</li> <li><a href="https://www.blackhat.com/presentations/bh-dc-07/Sabanal_Yason/Presentation/bh-dc-07-Sabanal_Yason.pdf">https://www.blackhat.com/presentations/bh-dc-07/Sabanal_Yason/Presentation/bh-dc-07-Sabanal_Yason.pdf</a></li> <li><a href="https://www.blackhat.com/presentations/bh-dc-07/Sabanal_Yason/Paper/bh-dc-07-Sabanal_Yason-WP.pdf">https://www.blackhat.com/presentations/bh-dc-07/Sabanal_Yason/Paper/bh-dc-07-Sabanal_Yason-WP.pdf</a></li> <li>Videos: <ul> <li><a href="https://archive.org/details/2007_BlackHat_Vegas-V72-Yason-Sabanal-Reversing_C">https://archive.org/details/2007_BlackHat_Vegas-V72-Yason-Sabanal-Reversing_C</a></li> <li><a href="https://www.youtube.com/watch?v=oJ3mOzD7rC8">https://www.youtube.com/watch?v=oJ3mOzD7rC8</a></li> <li>00: <a href="https://www.youtube.com/watch?v=Vy0z1baCh8s">https://www.youtube.com/watch?v=Vy0z1baCh8s</a></li> <li>01: <a href="https://www.youtube.com/watch?v=1wZ615YlMFs">https://www.youtube.com/watch?v=1wZ615YlMFs</a></li> <li>02: <a href="https://www.youtube.com/watch?v=dcoNjUn_ACI">https://www.youtube.com/watch?v=dcoNjUn_ACI</a></li> <li>03: <a href="https://www.youtube.com/watch?v=JZ8_QM-XM1k">https://www.youtube.com/watch?v=JZ8_QM-XM1k</a></li> </ul> </li> </ul> </li> <li>The ABI challenge <ul> <li>C++Now 2019; Arvid Norberg</li> <li><a href="https://www.youtube.com/watch?v=ncyQAjTyPwU">https://www.youtube.com/watch?v=ncyQAjTyPwU</a></li> </ul> </li> <li>The C++ ABI From the Ground Up <ul> <li>CppCon 2019; Louis Dionne</li> <li><a href="https://www.youtube.com/watch?v=DZ93lP1I7wU">https://www.youtube.com/watch?v=DZ93lP1I7wU</a></li> </ul> </li> <li>What's an ABI and why is it so complicated? <ul> <li>ACCU 2015; Jonathan Wakely</li> <li><a href="https://accu.org/content/conf2015/JonathanWakely-What%20Is%20An%20ABI%20And%20Why%20Is%20It%20So%20Complicated.pdf">https://accu.org/content/conf2015/JonathanWakely-What%20Is%20An%20ABI%20And%20Why%20Is%20It%20So%20Complicated.pdf</a></li> </ul> </li> </ul> <hr> <h1>Bash</h1> <ul> <li>ctypes.sh: a foreign function interface for bash <ul> <li><a href="https://github.com/taviso/ctypes.sh">https://github.com/taviso/ctypes.sh</a></li> </ul> </li> </ul> <hr> <h1>Stata</h1> <ul> <li>Stata commands for inline C++ code in do-files <ul> <li><a href="https://github.com/robertgrant/statacpp">https://github.com/robertgrant/statacpp</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/cryptocurrency/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/cryptocurrency/index.html <h2>Video</h2> <ul> <li><a href="https://www.youtube.com/watch?v=ORdWE_ffirg">Crypto: The World's Greatest Scam</a></li> </ul> <h2>Reading</h2> <ul> <li><a href="https://blog.dshr.org/2022/02/ee380-talk.html">EE380 talk</a> (<a href="../EE380Talk.pdf">PDF</a>) by David Rosenthal</li> <li><a href="http://bitcoinbook.cs.princeton.edu">Bitcoin and Cryptocurrency Technologies</a> - Arvind Narayanan, Joseph Bonneau, Edward Felten, Andrew Miller, Steven Goldfeder, Jeremy Clark (PDF)</li> <li><a href="https://rosenbaum.se/book/">Grokking Bitcoin</a> - Kalle Rosenbaum (HTML)</li> <li><a href="https://github.com/bitcoinbook/bitcoinbook">Mastering Bitcoin - Unlocking digital currencies</a> - Andreas M. Antonopoulos</li> </ul> https://tedneward.github.io/Research/reading/development/data-concurrency/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/data-concurrency/index.html <p><a href="http://dl.acm.org/citation.cfm?id=1394128&amp;bnc=1">"BASE: An Alternative to ACID"</a> (<a href="https://dl.acm.org/doi/pdf/10.1145/1394127.1394128">PDF</a>)</p> <p><a href="https://crdt.tech/">CRDTs.tech</a>: A Conflict-free Replicated Data Type (CRDT) is a data structure that simplifies distributed data storage systems and multi-user applications.</p> <h3>Parallel Programming</h3> <ul> <li><a href="http://cnx.org/contents/bb821554-7f76-44b1-89e7-8a2a759d1347%405.2">High Performance Computing</a> - Charles Severance &amp; Kevin Dowd (PDF, ePUB)</li> <li><a href="https://web.archive.org/web/20170702124132/https://hpc.llnl.gov/training/tutorials">High Performance Computing Training</a> (LLNL materials)</li> <li><a href="https://andreask.cs.illinois.edu/Teaching/HPCFall2012">High-Performance Scientific Computing</a> (class lectures and slides)</li> <li><a href="http://pages.tacc.utexas.edu/~eijkhout/istc/istc.html">Introduction to High-Performance Scientific Computing</a> - Victor Eijkhout</li> <li><a href="https://computing.llnl.gov/tutorials/parallel_comp/">Introduction to Parallel Computing</a> - Blaise Barney</li> <li><a href="https://www.kernel.org/pub/linux/kernel/people/paulmck/perfbook/perfbook.html">Is Parallel Programming Hard, And, If So, What Can You Do About It?</a> - Paul E. McKenney</li> <li><a href="http://heather.cs.ucdavis.edu/parprocbook">Programming on Parallel Machines; GPU, Multicore, Clusters and More</a> - Norm Matloff<br> Kerridge (PDF) (email address <em>requested</em>, not required)</li> <li><a href="https://us.fixstars.com/products/opencl/book/OpenCLProgrammingBook/contents/">The OpenCL Programming Book</a></li> </ul> https://tedneward.github.io/Research/reading/development/debugging/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/debugging/index.html <ul> <li>Program repair: Community-driven effort to facilitate discovery, access and systematization of data related to automated program repair research <ul> <li><a href="http://program-repair.org/">http://program-repair.org/</a></li> </ul> </li> <li>UNIX Debugger Translation Table: gdb, lldb, dbx, adb, sdb <ul> <li><a href="https://www.lurklurk.org/debuggers.html">https://www.lurklurk.org/debuggers.html</a></li> </ul> </li> <li><a href="https://wizardzines.com/zines/debugging-guide/">The Pocket Guide to Debugging</a></li> </ul> <hr> <h1>Standard Libraries</h1> <ul> <li>GLIBC (GNU C Library) <ul> <li>General GLIBC Debugging Techniques - <a href="http://sourceware.org/glibc/wiki/Debugging/Development_Debugging">http://sourceware.org/glibc/wiki/Debugging/Development_Debugging</a></li> <li>GDB pretty-printers for GLIBC <ul> <li><a href="http://sourceware.org/glibc/wiki/Debugging/Pretty_Printers">http://sourceware.org/glibc/wiki/Debugging/Pretty_Printers</a></li> </ul> </li> <li>Debugging the Loader - <a href="https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging">https://sourceware.org/glibc/wiki/Debugging/Loader_Debugging</a></li> <li>Backtraces - <a href="http://www.gnu.org/software/libc/manual/html_node/Backtraces.html">http://www.gnu.org/software/libc/manual/html_node/Backtraces.html</a></li> <li>Allocation Debugging - <a href="http://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html">http://www.gnu.org/software/libc/manual/html_node/Allocation-Debugging.html</a></li> </ul> </li> <li>libstdc++ Debug Mode: <a href="https://gcc.gnu.org/onlinedocs/libstdc++/manual/debug_mode.html">https://gcc.gnu.org/onlinedocs/libstdc++/manual/debug_mode.html</a> <ul> <li>Detecting incorrect C++ STL usage (libstdc++) - <a href="https://kristerw.blogspot.com/2018/03/detecting-incorrect-c-stl-usage.html">https://kristerw.blogspot.com/2018/03/detecting-incorrect-c-stl-usage.html</a></li> </ul> </li> <li>libc++ <ul> <li>Debug Mode: <a href="https://libcxx.llvm.org/docs/DesignDocs/DebugMode.html">https://libcxx.llvm.org/docs/DesignDocs/DebugMode.html</a></li> <li>GDB pretty-printers for libc++ <ul> <li><a href="https://github.com/llvm/llvm-project/blob/master/libcxx/utils/gdb/libcxx/printers.py">https://github.com/llvm/llvm-project/blob/master/libcxx/utils/gdb/libcxx/printers.py</a></li> <li><a href="https://chromium.googlesource.com/chromium/src/+/master/third_party/libcxx-pretty-printers/printers.py">https://chromium.googlesource.com/chromium/src/+/master/third_party/libcxx-pretty-printers/printers.py</a></li> <li><a href="https://github.com/koutheir/libcxx-pretty-printers">https://github.com/koutheir/libcxx-pretty-printers</a></li> </ul> </li> </ul> </li> <li>Visual C++ Debug Iterator Support: <a href="https://docs.microsoft.com/en-us/cpp/standard-library/debug-iterator-support">https://docs.microsoft.com/en-us/cpp/standard-library/debug-iterator-support</a></li> </ul> <hr> <h1>Readings</h1> <ul> <li>Debugging - Ian Lance Taylor - <a href="https://airs.com/ian/essays/debug/debug.html">https://airs.com/ian/essays/debug/debug.html</a></li> <li>Debugging in the (Very) Large: Ten Years of Implementation and Experience <ul> <li>Symposium on Operating Systems Principles (SOSP) 2009</li> <li>Kirk Glerum, Kinshuman Kinshumann, Steve Greenberg, Gabriel Aul, Vince Orgovan, Greg Nichols, David Grant, Gretchen Loihle, Galen Hunt</li> <li><a href="https://www.microsoft.com/en-us/research/publication/debugging-in-the-very-large-ten-years-of-implementation-and-experience/">https://www.microsoft.com/en-us/research/publication/debugging-in-the-very-large-ten-years-of-implementation-and-experience/</a></li> <li><a href="https://www.sigops.org/sosp/sosp09/papers/glerum-sosp09.pdf">https://www.sigops.org/sosp/sosp09/papers/glerum-sosp09.pdf</a></li> <li><a href="https://www.sigops.org/sosp/sosp09/slides/glerum-slides-sosp09.pdf">https://www.sigops.org/sosp/sosp09/slides/glerum-slides-sosp09.pdf</a></li> </ul> </li> <li>debugging-stories: A collection of debugging stories <ul> <li><a href="https://github.com/danluu/debugging-stories">https://github.com/danluu/debugging-stories</a></li> <li><a href="http://danluu.com/teach-debugging/">http://danluu.com/teach-debugging/</a></li> </ul> </li> <li>Delta Debugging <ul> <li><a href="https://www.st.cs.uni-saarland.de/dd/">https://www.st.cs.uni-saarland.de/dd/</a></li> <li>Delta: Heuristically minimizes interesting files <ul> <li><a href="http://delta.stage.tigris.org/">http://delta.stage.tigris.org/</a></li> <li>Minimizing Interesting Files with Delta <ul> <li><a href="http://delta.stage.tigris.org/using_delta.html">http://delta.stage.tigris.org/using_delta.html</a></li> </ul> </li> </ul> </li> <li>Yesterday, my program worked. Today, it does not. Why? <ul> <li>ESEC 1999</li> <li>Andreas Zeller</li> <li><a href="https://www.st.cs.uni-saarland.de/publications/details/zeller-esec-1999/">https://www.st.cs.uni-saarland.de/publications/details/zeller-esec-1999/</a></li> </ul> </li> <li>Picireny: Hierarchical Delta Debugging Framework <ul> <li><a href="https://github.com/renatahodovan/picireny">https://github.com/renatahodovan/picireny</a></li> </ul> </li> <li>Simplifying and Isolating Failure-Inducing Input <ul> <li>IEEE Transactions on Software Engineering 28(2) 2002</li> <li>Andreas Zeller, Ralf Hildebrandt</li> <li><a href="https://www.st.cs.uni-saarland.de/publications/details/zeller-tse-2002/">https://www.st.cs.uni-saarland.de/publications/details/zeller-tse-2002/</a></li> <li><a href="https://blog.acolyer.org/2015/11/16/simplifying-and-isolating-failure-inducing-input/">https://blog.acolyer.org/2015/11/16/simplifying-and-isolating-failure-inducing-input/</a></li> </ul> </li> <li>HDD: Hierarchical Delta Debugging <ul> <li>ICSE 2006</li> <li>Ghassan Misherghi, Zhendong Su</li> <li><a href="https://dl.acm.org/citation.cfm?id=1134307">https://dl.acm.org/citation.cfm?id=1134307</a></li> <li><a href="https://blog.acolyer.org/2015/11/17/hierarchical-delta-debugging/">https://blog.acolyer.org/2015/11/17/hierarchical-delta-debugging/</a></li> </ul> </li> <li>Generalizing and Criticizing Delta Debugging <ul> <li>2011; John Regehr</li> <li><a href="https://blog.regehr.org/archives/527">https://blog.regehr.org/archives/527</a></li> </ul> </li> <li>Debugging Inputs <ul> <li>International Conference on Software Engineering (ICSE) 2020</li> <li>Lukas Kirschner, Ezekiel Soremekun, and Andreas Zeller</li> <li><a href="https://www.dropbox.com/s/ddn3fe55lws1rdr/icse2020-ddmax.pdf">https://www.dropbox.com/s/ddn3fe55lws1rdr/icse2020-ddmax.pdf</a></li> <li>ddmax implementations &amp; experimental data: <a href="https://tinyurl.com/DebuggingInputs">https://tinyurl.com/DebuggingInputs</a></li> </ul> </li> </ul> </li> <li>Devon H. O'Dell <ul> <li>Building a Debugging Mindset <ul> <li><a href="https://9vx.org/post/building-a-debugging-mindset/">https://9vx.org/post/building-a-debugging-mindset/</a></li> <li><a href="https://www.infoq.com/presentations/debugging-mindset">https://www.infoq.com/presentations/debugging-mindset</a></li> <li><a href="https://speakerdeck.com/dho/building-a-debugging-mindset">https://speakerdeck.com/dho/building-a-debugging-mindset</a></li> </ul> </li> <li>Debugging: Psychology, Theory, and Application - <a href="https://9vx.org/post/debugging-psychology-theory-and-application/">https://9vx.org/post/debugging-psychology-theory-and-application/</a></li> <li>The Debugging Mindset - <a href="https://queue.acm.org/detail.cfm?id=3068754">https://queue.acm.org/detail.cfm?id=3068754</a></li> </ul> </li> <li>Henrik Warne <ul> <li>Great Programmers Write Debuggable Code - <a href="https://henrikwarne.com/2013/05/05/great-programmers-write-debuggable-code/">https://henrikwarne.com/2013/05/05/great-programmers-write-debuggable-code/</a></li> <li>Finding Bugs: Debugger versus Logging - <a href="https://henrikwarne.com/2014/01/01/finding-bugs-debugger-versus-logging/">https://henrikwarne.com/2014/01/01/finding-bugs-debugger-versus-logging/</a></li> <li>Learning From Your Bugs - <a href="https://henrikwarne.com/2016/04/28/learning-from-your-bugs/">https://henrikwarne.com/2016/04/28/learning-from-your-bugs/</a></li> </ul> </li> <li>How to Debug - John Regehr - <a href="https://blog.regehr.org/archives/199">https://blog.regehr.org/archives/199</a></li> <li>I tend to prefer debugging with release builds instead of debug builds - Ken Johnson (Skywing) - <a href="http://www.nynaeve.net/?p=184">http://www.nynaeve.net/?p=184</a></li> <li>The Inflection Point Hypothesis: A Principled Debugging Approach for Locating the Root Cause of a Failure <ul> <li>Symposium on Operating Systems Principles (SOSP) 2019</li> <li>Yongle Zhang, Kirk Rodrigues, Yu Luo, Michael Stumm, Ding Yuan</li> <li><a href="https://dl.acm.org/citation.cfm?id=3359650">https://dl.acm.org/citation.cfm?id=3359650</a></li> <li><a href="https://blog.acolyer.org/2019/11/08/the-inflection-point-hypothesis/">https://blog.acolyer.org/2019/11/08/the-inflection-point-hypothesis/</a></li> </ul> </li> <li>What does debugging a program look like? - Julia Evans - <a href="https://jvns.ca/blog/2019/06/23/a-few-debugging-resources/">https://jvns.ca/blog/2019/06/23/a-few-debugging-resources/</a></li> <li>When debugging a stack overflow, you want to focus on the repeating recursive part - Raymond Chen - <a href="https://blogs.msdn.microsoft.com/oldnewthing/20090107-00/?p=19573">https://blogs.msdn.microsoft.com/oldnewthing/20090107-00/?p=19573</a></li> </ul> <h2>Books, Book Reviews</h2> <ul> <li>Effective Debugging - <a href="https://www.spinellis.gr/debugging/">https://www.spinellis.gr/debugging/</a></li> <li>Four Books on Debugging - <a href="https://blog.regehr.org/archives/849">https://blog.regehr.org/archives/849</a></li> <li>Geoff Wozniak <ul> <li>On battle scars and debugging - <a href="http://wozniak.ca/blog/2018/01/15/On-battle-scars-and-debugging/">http://wozniak.ca/blog/2018/01/15/On-battle-scars-and-debugging/</a></li> <li>From intuition to methodology in debugging - <a href="http://wozniak.ca/blog/2018/02/04/From-intuition-to-methodology-in-debugging/">http://wozniak.ca/blog/2018/02/04/From-intuition-to-methodology-in-debugging/</a></li> <li>Formalizing debugging - <a href="https://wozniak.ca/blog/2018/03/25/Book-review-Formalizing-debugging/">https://wozniak.ca/blog/2018/03/25/Book-review-Formalizing-debugging/</a></li> </ul> </li> <li>Why Programs Fail: A Guide to Systematic Debugging - <a href="http://www.whyprogramsfail.com/">http://www.whyprogramsfail.com/</a></li> </ul> <h2>Concurrency</h2> <ul> <li>Interactive Debugging of Concurrent Programs under Relaxed Memory Models <ul> <li>CGO 2020</li> <li>Aakanksha Verma, Pankaj Kumar Kalita, Awanish Pandey, and Subhajit Roy</li> <li>Gambit: GDB Assisted Memory Behavior and Interference Tester</li> <li><a href="https://doi.org/10.1145/3368826.3377910">https://doi.org/10.1145/3368826.3377910</a></li> </ul> </li> </ul> <h3>Probe Effect</h3> <ul> <li>A probe effect in concurrent programs <ul> <li>Software: Practice and Experience 16 (3)(1986)</li> <li>J. Gait</li> <li><a href="https://doi.org/10.1002/spe.4380160304">https://doi.org/10.1002/spe.4380160304</a></li> </ul> </li> <li>Debugging Concurrent Programs <ul> <li>ACM Computing Surveys (CSUR) 21(4) 1989</li> <li>C. E. McDowell, D. P. Helmbold</li> <li><a href="https://users.soe.ucsc.edu/~dph/mypubs/debugConcProg89.pdf">https://users.soe.ucsc.edu/~dph/mypubs/debugConcProg89.pdf</a></li> </ul> </li> </ul> <h2>Implementation</h2> <ul> <li>An Efficient and Generic Reversible Debugger using the Virtual Machine based Approach <ul> <li>Virtual Execution Environments (VEE) 2005</li> <li>Toshihiko Koju, Shingo Takada, Norihisa Doi</li> <li><a href="https://www.usenix.org/events/vee05/full_papers/p79-koju.pdf">https://www.usenix.org/events/vee05/full_papers/p79-koju.pdf</a></li> </ul> </li> <li>Debin: Predicting Debug Information in Stripped Binaries <ul> <li>ACM CCS 2018</li> <li>Jingxuan He, Pesho Ivanov, Petar Tsankov, Veselin Raychev, Martin Vechev</li> <li><a href="https://dl.acm.org/citation.cfm?id=3243866">https://dl.acm.org/citation.cfm?id=3243866</a></li> <li><a href="https://www.youtube.com/watch?v=x1x_KtS-5Hs">https://www.youtube.com/watch?v=x1x_KtS-5Hs</a></li> <li><a href="https://github.com/eth-sri/debin">https://github.com/eth-sri/debin</a></li> <li><a href="https://files.sri.inf.ethz.ch/website/papers/ccs18-debin.pdf">https://files.sri.inf.ethz.ch/website/papers/ccs18-debin.pdf</a></li> </ul> </li> <li>Debuggers for Programming Languages <ul> <li>2002 Book Chapter in "The Compiler Design Handbook: Optimizations and Machine Code Generation"</li> <li>Sanjeev Kumar Aggarwal, M. Sarath Kumar</li> <li><a href="https://www.taylorfrancis.com/books/9781420040579/chapters/10.1201%2F9781420040579-12">https://www.taylorfrancis.com/books/9781420040579/chapters/10.1201%2F9781420040579-12</a></li> </ul> </li> <li>Debugging with the natives - Stephen Kell <ul> <li>part 1 - <a href="http://www.cl.cam.ac.uk/~srk31/blog/2016/02/25/#native-debugging-part-1">http://www.cl.cam.ac.uk/~srk31/blog/2016/02/25/#native-debugging-part-1</a></li> <li>part 2 - <a href="http://www.cl.cam.ac.uk/~srk31/blog/2017/01/30/#native-debugging-part-2">http://www.cl.cam.ac.uk/~srk31/blog/2017/01/30/#native-debugging-part-2</a></li> </ul> </li> <li>Debugging Native Extensions of Dynamic Languages <ul> <li>International Conference on Managed Languages &amp; Runtimes (ManLang) 2018</li> <li>Jacob Kreindl, Manuel Rigger, Hanspeter Mössenböck</li> <li><a href="https://doi.org/10.1145/3237009.3237017">https://doi.org/10.1145/3237009.3237017</a></li> <li><a href="http://ssw.jku.at/General/Staff/Kreindl/papers/ManLang_2018_SulongDebugging.pdf">http://ssw.jku.at/General/Staff/Kreindl/papers/ManLang_2018_SulongDebugging.pdf</a></li> </ul> </li> <li>Debugging with Intelligence via Probabilistic Inference <ul> <li>Zhaogui Xu, Shiqing Ma, Xiangyu Zhang, Shuofei Zhu, Baowen Xu</li> <li>International Conference on Software Engineering (ICSE) 2018</li> <li><a href="https://www.cs.purdue.edu/homes/ma229/papers/ICSE18.pdf">https://www.cs.purdue.edu/homes/ma229/papers/ICSE18.pdf</a></li> <li><a href="https://blog.acolyer.org/2018/06/19/debugging-with-intelligence-via-probabilistic-inference/">https://blog.acolyer.org/2018/06/19/debugging-with-intelligence-via-probabilistic-inference/</a></li> </ul> </li> <li>Debugopt: Debugging fully optimized natively compiled programs using multistage instrumentation <ul> <li>Science of Computer Programming 169 (2019)</li> <li>Jie Yin, Gang Tan, Hao Li, Xiaolong Bai, Yu Ping Wang, Shi Min Hu</li> <li><a href="https://www.sciencedirect.com/science/article/pii/S0167642318303629">https://www.sciencedirect.com/science/article/pii/S0167642318303629</a></li> <li><a href="https://oslab.cs.tsinghua.edu.cn/debugopt/debugopt.html">https://oslab.cs.tsinghua.edu.cn/debugopt/debugopt.html</a></li> </ul> </li> <li>Eli Bendersky - <a href="http://eli.thegreenplace.net/tag/debuggers">http://eli.thegreenplace.net/tag/debuggers</a> <ul> <li>How debuggers work <ul> <li>Part 1 - Basics - <a href="http://eli.thegreenplace.net/2011/01/23/how-debuggers-work-part-1">http://eli.thegreenplace.net/2011/01/23/how-debuggers-work-part-1</a></li> <li>Part 2 - Breakpoints - <a href="http://eli.thegreenplace.net/2011/01/27/how-debuggers-work-part-2-breakpoints">http://eli.thegreenplace.net/2011/01/27/how-debuggers-work-part-2-breakpoints</a></li> <li>Part 3 - Debugging information - <a href="http://eli.thegreenplace.net/2011/02/07/how-debuggers-work-part-3-debugging-information">http://eli.thegreenplace.net/2011/02/07/how-debuggers-work-part-3-debugging-information</a></li> </ul> </li> <li>An interesting tree serialization algorithm from DWARF - <a href="https://eli.thegreenplace.net/2011/09/29/an-interesting-tree-serialization-algorithm-from-dwarf">https://eli.thegreenplace.net/2011/09/29/an-interesting-tree-serialization-algorithm-from-dwarf</a></li> <li>The contents of DWARF sections - <a href="https://eli.thegreenplace.net/2011/12/26/the-contents-of-dwarf-sections">https://eli.thegreenplace.net/2011/12/26/the-contents-of-dwarf-sections</a></li> <li>Programmatic access to the call stack in C++ - <a href="https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/">https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/</a></li> </ul> </li> <li>Fast, Flexible, Polyglot Instrumentation Support for Debuggers and other Tools <ul> <li>The Art, Science, and Engineering of Programming, 2018, Vol. 2, Issue 3, Article 14</li> <li>Van De Vanter, Michael; Seaton, Chris; Haupt, Michael; Humer, Christian; Würthinger, Thomas</li> <li><a href="http://programming-journal.org/2018/2/14/">http://programming-journal.org/2018/2/14/</a></li> </ul> </li> <li>Framework for Instruction-level Tracing and Analysis of Program Executions <ul> <li>Virtual Execution Environments (VEE) 2006</li> <li><a href="https://www.usenix.org/legacy/events/vee06/full_papers/p154-bhansali.pdf">https://www.usenix.org/legacy/events/vee06/full_papers/p154-bhansali.pdf</a></li> <li>iDNA: Time Travel Debugging <ul> <li>Instruction-level Tracing: Framework &amp; Applications</li> <li>Sanjay Bhansali</li> <li><a href="http://www.cs.wisc.edu/areas/pl/seminar/fall05/Bhansali.ppt">http://www.cs.wisc.edu/areas/pl/seminar/fall05/Bhansali.ppt</a></li> </ul> </li> </ul> </li> <li>GCC gOlogy: studying the impact of optimizations on debugging <ul> <li><a href="http://www.fsfla.org/~lxoliva/writeups/gOlogy/gOlogy.txt">http://www.fsfla.org/~lxoliva/writeups/gOlogy/gOlogy.txt</a></li> <li>GNU Tools Cauldron 2018 slides: <a href="http://people.redhat.com/aoliva/writeups/gOlogy/slides.pdf">http://people.redhat.com/aoliva/writeups/gOlogy/slides.pdf</a></li> </ul> </li> <li>How breakpoints are set - <a href="http://majantali.net/2016/10/how-breakpoints-are-set/">http://majantali.net/2016/10/how-breakpoints-are-set/</a></li> <li>How do debuggers keep track of the threads in your program? <ul> <li><a href="http://timetobleed.com/how-do-debuggers-keep-track-of-the-threads-in-your-program/">http://timetobleed.com/how-do-debuggers-keep-track-of-the-threads-in-your-program/</a></li> </ul> </li> <li>How to code debuggers - Tomasz Wegrzanowski - <a href="https://t-a-w.blogspot.com/2007/03/how-to-code-debuggers.html">https://t-a-w.blogspot.com/2007/03/how-to-code-debuggers.html</a></li> <li>Making a low level (Linux) debugger <ul> <li>part 1: assembly - <a href="https://blog.asrpo.com/making_a_low_level_debugger">https://blog.asrpo.com/making_a_low_level_debugger</a></li> <li>part 2: C - <a href="https://blog.asrpo.com/making_a_low_level_debugger_part_2">https://blog.asrpo.com/making_a_low_level_debugger_part_2</a></li> <li>part 3: our first program - <a href="https://blog.asrpo.com/making_a_low_level_debugger_part_3">https://blog.asrpo.com/making_a_low_level_debugger_part_3</a></li> </ul> </li> <li>On-Stack Replacement, Distilled <ul> <li>Programming Language Design and Implementation (PLDI) 2018</li> <li>Daniele Cono D'Elia and Camil Demetrescu</li> <li><a href="https://pldi18.sigplan.org/event/pldi-2018-papers-on-stack-replacement-distilled">https://pldi18.sigplan.org/event/pldi-2018-papers-on-stack-replacement-distilled</a></li> <li><a href="http://season-lab.github.io/papers/osr-distilled-pldi18.pdf">http://season-lab.github.io/papers/osr-distilled-pldi18.pdf</a></li> <li><a href="https://github.com/dcdelia/tinyvm">https://github.com/dcdelia/tinyvm</a></li> <li>"As a novel application of OSR, we present a feasibility study on debugging of optimized code, showing how our techniques can be used to fix variables holding incorrect values at breakpoints due to optimizations."</li> </ul> </li> <li>Samy Al Bahra, Backtrace <ul> <li>Compiler debug quality suite - <a href="https://github.com/backtrace-labs/cdqs">https://github.com/backtrace-labs/cdqs</a></li> <li>Compile Once Debug Twice: Picking a Compiler for Debuggability <ul> <li><a href="https://backtrace.io/blog/compile-once-debug-twice-picking-a-compiler-for-debuggability-1of3/">https://backtrace.io/blog/compile-once-debug-twice-picking-a-compiler-for-debuggability-1of3/</a></li> </ul> </li> <li>Debugging the Debugger: Why Your Debugger Doesn’t Work When You Need it To <ul> <li><a href="https://backtrace.io/debuggingthedebugger/">https://backtrace.io/debuggingthedebugger/</a></li> </ul> </li> <li>Implementing a Debugger - Backtrace <ul> <li>The Fundamentals - <a href="http://backtrace.io/blog/blog/2016/08/11/debugger-internals/">http://backtrace.io/blog/blog/2016/08/11/debugger-internals/</a></li> <li>Building a Go Debugger - <a href="https://backtrace.io/blog/building-a-go-debugger/">https://backtrace.io/blog/building-a-go-debugger/</a></li> </ul> </li> </ul> </li> <li>Writing a basic Windows debugger - <a href="https://www.codeproject.com/Articles/43682/Writing-a-basic-Windows-debugger">https://www.codeproject.com/Articles/43682/Writing-a-basic-Windows-debugger</a></li> <li>Writing a Debugger - Joseph Kain - <a href="http://system.joekain.com/debugger/">http://system.joekain.com/debugger/</a></li> <li>Writing a Linux Debugger - Simon Brand <ul> <li>1. Setup - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/">https://blog.tartanllama.xyz/writing-a-linux-debugger-setup/</a></li> <li>2. Breakpoints - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/">https://blog.tartanllama.xyz/writing-a-linux-debugger-breakpoints/</a></li> <li>3. Registers and memory - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/">https://blog.tartanllama.xyz/writing-a-linux-debugger-registers/</a></li> <li>4. Elves and dwarves - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/">https://blog.tartanllama.xyz/writing-a-linux-debugger-elf-dwarf/</a></li> <li>5. Source and signals - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/">https://blog.tartanllama.xyz/writing-a-linux-debugger-source-signal/</a></li> <li>6. Source-level stepping - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/">https://blog.tartanllama.xyz/writing-a-linux-debugger-dwarf-step/</a></li> <li>7. Source-level breakpoints - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/">https://blog.tartanllama.xyz/writing-a-linux-debugger-source-break/</a></li> <li>8. Stack unwinding - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/">https://blog.tartanllama.xyz/writing-a-linux-debugger-unwinding/</a></li> <li>9. Handling variables - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/">https://blog.tartanllama.xyz/writing-a-linux-debugger-variables/</a></li> <li>10. Advanced topics - <a href="https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/">https://blog.tartanllama.xyz/writing-a-linux-debugger-advanced-topics/</a></li> <li>minidbg: A mini x86 linux debugger for teaching purposes - <a href="https://github.com/TartanLlama/minidbg">https://github.com/TartanLlama/minidbg</a></li> </ul> </li> <li>(Windows) Data Breakpoints - <a href="https://blogs.msdn.microsoft.com/reiley/2011/07/21/data-breakpoints/">https://blogs.msdn.microsoft.com/reiley/2011/07/21/data-breakpoints/</a></li> <li>(Windows) Side Effects of Debugger - <a href="https://blogs.msdn.microsoft.com/reiley/2011/08/27/side-effects-of-debugger/">https://blogs.msdn.microsoft.com/reiley/2011/08/27/side-effects-of-debugger/</a></li> </ul> <h3>Correctness</h3> <ul> <li>Correctness Proofs of Compilers and Debuggers: an Approach Based on Structural Operational Semantics <ul> <li>1992 Ph.D. dissertation; Fabio Q. B. da Silva</li> <li><a href="http://hdl.handle.net/1842/13542">http://hdl.handle.net/1842/13542</a></li> <li><a href="http://www.lfcs.inf.ed.ac.uk/reports/92/ECS-LFCS-92-241/">http://www.lfcs.inf.ed.ac.uk/reports/92/ECS-LFCS-92-241/</a></li> </ul> </li> <li>Debug Information Validation for Optimized Code <ul> <li>PLDI 2020</li> <li>Yuanbo Li, Shuo Ding, Qirun Zhang, Davide Italiano</li> <li><a href="https://helloqirun.github.io/papers/pldi20_yuanbo1.pdf">https://helloqirun.github.io/papers/pldi20_yuanbo1.pdf</a></li> <li><a href="https://www.cc.gatech.edu/~qrzhang/projects/debug/debug.html">https://www.cc.gatech.edu/~qrzhang/projects/debug/debug.html</a></li> <li><a href="https://pldi20.sigplan.org/details/pldi-2020-papers/60/Debug-Information-Validation-for-Optimized-Code">https://pldi20.sigplan.org/details/pldi-2020-papers/60/Debug-Information-Validation-for-Optimized-Code</a></li> </ul> </li> </ul> <h4>Testing</h4> <ul> <li>Comparing The Quality Of Debug Information Produced By Clang And GCC <ul> <li><a href="https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html">https://robert.ocallahan.org/2018/11/comparing-quality-of-debug-information.html</a></li> <li>debuginfo-quality: Evaluate the quality of debuginfo in an ELF binary <ul> <li><a href="https://github.com/rocallahan/debuginfo-quality">https://github.com/rocallahan/debuginfo-quality</a></li> </ul> </li> </ul> </li> <li>Debug Frame Checking: Check <code>.eh_frame</code> and <code>.debug_frame</code> information <ul> <li><a href="https://github.com/francesco-zappa-nardelli/eh_frame_check">https://github.com/francesco-zappa-nardelli/eh_frame_check</a></li> </ul> </li> <li>DExTer (Debugging Experience Tester) <ul> <li><a href="https://github.com/SNSystems/dexter">https://github.com/SNSystems/dexter</a></li> <li>Measuring the User Debugging Experience <ul> <li>2018 European LLVM Developers Meeting; Greg Bedwell</li> <li><a href="https://www.youtube.com/watch?v=XRT_GmpGjXE">https://www.youtube.com/watch?v=XRT_GmpGjXE</a></li> <li><a href="https://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience.pdf">https://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience.pdf</a></li> <li><a href="http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience_poster.png">http://llvm.org/devmtg/2018-04/slides/Bedwell-Measuring_the_User_Debugging_Experience_poster.png</a></li> <li><a href="https://www.snsystems.com/technology/tech-blog/measuring-the-user-debug-experience">https://www.snsystems.com/technology/tech-blog/measuring-the-user-debug-experience</a></li> </ul> </li> </ul> </li> <li>Feedback-Directed Differential Testing of Interactive Debuggers <ul> <li>ESEC/FSE 2018</li> <li>Daniel Lehmann, Michael Pradel</li> <li><a href="http://software-lab.org/publications/fse2018.pdf">http://software-lab.org/publications/fse2018.pdf</a></li> <li><a href="https://github.com/sola-da/DifferentialDebuggerTesting">https://github.com/sola-da/DifferentialDebuggerTesting</a></li> </ul> </li> <li>Interactive Metamorphic Testing of Debuggers <ul> <li>ISSTA 2019</li> <li>Sandro Tolksdorf, Daniel Lehmann, Michael Pradel</li> <li><a href="https://conf.researchr.org/event/issta-2019/issta-2019-technical-papers-interactive-metamorphic-testing-of-debuggers">https://conf.researchr.org/event/issta-2019/issta-2019-technical-papers-interactive-metamorphic-testing-of-debuggers</a></li> </ul> </li> <li>lldb-repro: a utility to transparently capture and replay debugger sessions through the command line driver <ul> <li>used to test the reproducers by running the test suite twice</li> <li><a href="https://github.com/llvm/llvm-project/tree/master/lldb/utils/lldb-repro">https://github.com/llvm/llvm-project/tree/master/lldb/utils/lldb-repro</a></li> <li><a href="https://lldb.llvm.org/resources/reproducers.html">https://lldb.llvm.org/resources/reproducers.html</a></li> </ul> </li> <li>Samy Al Bahra, Backtrace <ul> <li>Compiler debug quality suite - <a href="https://github.com/backtrace-labs/cdqs">https://github.com/backtrace-labs/cdqs</a></li> </ul> </li> </ul> <h2>Reverse Debugging</h2> <p>See also: <a href="#rr">RR</a>, <a href="https://github.com/MattPD/cpplinks/blob/master/debugging.md#time-travel-debugging">WinDbg - Time Travel Debugging</a></p> <ul> <li>A Review of Reverse Debugging <ul> <li>System, Software, SoC and Silicon Debug Conference (S4D) 2012</li> <li>Jakob Engblom</li> <li><a href="https://ieeexplore.ieee.org/document/6338149/">https://ieeexplore.ieee.org/document/6338149/</a></li> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.338.3420&amp;rep=rep1&amp;type=pdf">http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.338.3420&amp;rep=rep1&amp;type=pdf</a></li> </ul> </li> <li>Don’t Panic: Reverse Debugging of Kernel Drivers <ul> <li>Foundations of Software Engineering (ESEC/FSE) 2015</li> <li>Pavel Dovgalyuk, Denis Dmitriev, Vladimir Makarov</li> <li><a href="https://dl.acm.org/citation.cfm?doid=2786805.2803179">https://dl.acm.org/citation.cfm?doid=2786805.2803179</a></li> <li>Review (Jakob Engblom): Reverse Debug with Hardware in the Loop - <a href="http://jakob.engbloms.se/archives/2432">http://jakob.engbloms.se/archives/2432</a></li> </ul> </li> <li>Efficient Algorithms for Bidirectional Debugging <ul> <li>Programming Language Design and Implementation (PLDI) 2000</li> <li>B. Boothe</li> <li><a href="https://dl.acm.org/citation.cfm?id=349339">https://dl.acm.org/citation.cfm?id=349339</a></li> </ul> </li> <li>Expositor: Scriptable Time-Travel Debugging with First Class Traces <ul> <li>International Conference on Software Engineering (ICSE) 2013</li> <li>Yit Phang Khoo, Jeffrey S. Foster, Michael Hicks</li> <li><a href="http://www.cs.umd.edu/~mwh/papers/khoo13expositor.html">http://www.cs.umd.edu/~mwh/papers/khoo13expositor.html</a></li> <li>full version - <a href="http://www.cs.umd.edu/~mwh/papers/khoo13expositor-journal.html">http://www.cs.umd.edu/~mwh/papers/khoo13expositor-journal.html</a></li> <li>Scriptable time-travel debugging Python library for GDB/UndoDB <ul> <li><a href="https://bitbucket.org/khooyp/expositor/src/default/">https://bitbucket.org/khooyp/expositor/src/default/</a></li> </ul> </li> </ul> </li> <li>Improving the performance of reverse debugging <ul> <li>Programming and Computer Software 43(1) 2017</li> <li>Klimushenkova, M.A. &amp; Dovgalyuk, P.M.</li> <li><a href="https://link.springer.com/article/10.1134/S0361768817010042">https://link.springer.com/article/10.1134/S0361768817010042</a></li> </ul> </li> <li>POMP: Postmortem Program Analysis with Hardware-Enhanced Post-Crash Artifacts <ul> <li>USENIX Security 2017</li> <li>Jun Xu, Dongliang Mu, Xinyu Xing, Peng Liu, Ping Chen, Bing Mao</li> <li><a href="https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/xu-jun">https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/xu-jun</a></li> <li><a href="https://github.com/junxzm1990/pomp">https://github.com/junxzm1990/pomp</a></li> </ul> </li> <li>REPT: Reverse Debugging of Failures in Deployed Software <ul> <li>USENIX Symposium on Operating Systems Design and Implementation (OSDI) 2018</li> <li>Weidong Cui, Xinyang Ge, Baris Kasikci, Ben Niu, Upamanyu Sharma, Ruoyu Wang, Insu Yun</li> <li><a href="https://www.usenix.org/conference/osdi18/presentation/weidong">https://www.usenix.org/conference/osdi18/presentation/weidong</a></li> <li><a href="https://www.microsoft.com/en-us/research/publication/rept-reverse-debugging-of-failures-in-deployed-software/">https://www.microsoft.com/en-us/research/publication/rept-reverse-debugging-of-failures-in-deployed-software/</a></li> </ul> </li> <li>Reverse History <ul> <li>Part One – Background - <a href="http://jakob.engbloms.se/archives/1547">http://jakob.engbloms.se/archives/1547</a></li> <li>Part Two – Research - <a href="http://jakob.engbloms.se/archives/1554">http://jakob.engbloms.se/archives/1554</a></li> <li>Part Three – Products - <a href="http://jakob.engbloms.se/archives/1564">http://jakob.engbloms.se/archives/1564</a></li> </ul> </li> <li>Transition Watchpoints: Teaching Old Debuggers New Tricks <ul> <li>The Art, Science, and Engineering of Programming, 2017, Vol. 1, Issue 2, Article 16</li> <li>Kapil Arya, Tyler Denniston, Ariel Rabkin, Gene Cooperman</li> <li><a href="http://programming-journal.org/2017/1/16/">http://programming-journal.org/2017/1/16/</a></li> </ul> </li> </ul> <h2>Software Engineering</h2> <ul> <li>A Survey on Software Fault Localization <ul> <li>IEEE Transactions on Software Engineering 42(8) (2016)</li> <li>W. Eric Wong, Ruizhi Gao, Yihao Li, Rui Abreu, Franz Wotawa</li> <li><a href="https://www.researchgate.net/publication/291951202_A_Survey_on_Software_Fault_Localization">https://www.researchgate.net/publication/291951202_A_Survey_on_Software_Fault_Localization</a></li> </ul> </li> <li>Are Automated Debugging Techniques Actually Helping Programmers? <ul> <li>International Symposium on Software Testing and Analysis (ISSTA) 2011</li> <li>Chris Parnin, Alessandro Orso</li> <li><a href="https://dl.acm.org/citation.cfm?id=2001445">https://dl.acm.org/citation.cfm?id=2001445</a></li> </ul> </li> <li>Automated Debugging: Are We There Yet? <ul> <li>International Conference on Software Testing, Verification and Validation Workshops (ICSTW) 2011</li> <li>Alex Orso</li> <li><a href="http://ieeexplore.ieee.org/document/5954471/">http://ieeexplore.ieee.org/document/5954471/</a></li> <li>2013 talk <ul> <li><a href="https://www.slideshare.net/alexorso/20130204dagstuhl-alex-orso">https://www.slideshare.net/alexorso/20130204dagstuhl-alex-orso</a></li> <li><a href="https://pdfs.semanticscholar.org/2436/0dc1bd271dcf3ecb73622e2c7b1d54a008bf.pdf">https://pdfs.semanticscholar.org/2436/0dc1bd271dcf3ecb73622e2c7b1d54a008bf.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=WJHQnzLpVXk">https://www.youtube.com/watch?v=WJHQnzLpVXk</a></li> </ul> </li> </ul> </li> <li>Debugging Reinvented: Asking and Answering Why and Why Not Questions about Program Behavior <ul> <li>International Conference on Software Engineering ICSE 2008</li> <li>Andrew J. Ko, Brad A. Myers <ul> <li><a href="http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.1425">http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.159.1425</a></li> <li><a href="https://dl.acm.org/citation.cfm?id=1368130">https://dl.acm.org/citation.cfm?id=1368130</a></li> </ul> </li> <li>Microsoft Research Talk: <ul> <li><a href="https://www.microsoft.com/en-us/research/video/candidate-talk-debugging-reinvented-asking-and-answering-why-and-why-not-questions-about-program-behavior/">https://www.microsoft.com/en-us/research/video/candidate-talk-debugging-reinvented-asking-and-answering-why-and-why-not-questions-about-program-behavior/</a></li> <li><a href="https://www.youtube.com/watch?v=ScZfggGa4qs">https://www.youtube.com/watch?v=ScZfggGa4qs</a></li> <li><a href="https://archive.org/details/Microsoft_Research_Video_103778">https://archive.org/details/Microsoft_Research_Video_103778</a></li> </ul> </li> <li>"Static and dynamic program slicing algorithms for extracting and answering developers questions about program output that substantially decrease fault localization time."</li> <li><a href="https://blog.acolyer.org/2014/10/17/debugging-reinvented/">https://blog.acolyer.org/2014/10/17/debugging-reinvented/</a></li> </ul> </li> <li>On The Dichotomy of Debugging Behavior Among Programmers <ul> <li>ICSE 2018</li> <li>Moritz Beller, Niels Spruit, Diomidis Spinellis, Andy Zaidman</li> <li><a href="http://pure.tudelft.nl/ws/files/38319543/paper.pdf">http://pure.tudelft.nl/ws/files/38319543/paper.pdf</a></li> </ul> </li> <li>Where Is the Bug and How Is It Fixed? An Experiment with Practitioners <ul> <li>European Software Engineering Conference / Foundations of Software Engineering (ESEC/FSE) 2017</li> <li>Marcel Böhme, Ezekiel O. Soremekun, Sudipta Chattopadhyay, Emamurho Ugherughe, Andreas Zeller</li> <li><a href="https://doi.org/10.1145/3106237.3106255">https://doi.org/10.1145/3106237.3106255</a></li> <li><a href="https://www.comp.nus.edu.sg/~mboehme/paper/FSE17.pdf">https://www.comp.nus.edu.sg/~mboehme/paper/FSE17.pdf</a></li> <li>DBGBench - <a href="https://dbgbench.github.io/">https://dbgbench.github.io/</a></li> <li>"the correct fault locations, bug diagnoses, and software patches of 27 real errors in open-source C projects that were consolidated from hundreds of debugging sessions of professional software engineers"</li> </ul> </li> </ul> <h2>Transparency</h2> <ul> <li>2017 - Ninja: Towards Transparent Tracing and Debugging on ARM <ul> <li>USENIX Security 2017</li> <li>Zhenyu Ning, Fengwei Zhang</li> <li><a href="https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/ning">https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/ning</a></li> <li><a href="http://www.cs.wayne.edu/fengwei/paper/ninja-usenixsecurity17.pdf">http://www.cs.wayne.edu/fengwei/paper/ninja-usenixsecurity17.pdf</a></li> </ul> </li> <li>2016 - Towards Transparent Debugging <ul> <li>IEEE Transactions on Dependable and Secure Computing (TDSC'16), 2016.</li> <li>Fengwei Zhang, Kevin Leach, Angelos Stavrou, and Haining Wang</li> <li><a href="http://www.cs.wayne.edu/fengwei/paper/malt-tdsc16.pdf">http://www.cs.wayne.edu/fengwei/paper/malt-tdsc16.pdf</a></li> </ul> </li> <li>2015 - Using Hardware Features for Increased Debugging Transparency <ul> <li>36th IEEE Symposium on Security and Privacy (S&amp;P'15), 2015</li> <li>Fengwei Zhang, Kevin Leach, Angelos Stavrou, Haining Wang, and Kun Sun</li> <li><a href="http://www.cs.wayne.edu/fengwei/paper/malt-sp15.pdf">http://www.cs.wayne.edu/fengwei/paper/malt-sp15.pdf</a></li> </ul> </li> </ul> <hr> <h1>Software</h1> <ul> <li>dbg: A macro for printf-style debugging fans <ul> <li><a href="https://github.com/sharkdp/dbg-macro">https://github.com/sharkdp/dbg-macro</a></li> </ul> </li> <li>LibVMI: Simplified Virtual Machine Introspection <ul> <li>"LibVMI is a virtual machine introspection library. This means that it helps you access the memory of a running virtual machine. LibVMI provides primitives for accessing this memory using physical or virtual addresses and kernel symbols. LibVMI also supports accessing memory from a physical memory snapshot, which is helpful for debugging or forensic analysis."</li> <li><a href="https://github.com/libvmi/libvmi">https://github.com/libvmi/libvmi</a></li> </ul> </li> <li>PulseDbg: Hypervisor-based debugger <ul> <li><a href="https://github.com/honorarybot/PulseDBG">https://github.com/honorarybot/PulseDBG</a></li> </ul> </li> <li>PyREBox: a Python scriptable Reverse Engineering sandbox <ul> <li>"It is based on QEMU, and its goal is to aid reverse engineering by providing dynamic analysis and debugging capabilities from a different perspective. PyREBox allows to inspect a running QEMU VM, modify its memory or registers, and to instrument its execution, by creating simple scripts in python to automate any kind of analysis. QEMU (when working as a whole-system-emulator) emulates a complete system (CPU, memory, devices...). By using VMI techniques, it does not require to perform any modification into the guest operating system, as it transparently retrieves information from its memory at run-time."</li> <li><a href="https://github.com/Cisco-Talos/pyrebox">https://github.com/Cisco-Talos/pyrebox</a></li> <li><a href="http://blog.talosintelligence.com/2017/07/pyrebox.html">http://blog.talosintelligence.com/2017/07/pyrebox.html</a></li> </ul> </li> <li>pyvmidbg: LibVMI-based debug server, implemented in Python <ul> <li>Building a guest aware, stealth and agentless full-system debugger</li> <li><a href="https://github.com/Wenzel/pyvmidbg">https://github.com/Wenzel/pyvmidbg</a></li> <li>Building a Flexible Hypervisor-Level Debugger <ul> <li>Insomni'hack 2019; Mathieu Tarral</li> <li><a href="https://drive.google.com/open?id=1ZMUszfwWDOljdDfPOJgkEfSabNy0UAJR">https://drive.google.com/open?id=1ZMUszfwWDOljdDfPOJgkEfSabNy0UAJR</a><br> ‏* QIRA - QEMU Interactive Runtime Analyser</li> </ul> </li> <li><a href="http://qira.me/">http://qira.me/</a></li> <li><a href="https://github.com/BinaryAnalysisPlatform/qira/">https://github.com/BinaryAnalysisPlatform/qira/</a></li> </ul> </li> <li>Radare2 <ul> <li>Radare project started as a forensics tool, a scriptable commandline hexadecimal editor able to open disk files, but later support for analyzing binaries, disassembling code, debugging programs, attaching to remote gdb servers, etc.</li> <li><a href="http://www.radare.org/">http://www.radare.org/</a></li> <li><a href="https://github.com/radare/radare2">https://github.com/radare/radare2</a></li> <li>r2vmi: Hypervisor-Level Debugger based on Radare2 / LibVMI, using VMI IO and debug plugins <ul> <li><a href="https://github.com/Wenzel/r2vmi">https://github.com/Wenzel/r2vmi</a></li> <li>Hack.lu 2018: Hypervisor-Level Debugger: Benefits And Challenges - Mathieu Tarral <ul> <li><a href="https://www.youtube.com/watch?v=NnWYT-kCx_s">https://www.youtube.com/watch?v=NnWYT-kCx_s</a></li> </ul> </li> <li>r2con2018 - Hypervisor Level Debugger with r2 - Mathieu Tarral <ul> <li><a href="https://www.youtube.com/watch?v=JOJMgWa7E6A">https://www.youtube.com/watch?v=JOJMgWa7E6A</a></li> <li><a href="https://github.com/radareorg/r2con2018/tree/master/talks/10-hypervisor-level-debugger">https://github.com/radareorg/r2con2018/tree/master/talks/10-hypervisor-level-debugger</a></li> </ul> </li> </ul> </li> </ul> </li> <li>ret-sync: Reverse-Engineering Tools SYNChronization <ul> <li>A set of plugins to synchronize a debugging session (WinDbg/GDB/LLDB/OllyDbg/OllyDbg2/x64dbg) with IDA/Ghidra disassemblers.</li> <li><a href="https://github.com/bootleg/ret-sync">https://github.com/bootleg/ret-sync</a></li> </ul> </li> <li>rVMI - A New Paradigm For Full System Analysis <ul> <li>"rVMI is a debugger on steroids. It leverages Virtual Machine Introspection (VMI) and memory forensics to provide full system analysis. This means that an analyst can inspect userspace processes, kernel drivers, and preboot environments in a single tool."</li> <li><a href="https://github.com/fireeye/rVMI">https://github.com/fireeye/rVMI</a></li> <li><a href="https://www.fireeye.com/blog/threat-research/2017/09/rvmi-full-system-analysis.html">https://www.fireeye.com/blog/threat-research/2017/09/rvmi-full-system-analysis.html</a></li> <li>Black Hat USA 2017 <ul> <li><a href="https://www.blackhat.com/docs/us-17/thursday/us-17-Pfoh-rVMI-A-New-Paradigm-For-Full-System-Analysis.pdf">https://www.blackhat.com/docs/us-17/thursday/us-17-Pfoh-rVMI-A-New-Paradigm-For-Full-System-Analysis.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=KtoipviVJjw">https://www.youtube.com/watch?v=KtoipviVJjw</a></li> </ul> </li> </ul> </li> <li>Vivisect / Vdb / Vtrace <ul> <li>Vivisect - interactive disassembler</li> <li>Vtrace - a cross-platform &amp; cross-architecture debugging API</li> <li>VDB - a cross-platform &amp; cross-architecture debugger using Vtrace</li> <li><a href="http://visi.kenshoto.com/viki/MainPage">http://visi.kenshoto.com/viki/MainPage</a></li> <li><a href="https://github.com/vivisect/vivisect">https://github.com/vivisect/vivisect</a></li> <li>documentation: <a href="http://fitblip.pub/vdb-fork/sphinx/">http://fitblip.pub/vdb-fork/sphinx/</a></li> <li>fork &amp; documentation: <a href="http://fitblip.pub/vdb-fork/">http://fitblip.pub/vdb-fork/</a></li> <li>Using a Custom VDB Debugger for Exploit Analysis - <a href="https://www.fireeye.com/blog/threat-research/2013/02/custom-vdb-debugger-exploit-analysis.html">https://www.fireeye.com/blog/threat-research/2013/02/custom-vdb-debugger-exploit-analysis.html</a></li> <li>Malware Analysis with Vivisect - NEST - <a href="http://nest.unm.edu/files/5514/1254/9114/Malware_Analysis_with_Vivsect.pdf">http://nest.unm.edu/files/5514/1254/9114/Malware_Analysis_with_Vivsect.pdf</a></li> <li>Binary Vivisection <ul> <li><a href="https://www.singlehop.com/blog/binary-vivisection-part-1/">https://www.singlehop.com/blog/binary-vivisection-part-1/</a></li> <li><a href="https://www.singlehop.com/blog/binary-vivisection-part-2/">https://www.singlehop.com/blog/binary-vivisection-part-2/</a></li> <li><a href="https://www.singlehop.com/blog/binary-vivisection-part-3/">https://www.singlehop.com/blog/binary-vivisection-part-3/</a></li> </ul> </li> <li>FireEye Labs Query-Oriented Debugger <ul> <li>Command-line and Python debugger for instrumenting and modifying native software behavior on Windows and Linux</li> <li><a href="https://github.com/fireeye/flare-qdb">https://github.com/fireeye/flare-qdb</a></li> <li>Querying Dynamic State using the FireEye Labs Query-Oriented Debugger (flare-qdb) <ul> <li><a href="https://www.fireeye.com/blog/threat-research/2017/01/flare_script_series.html">https://www.fireeye.com/blog/threat-research/2017/01/flare_script_series.html</a></li> </ul> </li> </ul> </li> </ul> </li> <li>Voltron - <a href="https://github.com/snare/voltron">https://github.com/snare/voltron</a></li> <li>xendbg - A modern Xen debugger <ul> <li><a href="https://github.com/nccgroup/xendbg">https://github.com/nccgroup/xendbg</a></li> <li>"<code>xendbg</code> is a feature-complete reference implementation of a modern Xen VMI debugger, superseding Xen's own limited and rarely-maintained <a href="https://github.com/mirage/xen/tree/master/tools/debugger/gdbsx"><code>gdbsx</code></a>. It can debug both paravirtualized (PV) and hardware virtualized (HVM) guests, and provides both a standalone REPL and an LLDB server mode."</li> <li>Xendbg: A Full-Featured Debugger for the Xen Hypervisor <ul> <li><a href="https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/january/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/">https://www.nccgroup.trust/us/about-us/newsroom-and-events/blog/2019/january/xendbg-a-full-featured-debugger-for-the-xen-hypervisor/</a></li> </ul> </li> </ul> </li> </ul> <h2>GDB</h2> <ul> <li>GDB: The GNU Project Debugger <ul> <li><a href="https://sourceware.org/gdb/">https://sourceware.org/gdb/</a></li> <li><a href="https://sourceware.org/gdb/documentation/">https://sourceware.org/gdb/documentation/</a></li> <li><a href="https://sourceware.org/gdb/wiki/">https://sourceware.org/gdb/wiki/</a></li> </ul> </li> <li>GDB - The Architecture of Open Source Applications - Stan Shebs <ul> <li><a href="http://aosabook.org/en/gdb.html">http://aosabook.org/en/gdb.html</a></li> </ul> </li> </ul> <h3>Projects</h3> <ul> <li>CGDB: Console front-end to the GNU debugger <ul> <li><a href="https://github.com/cgdb/cgdb">https://github.com/cgdb/cgdb</a></li> </ul> </li> <li>Gdb Assembly Informant <ul> <li>steps through your assembly code one instruction at a time and diffs register values</li> <li><a href="https://github.com/thlorenz/gai">https://github.com/thlorenz/gai</a></li> </ul> </li> <li>GDB dashboard <ul> <li><a href="https://github.com/cyrus-and/gdb-dashboard">https://github.com/cyrus-and/gdb-dashboard</a></li> <li><a href="https://metricpanda.com/tips-for-productive-debugging-with-gdb">https://metricpanda.com/tips-for-productive-debugging-with-gdb</a></li> </ul> </li> <li>GDB helper scripts <ul> <li><a href="https://github.com/tromey/gdb-helpers">https://github.com/tromey/gdb-helpers</a></li> </ul> </li> <li>gdb-gui: A gdb gui written in Python, running inside gdb itself <ul> <li><a href="https://github.com/tromey/gdb-gui">https://github.com/tromey/gdb-gui</a></li> </ul> </li> <li>gdb-helpers: GDB helper scripts <ul> <li><a href="https://github.com/tromey/gdb-helpers">https://github.com/tromey/gdb-helpers</a></li> </ul> </li> <li>gdb-tools: Various tools to improve the gdb experience <ul> <li><a href="https://github.com/vuvova/gdb-tools">https://github.com/vuvova/gdb-tools</a></li> <li>gdb tools: duel and @PrettyPrinter - <a href="https://fosdem.org/2018/schedule/event/debugging_tools_gdb_tools/">https://fosdem.org/2018/schedule/event/debugging_tools_gdb_tools/</a></li> </ul> </li> <li>gdb-walkers: Bring mdb walkers to gdb, also add other helpful commands. <ul> <li><a href="https://github.com/hardenedapple/gdb-walkers">https://github.com/hardenedapple/gdb-walkers</a></li> <li>GDB pipelines -- convenience iteration over inferior data structures <ul> <li>Bringing MDB's "walkers" to GDB</li> <li>FOSDEM 2020; Matthew Malcomson</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_gdb_pipelines/">https://fosdem.org/2020/schedule/event/debugging_gdb_pipelines/</a></li> </ul> </li> </ul> </li> <li>GDBFrontend <ul> <li>"an easy, flexible and extensionable gui debugger"</li> <li><a href="https://github.com/rohanrhu/gdb-frontend">https://github.com/rohanrhu/gdb-frontend</a></li> </ul> </li> <li>gdbgui: A browser-based frontend for GDB <ul> <li><a href="https://gdbgui.com/">https://gdbgui.com/</a></li> <li><a href="https://github.com/cs01/gdbgui">https://github.com/cs01/gdbgui</a></li> </ul> </li> <li>GdbShellPipe: Enable piping of internal command output to external commands <ul> <li><a href="https://github.com/hq6/GdbShellPipe">https://github.com/hq6/GdbShellPipe</a></li> </ul> </li> <li>GDBundle: Plugin Manager for GDB and LLDB <ul> <li><a href="https://github.com/memfault/gdbundle">https://github.com/memfault/gdbundle</a></li> <li>gdbundle - GDB and LLDB's Missing Plugin Manager <ul> <li><a href="https://interrupt.memfault.com/blog/gdbundle-plugin-manager">https://interrupt.memfault.com/blog/gdbundle-plugin-manager</a></li> </ul> </li> </ul> </li> <li>Gede: a graphical frontend (GUI) to GDB written in Qt <ul> <li><a href="http://gede.acidron.com/">http://gede.acidron.com/</a></li> <li>mirror: <a href="https://github.com/Nanoseb/gede">https://github.com/Nanoseb/gede</a></li> </ul> </li> <li>GEF (GDB Enhanced Features) <ul> <li><a href="https://github.com/hugsy/gef">https://github.com/hugsy/gef</a></li> <li><a href="https://github.com/hugsy/gef-scripts">https://github.com/hugsy/gef-scripts</a></li> <li><a href="https://github.com/hugsy/gef-structs">https://github.com/hugsy/gef-structs</a></li> <li>GEF Tutorials - <a href="https://www.youtube.com/playlist?list=PLjAuO31Rg972WeMvdR_57Qu-aVM8T6DkQ">https://www.youtube.com/playlist?list=PLjAuO31Rg972WeMvdR_57Qu-aVM8T6DkQ</a></li> <li><a href="https://blahcat.github.io/2017/08/01/gef-at-black-hat-arsenal-us-2017/">https://blahcat.github.io/2017/08/01/gef-at-black-hat-arsenal-us-2017/</a></li> <li><a href="https://github.com/toolswatch/blackhat-arsenal-tools/blob/master/exploitation/gef.md">https://github.com/toolswatch/blackhat-arsenal-tools/blob/master/exploitation/gef.md</a></li> </ul> </li> <li>libdebugme: Automatically drop to gdb on error <ul> <li><a href="https://github.com/yugr/libdebugme">https://github.com/yugr/libdebugme</a></li> </ul> </li> <li>OnlineGDB <ul> <li>"OnlineGDB an online compiler and debugger tool for C/C++ languages. It is world's first online IDE which gives debugging facility with embedded gdb."</li> <li><a href="http://OnlineGDB.com">http://OnlineGDB.com</a></li> </ul> </li> <li>PEDA - Python Exploit Development Assistance for GDB <ul> <li><a href="https://github.com/longld/peda">https://github.com/longld/peda</a></li> <li><a href="https://eugenekolo.com/blog/better-disassembly-with-gdb-peda/">https://eugenekolo.com/blog/better-disassembly-with-gdb-peda/</a></li> <li><a href="http://ropshell.com/peda/Linux_Interactive_Exploit_Development_with_GDB_and_PEDA_Slides.pdf">http://ropshell.com/peda/Linux_Interactive_Exploit_Development_with_GDB_and_PEDA_Slides.pdf</a></li> </ul> </li> <li>PINCE: front-end reverse engineering tool for the GDB - <a href="https://github.com/korcankaraokcu/PINCE">https://github.com/korcankaraokcu/PINCE</a></li> <li>pwndbg - Exploit Development and Reverse Engineering with GDB Made Easy <ul> <li><a href="https://github.com/pwndbg/pwndbg">https://github.com/pwndbg/pwndbg</a></li> <li><a href="https://github.com/pwndbg/pwndbg/blob/dev/FEATURES.md">https://github.com/pwndbg/pwndbg/blob/dev/FEATURES.md</a></li> </ul> </li> <li>Pwngdb - <a href="https://github.com/scwuaptx/Pwngdb">https://github.com/scwuaptx/Pwngdb</a></li> <li>pygdbmi - Get Structured Output from GDB's Machine Interface - <a href="https://github.com/cs01/pygdbmi">https://github.com/cs01/pygdbmi</a></li> <li>QuickPatch: a GDB plug-in to patch an ELF file <ul> <li><a href="https://github.com/invictus1306/QuickPatch">https://github.com/invictus1306/QuickPatch</a></li> <li><a href="https://invictus1306.github.io/vulnerabilities/2019/10/20/quickpatch.html">https://invictus1306.github.io/vulnerabilities/2019/10/20/quickpatch.html</a></li> </ul> </li> <li>SymGDB - symbolic execution plugin for gdb - <a href="https://github.com/SQLab/symgdb">https://github.com/SQLab/symgdb</a></li> </ul> <h4>Editor Integration</h4> <ul> <li>GDB-MI: a package by Nick Roberts which makes Emacs use GDB/MI interface to talk with the GNU Debugger <ul> <li><a href="https://www.emacswiki.org/emacs/GDB-MI">https://www.emacswiki.org/emacs/GDB-MI</a></li> </ul> </li> <li>GDB graphical interface for GNU Emacs <ul> <li><a href="https://github.com/weirdNox/emacs-gdb">https://github.com/weirdNox/emacs-gdb</a></li> </ul> </li> <li>Native Debug <ul> <li>GDB &amp; LLDB Debugger support for VSCode</li> <li><a href="https://marketplace.visualstudio.com/items?itemName=webfreak.debug">https://marketplace.visualstudio.com/items?itemName=webfreak.debug</a></li> <li><a href="https://github.com/WebFreak001/code-debug">https://github.com/WebFreak001/code-debug</a></li> </ul> </li> <li>neogdb.vim: Vim GDB front-end for neovim <ul> <li><a href="https://github.com/huawenyu/neogdb.vim">https://github.com/huawenyu/neogdb.vim</a></li> </ul> </li> <li>RealGUD: An extensible, modular GNU Emacs front-end for interacting with external debuggers <ul> <li><a href="https://github.com/realgud/realgud">https://github.com/realgud/realgud</a></li> </ul> </li> </ul> <h4>Memory Debugging</h4> <ul> <li>pahole-gdb: pahole implementation for gdb - <a href="https://github.com/PhilArmstrong/pahole-gdb">https://github.com/PhilArmstrong/pahole-gdb</a></li> <li>gdb-heap <ul> <li><a href="https://github.com/rogerhu/gdb-heap">https://github.com/rogerhu/gdb-heap</a></li> <li><a href="https://fedorahosted.org/gdb-heap/">https://fedorahosted.org/gdb-heap/</a></li> <li><a href="https://fedoraproject.org/wiki/Features/MemoryDebuggingTools">https://fedoraproject.org/wiki/Features/MemoryDebuggingTools</a></li> </ul> </li> </ul> <h4>Plotting</h4> <ul> <li>GDBplotlib: Plotting and exporting of variables from GDB <ul> <li><a href="https://github.com/X-Neon/gdbplotlib">https://github.com/X-Neon/gdbplotlib</a></li> </ul> </li> <li>gdb-plot <ul> <li>a set of utils for: plotting from the gdb command line, saving c data to .mat files from gdb command line, exploring the stack frame</li> <li><a href="https://github.com/bthcode/gdb-plot">https://github.com/bthcode/gdb-plot</a></li> </ul> </li> </ul> <h4>Profiling</h4> <ul> <li>gdbpmp: A GDB Based Wallclock Profiler - <a href="https://github.com/markhpc/gdbpmp">https://github.com/markhpc/gdbpmp</a></li> <li>gdbprof: A wall clock time-based profiler built on GDB's Python interface <ul> <li><a href="https://github.com/Muon/gdbprof">https://github.com/Muon/gdbprof</a></li> <li><a href="https://github.com/markhpc/gdbprof">https://github.com/markhpc/gdbprof</a></li> </ul> </li> <li>GDB profiler: Rich man's profiler, a profiler for native OCaml and other executables <ul> <li><a href="https://github.com/copy/gdbprofiler">https://github.com/copy/gdbprofiler</a></li> </ul> </li> </ul> <h3>Readings</h3> <ul> <li>8 gdb tricks you should know - <a href="https://blogs.oracle.com/ksplice/8-gdb-tricks-you-should-know">https://blogs.oracle.com/ksplice/8-gdb-tricks-you-should-know</a></li> <li>Beej's Quick Guide to GDB - <a href="https://beej.us/guide/bggdb/">https://beej.us/guide/bggdb/</a></li> <li>Cheatsheet - <a href="https://github.com/jshaw87/Cheatsheets/blob/master/Cheatsheet_GDB.txt">https://github.com/jshaw87/Cheatsheets/blob/master/Cheatsheet_GDB.txt</a></li> <li>Displaying Stack Frames in gdb with Python - <a href="http://jefftrull.github.io/c++/gdb/python/2018/03/02/print-frame.html">http://jefftrull.github.io/c++/gdb/python/2018/03/02/print-frame.html</a></li> <li>Fast Tracing with GDB - <a href="https://suchakra.wordpress.com/2016/06/29/fast-tracing-with-gdb/">https://suchakra.wordpress.com/2016/06/29/fast-tracing-with-gdb/</a></li> <li>GDB Basics Tutorial <ul> <li><a href="https://platform.avatao.com/paths/a0dc20fc-f1b5-43c9-89fc-3a5fccfb5f0b/challenges/166366b3-2e89-49ee-86a3-023663d197b7">https://platform.avatao.com/paths/a0dc20fc-f1b5-43c9-89fc-3a5fccfb5f0b/challenges/166366b3-2e89-49ee-86a3-023663d197b7</a></li> </ul> </li> <li>GDB Custom Commands: Dynamic Arrays <ul> <li><a href="https://testfit.io/devblog/gdb_custom_commands_dynamic_arrays">https://testfit.io/devblog/gdb_custom_commands_dynamic_arrays</a></li> </ul> </li> <li>GDB Debugging Automation with Python: Implementing a memory leak detector - <a href="https://nativecoding.wordpress.com/2016/07/31/gdb-debugging-automation-with-python/">https://nativecoding.wordpress.com/2016/07/31/gdb-debugging-automation-with-python/</a></li> <li>gdb Debugging Full Example (Tutorial): ncurses - <a href="http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html">http://www.brendangregg.com/blog/2016-08-09/gdb-example-ncurses.html</a></li> <li>GDB scripting and Indirect functions <ul> <li><a href="https://fasterthanli.me/blog/2020/gdb-scripting-and-indirect-functions/">https://fasterthanli.me/blog/2020/gdb-scripting-and-indirect-functions/</a></li> </ul> </li> <li>GDB Tips and Tricks - Shane Kirk <ul> <li>1: A Tale of Two Terminals - <a href="http://www.shanekirk.com/2017/08/gdb-tips-and-tricks-1-a-tale-of-two-terminals/">http://www.shanekirk.com/2017/08/gdb-tips-and-tricks-1-a-tale-of-two-terminals/</a></li> <li>2: Setting Breakpoints with Regular Expressions - <a href="http://www.shanekirk.com/2017/08/gdb-tips-and-tricks-2-setting-breakpoints-with-regular-expressions/">http://www.shanekirk.com/2017/08/gdb-tips-and-tricks-2-setting-breakpoints-with-regular-expressions/</a></li> <li>3: Saving and Restoring Breakpoints Using Files - <a href="http://www.shanekirk.com/2017/09/gdb-tips-and-tricks-3-saving-and-restoring-breakpoints-using-files/">http://www.shanekirk.com/2017/09/gdb-tips-and-tricks-3-saving-and-restoring-breakpoints-using-files/</a></li> <li>4: Reverse Debugging - <a href="http://www.shanekirk.com/2017/10/gdb-tips-and-tricks-4-reverse-debugging/">http://www.shanekirk.com/2017/10/gdb-tips-and-tricks-4-reverse-debugging/</a></li> <li>5: The Display Command - <a href="http://www.shanekirk.com/2017/12/gdb-tips-and-tricks-5-the-display-command/">http://www.shanekirk.com/2017/12/gdb-tips-and-tricks-5-the-display-command/</a></li> <li>6: Examining Data Types - <a href="http://www.shanekirk.com/2018/02/gdb-tips-and-tricks-6-examining-data-types/">http://www.shanekirk.com/2018/02/gdb-tips-and-tricks-6-examining-data-types/</a></li> </ul> </li> <li>GDB Tutorial - <a href="https://www.gdb-tutorial.net/">https://www.gdb-tutorial.net/</a></li> <li>GDB Wiki - <a href="https://sourceware.org/gdb/wiki/">https://sourceware.org/gdb/wiki/</a></li> <li>gdbWatchPoint: GDB tips &amp; tricks <ul> <li><a href="https://undo.io/resources/gdb-watchpoint/">https://undo.io/resources/gdb-watchpoint/</a></li> <li><a href="https://www.youtube.com/playlist?list=PLrADU7tRzDD4xHv_zVoLKYg9mXYjXGhjG">https://www.youtube.com/playlist?list=PLrADU7tRzDD4xHv_zVoLKYg9mXYjXGhjG</a></li> </ul> </li> <li>Hitchikers Guide To The GDB - <a href="http://apoorvaj.io/hitchhikers-guide-to-the-gdb.html">http://apoorvaj.io/hitchhikers-guide-to-the-gdb.html</a></li> <li>How to point GDB to your sources - <a href="https://alex.dzyoba.com/blog/gdb-source-path/">https://alex.dzyoba.com/blog/gdb-source-path/</a></li> <li>Howto: GDB Remote Serial Protocol <ul> <li><a href="https://www.embecosm.com/appnotes/ean4/embecosm-howto-rsp-server-ean4-issue-2.html">https://www.embecosm.com/appnotes/ean4/embecosm-howto-rsp-server-ean4-issue-2.html</a></li> </ul> </li> <li>Introduction to Debuggers - Saumil Shah - <a href="http://www.slideshare.net/saumilshah/introduction-to-debuggers">http://www.slideshare.net/saumilshah/introduction-to-debuggers</a></li> <li>Jeff Trull <ul> <li>Displaying Stack Frames in gdb with Python - <a href="http://jefftrull.github.io/c++/gdb/python/2018/03/02/print-frame.html">http://jefftrull.github.io/c++/gdb/python/2018/03/02/print-frame.html</a></li> <li>Improving C++ backtraces with the gdb Python API - <a href="http://jefftrull.github.io/c++/gdb/python/2018/04/24/improved-backtrace.html">http://jefftrull.github.io/c++/gdb/python/2018/04/24/improved-backtrace.html</a></li> <li>Skipping library code in gdb with help from libClang - <a href="http://jefftrull.github.io/c++/gdb/python/libclang/llvm/2018/04/30/stepping-with-libclang.html">http://jefftrull.github.io/c++/gdb/python/libclang/llvm/2018/04/30/stepping-with-libclang.html</a></li> </ul> </li> <li>Julia Evans <ul> <li>Three steps to learning GDB - <a href="https://jvns.ca/blog/2014/02/10/three-steps-to-learning-gdb/">https://jvns.ca/blog/2014/02/10/three-steps-to-learning-gdb/</a></li> <li>How does gdb work? - <a href="https://jvns.ca/blog/2016/08/10/how-does-gdb-work/">https://jvns.ca/blog/2016/08/10/how-does-gdb-work/</a></li> <li>How does gdb call functions? - <a href="https://jvns.ca/blog/2018/01/04/how-does-gdb-call-functions/">https://jvns.ca/blog/2018/01/04/how-does-gdb-call-functions/</a></li> </ul> </li> <li>Notes on GDB - <a href="https://publicclu2.blogspot.com/2013/05/notes-on-gdb.html">https://publicclu2.blogspot.com/2013/05/notes-on-gdb.html</a></li> <li>Object Inspection in GDB - <a href="https://hgad.net/posts/object-inspection-in-gdb/">https://hgad.net/posts/object-inspection-in-gdb/</a></li> <li>Process Injection with GDB - <a href="https://magisterquis.github.io/2018/03/11/process-injection-with-gdb.html">https://magisterquis.github.io/2018/03/11/process-injection-with-gdb.html</a></li> <li>Python Interpreter in GNU Debugger <ul> <li><a href="https://www.pythonsheets.com/appendix/python-gdb.html">https://www.pythonsheets.com/appendix/python-gdb.html</a></li> </ul> </li> <li>RevEngE is a dish served cold: Debug-Oriented Malware Decompilation and Reassembly <ul> <li>Reversing and Offensive-oriented Trends Symposium (ROOTS) 2019</li> <li>Marcus Botacin, Lucas Galante, Paulo de Geus, André Grégio</li> <li><a href="https://lasca.ic.unicamp.br/paulo/papers/2019-ROOTS-marcus.botacin-lucas.galante-RevEngE.pdf">https://lasca.ic.unicamp.br/paulo/papers/2019-ROOTS-marcus.botacin-lucas.galante-RevEngE.pdf</a></li> <li><a href="https://corvus.inf.ufpr.br/">https://corvus.inf.ufpr.br/</a></li> <li>RevEngE: Reverse Engineering Engine <ul> <li>Explore debugging extensions and malware decompilation capabilities based on dynamic GDB debugging sessions.</li> <li><a href="https://github.com/marcusbotacin/Reverse.Engineering.Engine">https://github.com/marcusbotacin/Reverse.Engineering.Engine</a></li> </ul> </li> </ul> </li> <li>Skipping standard C++ library during debug session in gdb - <a href="https://xaizek.github.io/2016-05-26/skipping-standard-library-in-gdb/">https://xaizek.github.io/2016-05-26/skipping-standard-library-in-gdb/</a></li> <li>Victor Stinner's Notes - GDB: GNU debugger - <a href="http://vstinner.readthedocs.io/gdb.html">http://vstinner.readthedocs.io/gdb.html</a></li> </ul> <h3>Talks</h3> <ul> <li>A flexible GDB (GNU Debugger) target description for processor diversity – SFO17-210 <ul> <li><a href="http://connect.linaro.org/resource/sfo17/sfo17-210/">http://connect.linaro.org/resource/sfo17/sfo17-210/</a></li> </ul> </li> <li>Become a GDB Power User - ACCU 2016 - Greg Law <ul> <li><a href="https://www.youtube.com/watch?v=713ay4bZUrw">https://www.youtube.com/watch?v=713ay4bZUrw</a></li> <li>including Q&amp;A: <a href="https://www.youtube.com/watch?v=6ag7yvhDAiE">https://www.youtube.com/watch?v=6ag7yvhDAiE</a></li> </ul> </li> <li>Debugging Linux C++ <ul> <li>CppCon 2018; Greg Law</li> <li><a href="https://www.youtube.com/watch?v=V1t6faOKjuQ">https://www.youtube.com/watch?v=V1t6faOKjuQ</a></li> </ul> </li> <li>GDB: C++ conversion &amp; dogfooding C++ <ul> <li>GNU Tools Cauldron 2017; Pedro Alves</li> <li>Slides: <a href="https://gcc.gnu.org/wiki/cauldron2017?action=AttachFile&amp;do=view&amp;target=gdb+-+C%2B%2B+conversion+%26+dogfood.pdf">https://gcc.gnu.org/wiki/cauldron2017?action=AttachFile&amp;do=view&amp;target=gdb+-+C%2B%2B+conversion+%26+dogfood.pdf</a></li> <li>Video: <a href="https://slideslive.com/38902352/gdb-c-conversion-dogfooding-c">https://slideslive.com/38902352/gdb-c-conversion-dogfooding-c</a></li> </ul> </li> <li>GDB - A Lot More Than You Knew - CppCon 2016 - Greg Law <ul> <li><a href="https://www.youtube.com/watch?v=-n9Fkq1e6sg">https://www.youtube.com/watch?v=-n9Fkq1e6sg</a></li> </ul> </li> <li>Give me 15 minutes &amp; I'll change your view of GDB - CppCon 2015 - Greg Law <ul> <li><a href="https://www.youtube.com/watch?v=PorfLSr3DDI">https://www.youtube.com/watch?v=PorfLSr3DDI</a></li> </ul> </li> <li>How custom GDB commands help in C++ development <ul> <li>Munich C++ User Group 2018 (Lightning Talk); Michael Krasnyk</li> <li><a href="https://www.youtube.com/watch?v=QtTYXE1wSVs">https://www.youtube.com/watch?v=QtTYXE1wSVs</a></li> </ul> </li> <li>Improving Debuggability with GDB's Python API - C++Now 2018 - Jeff Trull <ul> <li><a href="https://www.youtube.com/watch?v=mLPp1x_1h3g">https://www.youtube.com/watch?v=mLPp1x_1h3g</a></li> </ul> </li> <li>Liberating the Debugging Experience with the GDB Python API <ul> <li>ACCU Bay Area Meetup 2018; Jeff Trull</li> <li><a href="https://github.com/accuBayArea/Slides/blob/master/slides/2018-08-08-jeff.pdf">https://github.com/accuBayArea/Slides/blob/master/slides/2018-08-08-jeff.pdf</a></li> <li>CppCon 2018 - <a href="https://www.youtube.com/watch?v=ck_jCH_G7pA">https://www.youtube.com/watch?v=ck_jCH_G7pA</a></li> <li>gdb_python_api: Experiments with the GDB Python API <ul> <li><a href="https://github.com/jefftrull/gdb_python_api">https://github.com/jefftrull/gdb_python_api</a></li> </ul> </li> </ul> </li> <li>More GDB wizardry and 8 other essential Linux application debugging tools <ul> <li>ACCU 2019; Greg Law</li> <li><a href="https://www.youtube.com/watch?v=Yq6g_kvyvPU">https://www.youtube.com/watch?v=Yq6g_kvyvPU</a></li> </ul> </li> <li>Programmatic Debugging with GDB and Python <ul> <li>PyCon APAC/Taiwan 2015; Scott Tsai</li> <li><a href="https://www.youtube.com/watch?v=oAYbt2PsKng">https://www.youtube.com/watch?v=oAYbt2PsKng</a></li> <li><a href="https://docs.google.com/presentation/d/15qOKBh9FDjCeGS-xAHXZSJDS5_aoZk0Caz12FL_f294/">https://docs.google.com/presentation/d/15qOKBh9FDjCeGS-xAHXZSJDS5_aoZk0Caz12FL_f294/</a></li> </ul> </li> <li>SecurityTube GDB Expert (SGDE) <ul> <li>Walkthroughs: <a href="https://github.com/Kan1shka9/Securitytube-Gnu-Debugger-Expert">https://github.com/Kan1shka9/Securitytube-Gnu-Debugger-Expert</a></li> <li>Course Videos: <a href="http://www.securitytube.net/tags/sgde">http://www.securitytube.net/tags/sgde</a></li> <li><a href="https://www.youtube.com/playlist?list=PLiP0FxVgYuUz0kdK7L7YaI5n4qkOuymue">https://www.youtube.com/playlist?list=PLiP0FxVgYuUz0kdK7L7YaI5n4qkOuymue</a></li> </ul> </li> <li>The GDB Text User Interface <ul> <li>FOSDEM 2020; Tom Tromey</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_gdb_tui/">https://fosdem.org/2020/schedule/event/debugging_gdb_tui/</a></li> </ul> </li> <li>Understanding, Scripting, and Extending GDB (2017) <ul> <li><a href="https://mcgdb.0x972.info/team/">https://mcgdb.0x972.info/team/</a></li> <li><a href="https://github.com/kpouget/tuto-gdb.py">https://github.com/kpouget/tuto-gdb.py</a></li> <li>Slides: <a href="https://mcgdb.0x972.info/files/tuto-gdb-py.presentation.pdf">https://mcgdb.0x972.info/files/tuto-gdb-py.presentation.pdf</a></li> </ul> </li> <li>Your Application versus GDB - FOSDEM 2014 - Tom Tromey <ul> <li><a href="https://archive.fosdem.org/2014/schedule/event/your_application_versus_gdb/">https://archive.fosdem.org/2014/schedule/event/your_application_versus_gdb/</a></li> <li><a href="https://www.youtube.com/watch?v=RwDA3oIOtWw">https://www.youtube.com/watch?v=RwDA3oIOtWw</a></li> </ul> </li> </ul> <h2>LLDB</h2> <ul> <li>The LLDB Debugger - <a href="https://lldb.llvm.org/">https://lldb.llvm.org/</a></li> </ul> <h3>Projects</h3> <ul> <li>ds2: Debug server for lldb <ul> <li>ds2 is a debug server designed to be used with LLDB to perform remote debugging of Linux, Android, FreeBSD, Windows and Windows Phone targets. Windows/Windows Phone support is still under active development.</li> <li><a href="https://github.com/facebook/ds2">https://github.com/facebook/ds2</a></li> <li>2015 LLVM Developers’ Meeting: Stephane Sezer "ds2, a tiny debug server used with lldb"</li> <li><a href="https://www.youtube.com/watch?v=n00EhLskJWk&amp;list=PL_R5A0lGi1AA4Lv2bBFSwhgDaHvvpVU21&amp;index=27">https://www.youtube.com/watch?v=n00EhLskJWk&amp;list=PL_R5A0lGi1AA4Lv2bBFSwhgDaHvvpVU21&amp;index=27</a></li> </ul> </li> <li>GDBundle: Plugin Manager for GDB and LLDB <ul> <li><a href="https://github.com/memfault/gdbundle">https://github.com/memfault/gdbundle</a></li> <li>gdbundle - GDB and LLDB's Missing Plugin Manager <ul> <li><a href="https://interrupt.memfault.com/blog/gdbundle-plugin-manager">https://interrupt.memfault.com/blog/gdbundle-plugin-manager</a></li> </ul> </li> </ul> </li> <li>LLDB Scripts: A collection of LLDB aliases/regexes and Python scripts to aid in your debugging sessions <ul> <li><a href="https://github.com/DerekSelander/LLDB">https://github.com/DerekSelander/LLDB</a></li> <li><a href="https://www.raywenderlich.com/162020/custom-lldb-commands-practice">https://www.raywenderlich.com/162020/custom-lldb-commands-practice</a></li> <li><a href="https://www.raywenderlich.com/161938/assembly-register-calling-convention-tutorial">https://www.raywenderlich.com/161938/assembly-register-calling-convention-tutorial</a></li> </ul> </li> <li>lldbinit: Similar implementation of .gdbinit from fG! for lldb in python <ul> <li><a href="https://github.com/deroko/lldbinit">https://github.com/deroko/lldbinit</a></li> </ul> </li> <li>LLDBINIT: A gdbinit clone for LLDB <ul> <li><a href="https://github.com/gdbinit/lldbinit">https://github.com/gdbinit/lldbinit</a></li> <li><a href="https://www.zerodayinitiative.com/blog/2020/4/20/mindshare-using-lldbinit-to-enhance-the-lldb-debugger">https://www.zerodayinitiative.com/blog/2020/4/20/mindshare-using-lldbinit-to-enhance-the-lldb-debugger</a></li> </ul> </li> <li>Vegvisir: A browser based GUI for LLDB Debugger <ul> <li><a href="https://github.com/ant4g0nist/vegvisir">https://github.com/ant4g0nist/vegvisir</a></li> </ul> </li> <li>vplot: C++ container graph visualization for lldb <ul> <li><a href="https://github.com/egladysh/vplot">https://github.com/egladysh/vplot</a></li> </ul> </li> </ul> <h4>Editor Integration</h4> <ul> <li>CodeLLDB: a LLDB front end for Visual Studio Code - <a href="https://github.com/vadimcn/vscode-lldb">https://github.com/vadimcn/vscode-lldb</a></li> <li>LLDB Vim Frontend - <a href="https://github.com/gilligan/vim-lldb">https://github.com/gilligan/vim-lldb</a></li> <li>LLDB Neovim Frontend - <a href="https://github.com/dbgx/lldb.nvim">https://github.com/dbgx/lldb.nvim</a></li> <li>Native Debug <ul> <li>GDB &amp; LLDB Debugger support for VSCode</li> <li><a href="https://marketplace.visualstudio.com/items?itemName=webfreak.debug">https://marketplace.visualstudio.com/items?itemName=webfreak.debug</a></li> <li><a href="https://github.com/WebFreak001/code-debug">https://github.com/WebFreak001/code-debug</a></li> </ul> </li> </ul> <h3>Readings</h3> <ul> <li>LLDB Cheat Sheet <ul> <li><a href="https://www.nesono.com/content/lldb-cheat-sheet">https://www.nesono.com/content/lldb-cheat-sheet</a></li> <li><a href="https://www.nesono.com/sites/default/files/lldb%20cheat%20sheet.pdf">https://www.nesono.com/sites/default/files/lldb%20cheat%20sheet.pdf</a></li> </ul> </li> <li>LLDB for GDB Users – Command Summary <ul> <li><a href="https://developer.apple.com/library/content/documentation/General/Conceptual/lldb-guide/chapters/A3-GDB-Summary.html">https://developer.apple.com/library/content/documentation/General/Conceptual/lldb-guide/chapters/A3-GDB-Summary.html</a></li> </ul> </li> <li>LLDB Scripts - Debugging the Swift Compiler <ul> <li><a href="https://github.com/apple/swift/blob/master/docs/DebuggingTheCompiler.rst#lldb-scripts">https://github.com/apple/swift/blob/master/docs/DebuggingTheCompiler.rst#lldb-scripts</a></li> </ul> </li> <li>LLDB to GDB Command Map - <a href="https://lldb.llvm.org/lldb-gdb.html">https://lldb.llvm.org/lldb-gdb.html</a></li> <li>Beyond Debug Information: Improving Program Reconstruction in LLDB using C++ Modules <ul> <li>2019 Master’s Thesis; Raphael Isemann</li> <li><a href="https://hdl.handle.net/20.500.12380/300037">https://hdl.handle.net/20.500.12380/300037</a></li> </ul> </li> </ul> <h3>Talks</h3> <ul> <li>Support for mini-debuginfo in LLDB <ul> <li>How to read the .gnu_debugdata section.</li> <li>FOSDEM 2020; Konrad Kleine</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_mini/">https://fosdem.org/2020/schedule/event/debugging_mini/</a></li> </ul> </li> <li>Better C++ debugging using Clang Modules in LLDB <ul> <li>2019 LLVM Developers’ Meeting; Raphael Isemann</li> <li><a href="https://www.youtube.com/watch?v=vuNZLlHhy0k">https://www.youtube.com/watch?v=vuNZLlHhy0k</a></li> <li><a href="http://llvm.org/devmtg/2019-10/talk-abstracts.html#tech13">http://llvm.org/devmtg/2019-10/talk-abstracts.html#tech13</a></li> </ul> </li> <li>Bringing RenderScript to LLDB <ul> <li>2016 EuroLLVM Developers' Meeting; Luke Drummond &amp; Ewan Crawford, Codeplay</li> <li><a href="https://www.youtube.com/watch?v=BBC61L0QKCM">https://www.youtube.com/watch?v=BBC61L0QKCM</a></li> <li><a href="http://www.llvm.org/devmtg/2016-03/Presentations/EuroLLVM2016-E.Crawford_and_L.Drummond-Bringing_RenderScript_to_LLDB.pdf">http://www.llvm.org/devmtg/2016-03/Presentations/EuroLLVM2016-E.Crawford_and_L.Drummond-Bringing_RenderScript_to_LLDB.pdf</a></li> </ul> </li> <li>Debugging Tips and Tricks <ul> <li>WWDC 2016; Kate Stone, Enrico Granata, Sean Callanan, Jim Ingham</li> <li><a href="https://developer.apple.com/videos/play/wwdc2016/417">https://developer.apple.com/videos/play/wwdc2016/417</a></li> </ul> </li> <li>Debugging with LLDB <ul> <li>WWDC 2012; Greg Clayton</li> <li><a href="https://developer.apple.com/videos/play/wwdc2012/415/">https://developer.apple.com/videos/play/wwdc2012/415/</a></li> </ul> </li> <li>Debugging with LLVM: A quick introduction to LLDB and LLVM sanitizers <ul> <li>FOSDEM 2020; Andrzej Warzynski; Graham Hunter</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_with_llvm/">https://fosdem.org/2020/schedule/event/debugging_with_llvm/</a></li> </ul> </li> <li>ds2: a tiny debug server used with lldb <ul> <li>2015 LLVM Developers’ Meeting; Stephane Sezer, Facebook</li> <li><a href="https://www.youtube.com/watch?v=n00EhLskJWk">https://www.youtube.com/watch?v=n00EhLskJWk</a></li> <li><a href="https://github.com/facebook/ds2">https://github.com/facebook/ds2</a></li> </ul> </li> <li>LLDB - a new C++ debugger <ul> <li>DevConf.CZ 2019; Jan Kratochvíl</li> <li><a href="https://www.youtube.com/watch?v=2BjKNaiB3yM">https://www.youtube.com/watch?v=2BjKNaiB3yM</a></li> <li><a href="https://devconfcz2019.sched.com/event/Jcht">https://devconfcz2019.sched.com/event/Jcht</a></li> </ul> </li> <li>LLDB Reproducers <ul> <li>2019 EuroLLVM Developers’ Meeting; Jonas Devlieghere</li> <li><a href="https://www.youtube.com/watch?v=ygdzXnfyvbg">https://www.youtube.com/watch?v=ygdzXnfyvbg</a></li> <li><a href="http://llvm.org/devmtg/2019-04/talks.html#Talk_12">http://llvm.org/devmtg/2019-04/talks.html#Talk_12</a></li> </ul> </li> <li>LLDB Tutorial: Adding debugger support for your target <ul> <li>2016 EuroLLVM Developers' Meeting; Deepak Panickal &amp; Andrzej Warzynski, Codeplay</li> <li><a href="https://github.com/codeplaysoftware/lldb-msp430">https://github.com/codeplaysoftware/lldb-msp430</a></li> <li><a href="http://www.llvm.org/devmtg/2016-03/Tutorials/LLDB-tutorial.pdf">http://www.llvm.org/devmtg/2016-03/Tutorials/LLDB-tutorial.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=9hhDZeV0fYU">https://www.youtube.com/watch?v=9hhDZeV0fYU</a></li> </ul> </li> <li>LLDB: Beyond "po" <ul> <li>WWDC 2019; Davide Italiano, Jonas Devlieghere</li> <li><a href="https://developer.apple.com/videos/play/wwdc2019/429/">https://developer.apple.com/videos/play/wwdc2019/429/</a></li> <li><a href="https://www.raywenderlich.com/3868932-wwdc-2019-top-10-videos#toc-anchor-011">https://www.raywenderlich.com/3868932-wwdc-2019-top-10-videos#toc-anchor-011</a></li> </ul> </li> <li>Migrating from GDB to LLDB <ul> <li>WWDC 2011; Jim Ingham</li> <li><a href="https://developer.apple.com/videos/play/wwdc2011/321/">https://developer.apple.com/videos/play/wwdc2011/321/</a></li> </ul> </li> </ul> <h2>RR</h2> <ul> <li>RR: Record and Replay Framework <ul> <li><a href="http://rr-project.org/">http://rr-project.org/</a></li> <li><a href="https://github.com/mozilla/rr">https://github.com/mozilla/rr</a></li> </ul> </li> <li>rr Paper: "Lightweight User-Space Record And Replay" <ul> <li><a href="http://robert.ocallahan.org/2016/10/rr-paper-lightweight-user-space-record.html">http://robert.ocallahan.org/2016/10/rr-paper-lightweight-user-space-record.html</a></li> </ul> </li> <li>Control Flow Visualizer (CFViz): an rr / gdb plugin <ul> <li><a href="https://botondballo.wordpress.com/2017/12/22/control-flow-visualizer-cfviz-an-rr-gdb-plugin/">https://botondballo.wordpress.com/2017/12/22/control-flow-visualizer-cfviz-an-rr-gdb-plugin/</a></li> <li><a href="https://bitbucket.org/botond/cfviz">https://bitbucket.org/botond/cfviz</a></li> </ul> </li> <li>Engineering Record And Replay For Deployability <ul> <li>USENIX ATC 2017</li> <li>Robert O'Callahan, Chris Jones, Nathan Froyd, Kyle Huey, Albert Noll, Nimrod Partush</li> <li><a href="https://arxiv.org/abs/1705.05937">https://arxiv.org/abs/1705.05937</a></li> <li><a href="https://www.usenix.org/conference/atc17/technical-sessions/presentation/ocallahan">https://www.usenix.org/conference/atc17/technical-sessions/presentation/ocallahan</a></li> </ul> </li> <li>Improved debugging with rr - <a href="https://techtalk.intersec.com/2018/03/improved-debugging-with-rr/">https://techtalk.intersec.com/2018/03/improved-debugging-with-rr/</a></li> <li>rr-dataflow: An 'origin' command that continues to the origin of a piece of data in rr <ul> <li><a href="https://github.com/jrmuizel/rr-dataflow">https://github.com/jrmuizel/rr-dataflow</a></li> <li>Using rr-dataflow: Why and How? - <a href="https://www.mgaudet.ca/technical/2018/10/18/using-rr-dataflow-why-and-how">https://www.mgaudet.ca/technical/2018/10/18/using-rr-dataflow-why-and-how</a></li> </ul> </li> <li>Timeless Debugging of Complex Software: Root Cause Analysis of a Non-Deterministic JavaScriptCore Bug <ul> <li><a href="http://blog.ret2.io/2018/06/19/pwn2own-2018-root-cause-analysis/">http://blog.ret2.io/2018/06/19/pwn2own-2018-root-cause-analysis/</a></li> </ul> </li> </ul> <h2>OS-specific</h2> <h3>iOS</h3> <ul> <li>Chisel: a collection of LLDB commands to assist debugging iOS apps <ul> <li><a href="https://github.com/facebook/chisel">https://github.com/facebook/chisel</a></li> </ul> </li> <li>KTRW: An iOS kernel debugger based on a KTRR bypass for A11 iPhones that works with LLDB <ul> <li><a href="https://github.com/googleprojectzero/ktrw">https://github.com/googleprojectzero/ktrw</a></li> <li>KTRW: The journey to build a debuggable iPhone <ul> <li><a href="https://googleprojectzero.blogspot.com/2019/10/ktrw-journey-to-build-debuggable-iphone.html">https://googleprojectzero.blogspot.com/2019/10/ktrw-journey-to-build-debuggable-iphone.html</a></li> </ul> </li> </ul> </li> </ul> <h3>Linux</h3> <ul> <li>crash-python: a semantic debugger for the Linux kernel <ul> <li><a href="https://github.com/jeffmahoney/crash-python">https://github.com/jeffmahoney/crash-python</a></li> </ul> </li> <li>debuginfod <ul> <li>elfutils debuginfod is a client/server in elfutils 0.178+ that automatically distributes elf/dwarf/source-code from servers to clients such as debuggers across HTTP <ul> <li><a href="https://sourceware.org/elfutils/Debuginfod.html">https://sourceware.org/elfutils/Debuginfod.html</a></li> </ul> </li> <li>Deploying debuginfod servers for your developers <ul> <li><a href="https://developers.redhat.com/blog/2019/12/17/deploying-debuginfod-servers-for-your-developers/">https://developers.redhat.com/blog/2019/12/17/deploying-debuginfod-servers-for-your-developers/</a></li> </ul> </li> <li>The elfutils debuginfod server <ul> <li>FOSDEM 2020; Mark Wielaard, Frank Ch. Eigler</li> <li><a href="https://fosdem.org/2020/schedule/event/debugging_debuginfod/">https://fosdem.org/2020/schedule/event/debugging_debuginfod/</a></li> </ul> </li> <li>Introducing debuginfod, the elfutils debuginfo server <ul> <li><a href="https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/">https://developers.redhat.com/blog/2019/10/14/introducing-debuginfod-the-elfutils-debuginfo-server/</a></li> </ul> </li> <li>elfutils debuginfo-server <ul> <li>GNU Tools Cauldron 2019; Frank Ch. Eigler, Aaron Merey</li> <li><a href="https://www.youtube.com/watch?v=cyOXWT_EBJ0">https://www.youtube.com/watch?v=cyOXWT_EBJ0</a></li> <li><a href="https://gcc.gnu.org/wiki/cauldron2019talks?action=AttachFile&amp;do=view&amp;target=dbgserver.pdf">https://gcc.gnu.org/wiki/cauldron2019talks?action=AttachFile&amp;do=view&amp;target=dbgserver.pdf</a></li> </ul> </li> </ul> </li> <li>drgn: Scriptable debugger library <ul> <li><a href="https://github.com/osandov/drgn/">https://github.com/osandov/drgn/</a></li> <li><a href="https://drgn.readthedocs.io/">https://drgn.readthedocs.io/</a></li> <li>A kernel debugger in Python: drgn <ul> <li><a href="https://lwn.net/Articles/789641/">https://lwn.net/Articles/789641/</a></li> </ul> </li> </ul> </li> <li>edb: a cross platform x86/x86-64 debugger <ul> <li><a href="https://github.com/eteran/edb-debugger">https://github.com/eteran/edb-debugger</a></li> </ul> </li> <li>ich: Linux crash harness with runtime process instrumentation <ul> <li><a href="https://github.com/zznop/ich">https://github.com/zznop/ich</a></li> </ul> </li> <li>libkdumpfile: Kernel coredump file access <ul> <li><a href="https://github.com/ptesarik/libkdumpfile">https://github.com/ptesarik/libkdumpfile</a></li> </ul> </li> <li>libthread_db <ul> <li>Notes about an odd, esoteric, yet incredibly useful library: libthread_db <ul> <li><a href="http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/">http://timetobleed.com/notes-about-an-odd-esoteric-yet-incredibly-useful-library-libthread_db/</a></li> </ul> </li> </ul> </li> <li>ORC (Oops Rewind Capability) Unwinder <ul> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ee9f8fce99640811b2b8e79d0d1dbe8bab69ba67">Commit ee9f8fce9964</a></li> <li><a href="https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=39358a033b2e4432052265c1fa0f36f572d8cfb5">Commit 39358a033b2e</a></li> <li>orc-unwinder.txt - <a href="https://github.com/torvalds/linux/blob/master/Documentation/x86/orc-unwinder.txt">https://github.com/torvalds/linux/blob/master/Documentation/x86/orc-unwinder.txt</a></li> <li>The Linux x86 ORC Stack Unwinder - <a href="http://www.codeblueprint.co.uk/2017/07/31/the-orc-unwinder.html">http://www.codeblueprint.co.uk/2017/07/31/the-orc-unwinder.html</a></li> <li>The ORCs are coming - <a href="https://lwn.net/Articles/728339/">https://lwn.net/Articles/728339/</a></li> <li>x86: ORC unwinder (previously undwarf) - <a href="https://lwn.net/Articles/727553/">https://lwn.net/Articles/727553/</a></li> </ul> </li> <li>plutonium-dbg: A kernel-based debugger for Linux applications <ul> <li><a href="https://github.com/plutonium-dbg/plutonium-dbg">https://github.com/plutonium-dbg/plutonium-dbg</a></li> <li>Kernel-Assisted Debugging of Linux Applications <ul> <li>Reversing and Offensive-oriented Trends Symposium (ROOTS) 2018</li> <li>Tobias Holl, Philipp Klocke, Fabian Franzen, and Julian Kirsch</li> <li><a href="https://vimeo.com/307238462">https://vimeo.com/307238462</a></li> <li><a href="https://dl.acm.org/citation.cfm?id=3289596">https://dl.acm.org/citation.cfm?id=3289596</a></li> </ul> </li> </ul> </li> <li>Scout - Instruction based research debugger <ul> <li>an extendable basic debugger designed for use in those cases that there is no built-in debugger / gdb-stub in the debugee process / firmware</li> <li><a href="https://github.com/CheckPointSW/Scout">https://github.com/CheckPointSW/Scout</a></li> <li><a href="https://scout-debugger.readthedocs.io/">https://scout-debugger.readthedocs.io/</a></li> </ul> </li> <li>sdb: The Slick/Simple Debugger <ul> <li>A postmortem and live debugger</li> <li><a href="https://github.com/delphix/sdb">https://github.com/delphix/sdb</a></li> <li>Debugging ZFS: From Illumos to Linux <ul> <li>OpenZFS Developer Summit 2019; Serapheim Dimitropoulos</li> <li><a href="https://drive.google.com/file/d/1oho9X5bkW-I-yJ-pVD8VqkaloxhGepzT/view">https://drive.google.com/file/d/1oho9X5bkW-I-yJ-pVD8VqkaloxhGepzT/view</a></li> </ul> </li> </ul> </li> <li>vmlinux-to-elf: A tool to recover a fully analyzable ELF from a raw kernel, through extracting the kernel symbol table (kallsyms) <ul> <li><a href="https://github.com/marin-m/vmlinux-to-elf">https://github.com/marin-m/vmlinux-to-elf</a></li> </ul> </li> </ul> <h4>Linux - Talks</h4> <ul> <li>Linux Kernel Debugging: Going Beyond Printk Messages <ul> <li>Embedded Linux Conference Europe (ELCE) 2019; Sergio Prado</li> <li><a href="https://www.youtube.com/watch?v=NDXYpR_m1CU">https://www.youtube.com/watch?v=NDXYpR_m1CU</a></li> <li><a href="https://elinux.org/images/1/14/Linuxkerneldebugging.pdf">https://elinux.org/images/1/14/Linuxkerneldebugging.pdf</a></li> <li><a href="https://e-labworks.com/talks/elce2019">https://e-labworks.com/talks/elce2019</a></li> </ul> </li> </ul> <h3>macOS</h3> <ul> <li>LLDBagility <ul> <li><a href="https://github.com/quarkslab/LLDBagility/">https://github.com/quarkslab/LLDBagility/</a></li> <li>a tool for debugging macOS virtual machines with the aid of the Fast Debugging Protocol (FDP)</li> <li><a href="https://blog.quarkslab.com/an-overview-of-macos-kernel-debugging.html">https://blog.quarkslab.com/an-overview-of-macos-kernel-debugging.html</a></li> <li><a href="https://blog.quarkslab.com/lldbagility-practical-macos-kernel-debugging.html">https://blog.quarkslab.com/lldbagility-practical-macos-kernel-debugging.html</a></li> </ul> </li> <li>Mac OS X Debugging Magic - <a href="https://developer.apple.com/library/content/technotes/tn2124/_index.html">https://developer.apple.com/library/content/technotes/tn2124/_index.html</a></li> </ul> <h3>Windows</h3> <ul> <li>An RPC Debugging Framework with Visual Studio <ul> <li><a href="http://donw.io/post/rpc-debugging-vs/">http://donw.io/post/rpc-debugging-vs/</a></li> </ul> </li> <li>ArkDasm: 64-bit interactive disassembler and debugger for Windows <ul> <li><a href="http://www.arkdasm.com/">http://www.arkdasm.com/</a></li> </ul> </li> <li>Creating a Public Symbol Server, Easily <ul> <li><a href="https://randomascii.wordpress.com/2020/03/14/creating-a-public-symbol-server-easily/">https://randomascii.wordpress.com/2020/03/14/creating-a-public-symbol-server-easily/</a></li> </ul> </li> <li>DbgShell: A PowerShell front-end for the Windows debugger engine <ul> <li><a href="https://github.com/Microsoft/DbgShell/">https://github.com/Microsoft/DbgShell/</a></li> </ul> </li> <li>Debug Recipes (.NET and Windows problems) - <a href="https://github.com/lowleveldesign/debug-recipes">https://github.com/lowleveldesign/debug-recipes</a></li> <li>Debugging in Visual Studio - <a href="https://docs.microsoft.com/en-us/visualstudio/debugger/debugger-feature-tour">https://docs.microsoft.com/en-us/visualstudio/debugger/debugger-feature-tour</a></li> <li>Debugging Tools for Windows (WinDbg, KD, CDB, NTSD) <ul> <li><a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/">https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/</a></li> </ul> </li> <li>DuBStep (Dynamic Breakpoint API): A Library for creating hardware execution and data breakpoints at runtime on Win32/Win64 <ul> <li><a href="https://github.com/justinboswell/dubstep">https://github.com/justinboswell/dubstep</a></li> </ul> </li> <li>kdmp-parser: Windows kernel dump C++ parser <ul> <li><a href="https://github.com/0vercl0k/kdmp-parser">https://github.com/0vercl0k/kdmp-parser</a></li> </ul> </li> <li>makin - reveal anti-debug tricks (Windows) - <a href="https://github.com/secrary/makin">https://github.com/secrary/makin</a></li> <li>Nanomite - Windows Debugger for x64 and x86 - <a href="https://github.com/zer0fl4g/Nanomite/">https://github.com/zer0fl4g/Nanomite/</a></li> <li>OllyDbg - <a href="http://www.ollydbg.de/">http://www.ollydbg.de/</a></li> <li>ScyllaHide: Advanced usermode anti-anti-debugger <ul> <li><a href="https://github.com/x64dbg/ScyllaHide">https://github.com/x64dbg/ScyllaHide</a></li> </ul> </li> <li>TitanHide: Hiding kernel-driver for x86/x64 <ul> <li>a driver intended to hide debuggers from certain processes</li> <li><a href="https://github.com/mrexodia/TitanHide">https://github.com/mrexodia/TitanHide</a></li> </ul> </li> <li>Windows Debugging Tools - Debugging Resources <ul> <li><a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-resources">https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/debugging-resources</a></li> </ul> </li> <li>x64dbg <ul> <li><a href="https://x64dbg.com/">https://x64dbg.com/</a></li> <li><a href="https://github.com/x64dbg/x64dbg">https://github.com/x64dbg/x64dbg</a></li> </ul> </li> </ul> <h4>Visual Studio Debugger</h4> <ul> <li><a href="https://docs.microsoft.com/en-us/visualstudio/debugger/">https://docs.microsoft.com/en-us/visualstudio/debugger/</a></li> <li><a href="https://devblogs.microsoft.com/visualstudio/tag/debug/">https://devblogs.microsoft.com/visualstudio/tag/debug/</a></li> </ul> <h4>WinDbg</h4> <ul> <li>Introduction to WinDbg and debugging Windows - series by Anand George <ul> <li><a href="https://www.youtube.com/playlist?list=PLhx7-txsG6t6n_E2LgDGqgvJtCHPL7UFu">https://www.youtube.com/playlist?list=PLhx7-txsG6t6n_E2LgDGqgvJtCHPL7UFu</a></li> </ul> </li> <li>Modern Debugging with WinDbg Preview <ul> <li>DEFCON 27 (2019) Workshop Materials</li> <li><a href="https://github.com/hugsy/defcon_27_windbg_workshop/">https://github.com/hugsy/defcon_27_windbg_workshop/</a></li> <li>WinDbg cheatsheet <ul> <li><a href="https://github.com/hugsy/defcon_27_windbg_workshop/blob/master/windbg_cheatsheet.md">https://github.com/hugsy/defcon_27_windbg_workshop/blob/master/windbg_cheatsheet.md</a></li> </ul> </li> </ul> </li> <li>Time Travel Debugging - James McNellis <ul> <li>Meeting C++ 2018 - <a href="https://www.youtube.com/watch?v=MyVQnP-U_ho">https://www.youtube.com/watch?v=MyVQnP-U_ho</a></li> <li>Pacific++ 2018 - <a href="https://www.youtube.com/watch?v=BVslyei0804">https://www.youtube.com/watch?v=BVslyei0804</a></li> </ul> </li> <li>WinDbg Basics for Malware Analysis <ul> <li><a href="https://www.youtube.com/watch?v=QuFJpH3My7A">https://www.youtube.com/watch?v=QuFJpH3My7A</a></li> </ul> </li> </ul> <h5>Readings</h5> <ul> <li>Collection of WinDBG resources - <a href="https://blogs.msdn.microsoft.com/reiley/2012/07/28/collection-of-windbg-resources/">https://blogs.msdn.microsoft.com/reiley/2012/07/28/collection-of-windbg-resources/</a></li> <li>Break On Process Creation in WinDbg <ul> <li><a href="https://shakreiner.com/posts/break-on-process-windbg/">https://shakreiner.com/posts/break-on-process-windbg/</a></li> </ul> </li> <li>Debugger Data Model, Javascript &amp; X64 Exception Handling - <a href="https://doar-e.github.io/blog/2017/12/01/debugger-data-model/">https://doar-e.github.io/blog/2017/12/01/debugger-data-model/</a></li> <li>Debugging Tools for Windows - <a href="https://blogs.msdn.microsoft.com/windbg/">https://blogs.msdn.microsoft.com/windbg/</a></li> <li>Exploiting flaws in Windbg: how to escape or fool debuggers from existing flaws <ul> <li>Journal of Computer Virology and Hacking Techniques (2020)</li> <li>François Plumerault, Baptiste David</li> <li><a href="http://dx.doi.org/10.1007/s11416-020-00347-x">http://dx.doi.org/10.1007/s11416-020-00347-x</a></li> </ul> </li> <li>JavaScript bridge makes malware analysis with WinDbg easier <ul> <li><a href="https://blog.talosintelligence.com/2019/02/windbg-malware-analysis-with-javascript.html">https://blog.talosintelligence.com/2019/02/windbg-malware-analysis-with-javascript.html</a></li> </ul> </li> <li>Stupid debugger tricks: Calling functions and methods - <a href="https://blogs.msdn.microsoft.com/oldnewthing/20070427-00/?p=27083">https://blogs.msdn.microsoft.com/oldnewthing/20070427-00/?p=27083</a></li> <li>Time travel debugging: It’s a blast! (from the past) <ul> <li><a href="https://blogs.technet.microsoft.com/srd/2019/05/29/time-travel-debugging-its-a-blast-from-the-past/">https://blogs.technet.microsoft.com/srd/2019/05/29/time-travel-debugging-its-a-blast-from-the-past/</a></li> </ul> </li> <li>Tutorial: Using WinDBG to call arbitrary functions — WinDBG kung-fu series <ul> <li><a href="http://cfc.kizzx2.com/index.php/tutorial-using-windbg-to-call-arbitrary-functions-windbg-kung-fu-series/">http://cfc.kizzx2.com/index.php/tutorial-using-windbg-to-call-arbitrary-functions-windbg-kung-fu-series/</a></li> </ul> </li> <li>Undocumented WinDBG - <a href="https://blogs.msdn.microsoft.com/reiley/2011/10/30/undocumented-windbg/">https://blogs.msdn.microsoft.com/reiley/2011/10/30/undocumented-windbg/</a></li> <li>Use Windows Debuggers for Non-Debugging Tasks - <a href="https://blogs.msdn.microsoft.com/reiley/2011/10/23/use-windows-debuggers-for-non-debugging-tasks/">https://blogs.msdn.microsoft.com/reiley/2011/10/23/use-windows-debuggers-for-non-debugging-tasks/</a></li> <li>Using Function Evaluation in WinDBG - <a href="https://blogs.msdn.microsoft.com/reiley/2012/08/18/using-function-evaluation-in-windbg/">https://blogs.msdn.microsoft.com/reiley/2012/08/18/using-function-evaluation-in-windbg/</a></li> <li>Windbg automation and extensions - <a href="https://nativecoding.wordpress.com/2016/01/10/automate-attach-to-process-on-windows-with-windbg/">https://nativecoding.wordpress.com/2016/01/10/automate-attach-to-process-on-windows-with-windbg/</a></li> <li>WinDbg Malware Analysis Cheat Sheet <ul> <li><a href="https://oalabs.openanalysis.net/2019/02/18/windbg-for-malware-analysis/">https://oalabs.openanalysis.net/2019/02/18/windbg-for-malware-analysis/</a></li> </ul> </li> <li>WinDbg, Debugger Objects, and JavaScript! Oh, My! - <a href="https://www.osr.com/blog/2017/05/18/windbg-debugger-objects-javascript-oh/">https://www.osr.com/blog/2017/05/18/windbg-debugger-objects-javascript-oh/</a></li> <li>Yet Another Hello World - <a href="https://blogs.msdn.microsoft.com/reiley/2011/09/29/yet-another-hello-world/">https://blogs.msdn.microsoft.com/reiley/2011/09/29/yet-another-hello-world/</a></li> </ul> <h5>Projects</h5> <ul> <li>0CCh Windbg extension - <a href="https://github.com/0cch/0cchext">https://github.com/0cch/0cchext</a></li> <li>DbgModelCppLib: A header-only C++ library for producing and consuming data from the debugger data model <ul> <li><a href="https://github.com/Microsoft/WinDbg-Libraries/tree/master/DbgModelCppLib">https://github.com/Microsoft/WinDbg-Libraries/tree/master/DbgModelCppLib</a></li> <li><a href="https://github.com/Microsoft/WinDbg-Samples/tree/master/DataModelHelloWorld/Cpp">https://github.com/Microsoft/WinDbg-Samples/tree/master/DataModelHelloWorld/Cpp</a></li> </ul> </li> <li>PyKd - Python extension for WinDBG to access Debug Engine <ul> <li><a href="https://githomelab.ru/pykd/pykd">https://githomelab.ru/pykd/pykd</a></li> <li>windbg-pack: Set of python scripts for WinDBG - <a href="https://githomelab.ru/pykd/windbg-pack">https://githomelab.ru/pykd/windbg-pack</a></li> </ul> </li> <li>TWindbg: PEDA-like debugger UI for WinDbg - <a href="https://github.com/bruce30262/TWindbg">https://github.com/bruce30262/TWindbg</a></li> <li>WDBGARK: WinDBG Anti-RootKit extension - <a href="https://github.com/swwwolf/wdbgark">https://github.com/swwwolf/wdbgark</a></li> <li>Winbagility: a tool to connect WinDbg on non /DEBUG Windows x64 systems <ul> <li><a href="http://winbagility.github.io">http://winbagility.github.io</a></li> <li><a href="https://github.com/Winbagility/Winbagility">https://github.com/Winbagility/Winbagility</a></li> </ul> </li> <li>WinDbg-Samples: Sample extensions, scripts, and API uses for WinDbg <ul> <li><a href="https://github.com/Microsoft/WinDbg-Samples">https://github.com/Microsoft/WinDbg-Samples</a></li> </ul> </li> <li>WinDBGtree: A command tree based on commands and extensions for Windows Kernel Debugging <ul> <li><a href="https://github.com/vagnerpilar/windbgtree">https://github.com/vagnerpilar/windbgtree</a></li> </ul> </li> </ul> <h5>Time Travel Debugging</h5> <ul> <li>Time Travel Debugging FAQ - <a href="https://blogs.msdn.microsoft.com/windbg/2017/10/20/time-travel-debugging-faq/">https://blogs.msdn.microsoft.com/windbg/2017/10/20/time-travel-debugging-faq/</a></li> <li>Time Travel Debugging - Overview - <a href="https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview">https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/time-travel-debugging-overview</a></li> <li>Time Travel Debugging - Sample App Walkthrough - <a href="https://aka.ms/ttdtutorial">https://aka.ms/ttdtutorial</a></li> <li>CppCon 2017: Time Travel Debugging: Root Causing Bugs in Commercial Scale Software <ul> <li><a href="https://www.youtube.com/watch?v=l1YJTg_A914">https://www.youtube.com/watch?v=l1YJTg_A914</a></li> </ul> </li> <li>Defrag Tools #186 - Time Travel Debugging - Advanced <ul> <li><a href="https://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-186-Time-Travel-Debugging-Advanced">https://channel9.msdn.com/Shows/Defrag-Tools/Defrag-Tools-186-Time-Travel-Debugging-Advanced</a></li> </ul> </li> </ul> <h2>Stack Trace &amp; Unwinding</h2> <ul> <li>BacktraceResolver: Very fast backtraces resolver <ul> <li><a href="https://github.com/markpapadakis/BacktraceResolver">https://github.com/markpapadakis/BacktraceResolver</a></li> </ul> </li> <li>Backward-cpp: a beautiful stack trace pretty printer for C++ <ul> <li><a href="https://github.com/bombela/backward-cpp">https://github.com/bombela/backward-cpp</a></li> </ul> </li> <li>Boost.Stacktrace <ul> <li><a href="http://boostorg.github.io/stacktrace/">http://boostorg.github.io/stacktrace/</a></li> <li><a href="https://github.com/boostorg/stacktrace">https://github.com/boostorg/stacktrace</a></li> <li><a href="http://www.boost.org/doc/libs/release/doc/html/stacktrace.html">http://www.boost.org/doc/libs/release/doc/html/stacktrace.html</a></li> </ul> </li> <li>CppCon 2017: Dave Watson “C++ Exceptions and Stack Unwinding” <ul> <li><a href="https://www.youtube.com/watch?v=_Ivd3qzgT7U">https://www.youtube.com/watch?v=_Ivd3qzgT7U</a></li> </ul> </li> <li>libgcc_s (GNU) - <a href="https://gcc.gnu.org/onlinedocs/gccint/Libgcc.html">https://gcc.gnu.org/onlinedocs/gccint/Libgcc.html</a> <ul> <li>Data Definitions for libgcc_s - <a href="https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-s-ddefs.html">https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-s-ddefs.html</a></li> <li>Interfaces for libgcc_s - <a href="https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-s.html">https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-s.html</a></li> <li>Interfaces Definitions for libgcc_s - <a href="https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-sman.html">https://refspecs.linuxfoundation.org/LSB_5.0.0/LSB-Core-generic/LSB-Core-generic/libgcc-sman.html</a></li> </ul> </li> <li>libunwind (nongnu.org) - <a href="http://www.nongnu.org/libunwind/">http://www.nongnu.org/libunwind/</a></li> <li>libunwind (PathScale) - <a href="https://github.com/pathscale/libunwind">https://github.com/pathscale/libunwind</a></li> <li>Programmatic access to the call stack in C++ - <a href="https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/">https://eli.thegreenplace.net/2015/programmatic-access-to-the-call-stack-in-c/</a></li> <li>pstack: Print stack traces of running processes. Uses its own ELF and DWARF parsing <ul> <li><a href="https://github.com/peadar/pstack">https://github.com/peadar/pstack</a></li> </ul> </li> <li>quickstack: a tool to take call stack traces with minimal overheads <ul> <li><a href="https://github.com/yoshinorim/quickstack">https://github.com/yoshinorim/quickstack</a></li> </ul> </li> <li>Unwind specification draft for GNU/Linux/ia64 (extends the exception ABI) - <a href="https://www.kernel.org/pub/linux/devel/gcc/unwind/">https://www.kernel.org/pub/linux/devel/gcc/unwind/</a></li> </ul> <hr> <h1>Talks</h1> <h2>2019</h2> <ul> <li>Modern Linux C++ Debugging Tools: Under the Covers <ul> <li>CppCon 2019; Greg Law</li> <li><a href="https://www.youtube.com/watch?v=WoRmXjVxuFQ">https://www.youtube.com/watch?v=WoRmXjVxuFQ</a></li> <li><a href="https://github.com/CppCon/CppCon2019/tree/master/Presentations/modern_linux_cpp_debugging_tools__under_the_covers">https://github.com/CppCon/CppCon2019/tree/master/Presentations/modern_linux_cpp_debugging_tools__under_the_covers</a></li> </ul> </li> </ul> <h2>2018</h2> <ul> <li>GNU Tools Cauldron 2018 <ul> <li>gOlogy: impact of -O* on -g <ul> <li>Alexandre Oliva</li> <li><a href="http://people.redhat.com/aoliva/writeups/gOlogy/slides.pdf">http://people.redhat.com/aoliva/writeups/gOlogy/slides.pdf</a></li> <li><a href="http://www.fsfla.org/~lxoliva/writeups/gOlogy/gOlogy.txt">http://www.fsfla.org/~lxoliva/writeups/gOlogy/gOlogy.txt</a></li> </ul> </li> <li>A collection of debug info improvements for the GNU Compiler Collection <ul> <li>Alexandre Oliva, Mark J. Wielaard</li> <li><a href="https://gnu.wildebeest.org/~mark/cauldron-2018/DebuginfoImprovements.pdf">https://gnu.wildebeest.org/~mark/cauldron-2018/DebuginfoImprovements.pdf</a></li> </ul> </li> </ul> </li> <li>Let's Write a Debugger! <ul> <li><a href="https://rego.linux.conf.au/schedule/presentation/91/">linux.conf.au 2018</a> - <a href="https://www.doc.ic.ac.uk/~lk1015/publications.html">Levente Kurusa</a></li> <li><a href="https://www.doc.ic.ac.uk/~lk1015/research/lca2018/talk_lca2018.pdf">https://www.doc.ic.ac.uk/~lk1015/research/lca2018/talk_lca2018.pdf</a></li> <li><a href="https://opensource.com/article/18/1/how-debuggers-really-work">https://opensource.com/article/18/1/how-debuggers-really-work</a></li> <li><a href="https://www.youtube.com/watch?v=qS51kIHWARM">https://www.youtube.com/watch?v=qS51kIHWARM</a></li> <li><a href="https://github.com/levex/debugger-talk">https://github.com/levex/debugger-talk</a></li> </ul> </li> </ul> <h2>2017</h2> <ul> <li>Debugging the debugger - BSDCan 2017, Samy Bahra <ul> <li><a href="https://www.youtube.com/watch?v=wyF-hGGPJMs">https://www.youtube.com/watch?v=wyF-hGGPJMs</a></li> <li>Compiler debug quality suite - <a href="https://github.com/backtrace-labs/cdqs">https://github.com/backtrace-labs/cdqs</a></li> <li><a href="http://www.bsdcan.org/2017/schedule/events/787.en.html">http://www.bsdcan.org/2017/schedule/events/787.en.html</a></li> </ul> </li> <li>Debugging Under Fire: Keep your Head when Systems have Lost their Mind • GOTO 2017 • Bryan Cantrill <ul> <li><a href="https://www.youtube.com/watch?v=30jNsCVLpAE">https://www.youtube.com/watch?v=30jNsCVLpAE</a></li> </ul> </li> <li>How C++ Debuggers work - Simon Brand <ul> <li>C++ Edinburgh 2018 <ul> <li><a href="https://www.youtube.com/watch?v=UiW24hzLy1M">https://www.youtube.com/watch?v=UiW24hzLy1M</a></li> </ul> </li> <li>Meeting C++ 2017 <ul> <li><a href="https://meetingcpp.com/2017/talks/items/How_Cpp_Debuggers_Work.html">https://meetingcpp.com/2017/talks/items/How_Cpp_Debuggers_Work.html</a></li> <li><a href="https://www.youtube.com/watch?v=Q3Rm95Mk03c">https://www.youtube.com/watch?v=Q3Rm95Mk03c</a></li> </ul> </li> </ul> </li> </ul> <h2>2016</h2> <ul> <li>Building a Debugging Mindset - QConSF 2016 - Devon H. O'Dell <ul> <li>video: <a href="https://www.infoq.com/presentations/debugging-mindset">https://www.infoq.com/presentations/debugging-mindset</a></li> <li>slides: <a href="https://speakerdeck.com/dho/building-a-debugging-mindset">https://speakerdeck.com/dho/building-a-debugging-mindset</a></li> <li>transcript: <a href="https://9vx.org/post/building-a-debugging-mindset/">https://9vx.org/post/building-a-debugging-mindset/</a></li> </ul> </li> <li>How do Debuggers (Really) Work - Pawel Moll <ul> <li>Linux Piter 2016 - <a href="https://www.youtube.com/watch?v=xqrxg3hl10o">https://www.youtube.com/watch?v=xqrxg3hl10o</a></li> <li>ELC 2015 <ul> <li><a href="https://www.youtube.com/watch?v=N1QSEY1El9o">https://www.youtube.com/watch?v=N1QSEY1El9o</a></li> <li><a href="https://vimeo.com/220644951">https://vimeo.com/220644951</a></li> <li><a href="http://events.linuxfoundation.org/sites/events/files/slides/slides_16.pdf">http://events.linuxfoundation.org/sites/events/files/slides/slides_16.pdf</a></li> </ul> </li> </ul> </li> <li>Post-mortem Debugging: could you be the one? - Surge 2016 - Abel Mathew <ul> <li><a href="https://www.youtube.com/watch?v=WHhorNLa934">https://www.youtube.com/watch?v=WHhorNLa934</a></li> </ul> </li> <li>Timeless Debugging - USENIX Enigma 2016 - George Hotz <ul> <li><a href="https://www.youtube.com/watch?v=eGl6kpSajag">https://www.youtube.com/watch?v=eGl6kpSajag</a></li> </ul> </li> </ul> <h2>2015</h2> <ul> <li>Debugging using an exact recording of a program's execution - C++Now 2015 - Julian Smith <ul> <li>Undo's Live Recorder</li> <li><a href="https://www.youtube.com/watch?v=NWWMogzgLS4">https://www.youtube.com/watch?v=NWWMogzgLS4</a></li> </ul> </li> </ul> <h2>2014</h2> <ul> <li>The VS Debugger: How It Works + Tips and Tricks - GoingNative 28 - Gabriel Ha, Gregg Miskelly, Steve Carroll <ul> <li><a href="https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-28-The-VS-Debugger-How-It-Works-Tips-and-Tricks">https://channel9.msdn.com/Shows/C9-GoingNative/GoingNative-28-The-VS-Debugger-How-It-Works-Tips-and-Tricks</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/crisp/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/crisp/index.html <p><a href="https://bitfieldconsulting.com/golang/crisp-code">Article</a>: Instead of <a href="../clean-code">CLEAN</a> code, write code that is...</p> <h4>Correct</h4> <p>Your code may have lots of other wonderful things to be said for it, but if it’s not correct, they don’t really matter. And, indeed, if we’re willing to give up correctness, we can presumably obtain any other attribute fairly easily. It might seem obvious and not worth saying, but if incorrect code exists, and I think it does, then surely it is worth saying, because apparently someone didn’t get the memo.</p> <p>What does it mean for code to be correct, actually? The straightforward answer is “it does what the writer intended”, but even that’s not quite right, because I might have intended the wrong thing! For example, I might write a prime number generator while labouring under the erroneous impression that all odd numbers are prime. In this case, my code might produce the odd integers, as I requested, but still not be correct.</p> <p>A good suite of tests, again like apple pie, is something just about everyone would love to have, even if they don’t always feel like putting in the necessary labour. And tests are a necessary, though not sufficient, part of any good program. No doubt there are some correct programs that don’t have tests, but who knows which ones they are? A well-written test expresses what the programmer thinks the code should do under a given set of circumstances, and running it should give us a certain level of confidence about whether it actually does or not. Tests themselves can be incorrect, though.</p> <p>This is one reason why we need to be very suspicious and sceptical of any given test. What does it look like it says? Does it actually say that? Is it the right thing to say? If it verifies the program’s result against some expected result, is the expectation correct? If the test claims to cover some particular section of code, does it actually test the behaviour of that code in all important ways, or does it merely cause it to be executed?</p> <p>Not having tests is a terrible situation to be in, but it’s actually slightly preferable to having tests that don’t work. A malfunctioning or insufficient test may give us false confidence in code that also happens to be incorrect. Therefore, once you’ve written your tests, read them all again extremely carefully, with an adversarial eye. Programmers are incurable optimists: we always think our code will work, despite much evidence to the contrary. Instead, we should assume that if there can be a bug, then there is a bug. Humility isn’t a virtue often associated with software engineers, but it’s certainly something that good test writers know all about.</p> <p>Even the most thoroughly tested code should not be assumed to be correct. It should be assumed to be incorrect. It almost certainly is.</p> <p><em>(Sounds more like "Correct" should be "Testable" or "Verifiable", if you ask me.)</em></p> <h4>Readable</h4> <p>Again, this might sound like something that doesn’t need saying: who’s arguing for unreadable code? Not me, but there seems to be a lot of it around, all the same. It’s not that anyone deliberately sets out to write unreadable code, of course; it just ends up that way because we mistakenly prioritise other virtues over readability.</p> <p>Performance is one such virtue, and there are cases where performance genuinely matters. Not as many as you might think, though: computers are pretty fast these days, and since most of what they do is in the service of us humans, they can usually afford to take their time about it.</p> <p>So, while readability isn’t quite as important as correctness, it’s more important than anything else. Readability, as Churchill said of courage, is rightly esteemed the first of qualities, because it is the quality which guarantees all others.</p> <p>What is it that makes code readable, or not? I don’t think readability is something that you can add to your code: I think it’s what you’re left with when you’ve taken away all the things that make it hard to understand.</p> <p>For example, a poorly-chosen variable name can put a bump in the reader’s path. Using different names for the same thing, or the same name for different things, is confusing. An unnecessary function call, added purely to satisfy some rule like “methods should be less than 10 lines”, breaks the reader’s flow.</p> <p>It’s like reading an article and coming across an unfamiliar word: should you pause to look it up, and risk losing the train of thought, or struggle on and try to infer the meaning from context?</p> <p>The best way to make any code more readable is, oddly enough, to start by reading it. But I mean something special by that. I don’t mean scrolling hurriedly through pages and pages of code, glancing and skimming to get a sense of what’s happening. That is a useful skill, but for other tasks.</p> <p>Instead, we need to read code in a careful, sequential, intentional way. We need to begin at the beginning, and go on till we come to the end. If it’s not clear where the program begins, then that’s the first thing we need to fix.</p> <p>As we follow the thread of execution of the program, we need to read each line carefully and then try to answer two questions about it:</p> <ol> <li>What does it say?</li> <li>What does it mean?</li> </ol> <p>For example, consider the following line:</p> <pre><code>requests++ </code></pre> <p>What it says is clear: it increases the value of some numeric variable requests by one. What’s not so easy to work out is what that means. What is being counted in the requests variable? Why is it being incremented? What significance does that have? Where did requests get its current value? What is its current value? When and where will that value be checked? Is there some maximum value? What happens when we reach it? And so on.</p> <p>There may be perfectly good answers to all these questions, but that’s not the point. The point is, can the reader answer them just by looking at the code? If not, what can we do to provide the answers, or make them easier to find? By reading our own code as though we were new to it, we see it with fresh eyes and discover the parts that need more cognitive effort to follow.</p> <p><em>(Goes back to the fundamental question: What is "good code"?)</em></p> <h4>Idiomatic</h4> <p>I don’t think this is quite the right word: I’d prefer “conventional”, but again that doesn’t fit with the CRISP backronym. Still, when people say “such and such is idiomatic”, what they really mean is “this is the conventional way to do it.”</p> <p>Conventions are useful: there are lots of possible ways to lay out the controls for a car, for example, and the one we’re used to isn’t demonstrably optimal. We’re just used to it. There’s no law stopping a car manufacturer from putting the pedals in a different arrangement, but they don’t. Even if there were some demonstrable benefit, it wouldn’t be worth the cognitive cost to users: getting confused between the accelerator and brake pedal would be unfortunate.</p> <p>Similarly, I think there’s great value in using conventional names for things: in an HTTP handler, the request pointer is always called r and the response writer w. If there’s a universal convention, it’s worth following. You’ll also have local and personal conventions. In my code, an arbitrary bytes.Buffer is always named buf, the compared values in my tests are always named want and got, and so on.</p> <p>An obvious example of a universal convention is err: Go programmers always use this name to refer to an arbitrary error value. While we wouldn’t usually re-use variable names within the same function, we do re-use err for all the various errors that there can be throughout the function. It’s a mistake to try to avoid this by creating variant names like err2, err3, and so on.</p> <p>Why? Because it requires a tiny bit more cognitive effort from the reader. If you see err, your mind glides right over it with perfect understanding. If you see some other name, your mind has to stop to resolve the puzzle.</p> <p>I call these tiny roadblocks “cognitive microaggressions”. Each is so tiny that its individual effect is almost unnoticeable, yet they soon pile up, and if you encounter them all day and every day, they have a significant effect on your quality of life.</p> <p>As you write each line of code, you should be thinking “How much effort does it take to understand this? Could I reduce that somehow?” The secret of great software engineering is doing a lot of little things well. Picking the right names, organising code in a logical fashion, having one idea per line of code: all these go towards cumulatively making your code readable and pleasant to work with.</p> <p>How do you learn what’s idiomatic and conventional? By reading other people’s programs, in the same careful and intentional way that we read our own. You can’t write a good novel if you’ve never read a novel, and the same applies to programming.</p> <p>Programs found in the wild are of variable quality, and you should make sure to sample as wide a range as you can. It’s just as useful to read bad programs as good ones, though for different reasons. You’ll find mistakes even in good programs, and when you do, you’ll have learned something. But the most useful thing of all is to learn what everybody does.</p> <p><em>(This is probably the bigger issue around good code, because the code needs to be readable far more than it needs to be anything else. Now, the hard part is, how do we think about "conventional" here such that it can be measurable or testable, as in via static language analysis tools?)</em></p> <h4>Simple</h4> <p>Ah, simplicity. Is there any more slippery and deceptive concept? Everyone thinks they know simplicity when they see it, but oddly enough no two people agree on what “simple” means in practice. As Rich Hickey has pointed out, simple isn’t the same as easy. “Easy” is familiar, low-effort, the thing we reach for without thinking. That usually results in “complex”, so getting it to “simple” can take a good deal of effort and thought.</p> <p>One attribute that simple code tends to have is directness: it does just what it says on the tin. It doesn’t have weird and unexpected side effects, or conflate several unrelated things. Directness is inversely related to concision: if there are three similar tasks to do, a simple program will do it with three similar functions.</p> <p>This is one reason people find it hard to write simple code: we’re all terrified of repeating ourselves. The DRY principle is so ingrained that we even use it as a verb: “we need to DRY up this function” (as Calvin noted, verbing weirds language).</p> <p><em>(I am just one good push away from writing a "DRY is an antipattern" post.)</em></p> <p>But there’s nothing wrong with repetition in itself. I say again, there’s nothing wrong with repetition in itself (see?) It can be a valuable way of conveying emphasis. And if we find ourselves creating new abstractions to no purpose other than avoiding repetition, then we’ve gone wrong somewhere: that makes the program more complex, not more simple.</p> <p>That leads us to another aspect of simplicity: frugality. Doing a lot with a little. A package that does one thing is simpler than one that does ten things. The fewer functions there are, and the shallower the stack, the simpler the program. That might result in some long functions, and that’s okay. All due respect to Uncle Bob, but a function should be as long as it needs to be. Adding complexity purely to reduce the length of a function is a good example of a blindly-applied rule defeating common sense.</p> <p>Simplicity, as Alan Perlis observed, does not precede complexity, but follows it. In other words, write the program first, then make it simple. Read the code and ask what it says, then ask yourself if you can find a simpler way to write the same thing.</p> <p>Another place you can find simplicity is in naturalness. Any given language has its own Tao, its own natural forms and structures, and working with them generally gets better results than working against them. For example, you can use Go as though it were Java, or Python, or Rust, but it’s simpler to write Go programs in Go. Easier, too. <em>(This goes back to conventionality, I think.)</em></p> <h4>Performant</h4> <p>You might think it strange that I’ve listed this one last. Isn’t practically everything you hear or read about programming related to performance in some way? Yes, it is. But I don’t think that’s necessarily because performance is important so much as that it’s easy to talk about. What can be measured is what will be maximised, and it’s easy to measure performance: you can use a stopwatch. It’s much harder to quantify things like simplicity, or even correctness. But if the code isn’t correct, who cares how fast it runs? To put it another way, we can make a given function arbitrarily efficient if it doesn’t have to be correct.</p> <p>Similarly, if it’s not simple, we will waste far more programmer time on understanding it than we could ever have shaved off the CPU time it takes. And programmers are more expensive to run, per hour, than any CPU. Doesn’t it make sense to write code in a way that optimises for programmer time, not CPU time? As I’ve noted elsewhere, performance doesn’t matter for the vast majority of programs. When it does, the best solution isn’t usually to make your code harder to read. The saying “slow is smooth, and smooth is fast” applies here. If you tangle up your program in hopeless complexity to save a couple of microseconds off some loop, great, but that’s the last optimisation anyone will ever make to it. Attempting to speed up complex code is usually counterproductive, because if you don’t understand it you can’t optimise it. On the other hand, simple programs are easy to speed up when you have to.</p> <p>Nevertheless, we should be aware of the performance implications of the choices we make. If we didn’t need this task doing fairly quickly, we wouldn’t have given it to a computer. Another way to think about it is that an efficient program can do more in a given time, even if the actual time taken is irrelevant.</p> <p>To be fair, even an inefficient program will run pretty fast: “the inside of a computer is dumb as hell,” Richard Feynman observed, “but it goes like mad”. That’s not to say that we can afford to waste compute cycles, because all computation costs energy, and we are already heating up our planet at a ludicrously unsustainable rate. It would be a shame if we ended up emitting a few extra gigatons of carbon just because we made some clumsy choice of data structure.</p> <p>The idea of “mechanical sympathy” is helpful to bear in mind when you’re programming. It means that you have some understanding of how the machine works, fundamentally, and you take care not to abuse it or get in its way. For example, if you don’t know what memory is, how can you write efficient programs?</p> <p>I often see code that blithely slurps in entire data files into a single structure, then proceeds to actually process them just a few bytes at a time. I learned to program on a machine with 1K of memory, or about a thousand words, and that will teach you the value of a byte, believe me. I’m writing this, some years later, on a machine with about 16 million words of memory, and don’t run out of them quite so frequently, but nonetheless, memory matters. <em>(Ehh.... it matters when it matters, but not until then. Getting it right still trumps getting it fast.)</em></p> <p>For one thing, the more data you’re schlepping around the system, the longer it takes. And for another thing, however big your Kubernetes cluster, it nonetheless consists of physical machines with fixed, finite memory, and no container can use more than the total RAM of a single node. So look after your bytes, and the gigabytes will look after themselves. <em>(Yeah, not sure I'm a fan of this, tbh. Performance should be a deliberate choice, not something that is done by rote.)</em></p> https://tedneward.github.io/Research/reading/development/clean-code/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/clean-code/index.html <p><em>(by Robert Martin, aka "Uncle Bob", aka "the most misogynistic man in tech")</em></p> <p>A nice <a href="https://qntm.org/clean">takedown</a> of the book.</p> https://tedneward.github.io/Research/reading/development/compsci/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/compsci/index.html <h1>Intermediate-level material</h1> <ul> <li><a href="https://smile.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1848000693/"><em>The Algorithm Design Manual</em></a>, with <a href="https://www3.cs.stonybrook.edu/~skiena/373/videos/">video lectures</a> from the author (Steven Skiena).</li> <li><a href="https://smile.amazon.com/How-Solve-Mathematical-Princeton-Science/dp/069116407X/"><em>How to Solve It</em></a> by Polya, and there's a <a href="https://www.amazon.com/How-Solve-Heuristics-Zbigniew-Michalewicz/dp/3540224947">modern edition</a> that's more expansive and detailed. Both should be read, front to back.</li> <li><em>Structure and Interpretation of Computer Programs</em> (SICP): <a href="https://mitpress.mit.edu/sites/default/files/sicp/full-text/book/book.html">book</a> and <a href="http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-001-structure-and-interpretation-of-computer-programs-spring-2005/video-lectures/">MIT video lectures</a> or <a href="https://archive.org/details/ucberkeley-webcast-PL3E89002AA9B9879E?sort=titleSorter">Brian Harvey's SICP lectures, UCBerkeley 61A</a>. Mind-blowing concepts around code, data, and the malleability of the two that comes from spending time in a Lisp/Scheme. <ul> <li><a href="https://composingprograms.com/"><em>Composing Programs</em></a>, "in the tradition" of SICP but uses Python</li> <li><a href="http://www.htdp.org/"><em>How to Design Programs</em></a></li> </ul> </li> </ul> <h1>Most influential books for programmers</h1> <p>These are books considered most influential for programmers from this <a href="http://stackoverflow.com/questions/1711/what-is-the-single-most-influential-book-every-programmer-should-read">StackOverflow thread</a>.</p> <h2>Influential Books List</h2> <p><em>(I need to break this list out into other list categories)</em></p> <ol> <li><a href="http://cc2e.com/">Code Complete (2nd edition)</a> by Steve McConnell</li> <li><a href="http://pragprog.com/the-pragmatic-programmer">The Pragmatic Programmer</a> by Andrew Hunt and David Thomas</li> <li><a href="http://cm.bell-labs.com/cm/cs/cbook/">The C Programming Language</a> by Brian Kernighan and Dennis Ritchie</li> <li><a href="http://mitpress.mit.edu/books/introduction-algorithms">Introduction to Algorithms</a> by Cormen, Leiserson, Rivest &amp; Stein</li> <li><a href="http://c2.com/cgi/wiki?DesignPatternsBook">Design Patterns</a> by the Gang of Four</li> <li><a href="http://martinfowler.com/books/refactoring.html">Refactoring: Improving the Design of Existing Code</a> by Martin Fowler</li> <li><a href="http://www.amazon.com/The-Mythical-Man-Month-Engineering-Anniversary/dp/0201835959">The Mythical Man Month</a> by Fred Brooks</li> <li><a href="http://www-cs-faculty.stanford.edu/~uno/taocp.html">The Art of Computer Programming</a> by Donald Knuth</li> <li><a href="http://www.amazon.com/Compilers-Principles-Techniques-Tools-Edition/dp/0321486811">Compilers: Principles, Techniques and Tools</a> by Alfred V. Aho, Ravi Sethi and Jeffrey D. Ullman</li> <li><a href="http://www.amazon.com/G%C3%B6del-Escher-Bach-Eternal-Golden/dp/0465026567">Gödel, Escher, Bach</a> by Douglas Hofstadter</li> <li><a href="http://www.aristeia.com/books.html">Effective C++</a> by Scott Meyers</li> <li><a href="http://www.aristeia.com/books.html">More Effective C++</a> by Scott Meyers</li> <li><a href="http://www.charlespetzold.com/code/">CODE</a> by Charles Petzold</li> <li><a href="http://www.cs.bell-labs.com/cm/cs/pearls/">Programming Pearls</a> by Jon Bentley</li> <li><a href="http://www.informit.com/store/working-effectively-with-legacy-code-9780131177055?aid=15d186bd-1678-45e9-8ad3-fe53713e811b">Working Effectively with Legacy Code</a> by Michael C. Feathers</li> <li><a href="http://www.amazon.com/Peopleware-Productive-Projects-Second-Edition/dp/0932633439">Peopleware</a> by Demarco and Lister</li> <li><a href="http://www.codersatwork.com/">Coders at Work</a> by Peter Seibel</li> <li><a href="http://www.amazon.com/Surely-Feynman-Adventures-Curious-Character/dp/0393316041">Surely You're Joking, Mr. Feynman!</a> by Richard Feynman</li> <li><a href="http://www.amazon.com/Effective-Java-Edition-Joshua-Bloch/dp/0321356683">Effective Java 2nd edition</a> by Joshua Bloch</li> <li><a href="http://martinfowler.com/books/eaa.html">Patterns of Enterprise Application Architecture</a> by Martin Fowler</li> <li><a href="https://mitpress.mit.edu/books/little-schemer-fourth-edition">The Little Schemer</a> by Matthias Felleisen and Daniel P. Friedman</li> <li><a href="https://mitpress.mit.edu/books/seasoned-schemer-second-edition">The Seasoned Schemer</a> by Matthias Felleisen and Daniel P. Friedman</li> <li><a href="http://www.rubyinside.com/media/poignant-guide.pdf">Why's (Poignant) Guide to Ruby</a> by why the lucky stiff (free PDF)</li> <li><a href="http://www.amazon.com/The-Inmates-Are-Running-Asylum/dp/0672326140">The Inmates Are Running The Asylum: Why High Tech Products Drive Us Crazy and How to Restore the Sanity</a> by Alan Cooper</li> <li><a href="http://www.catb.org/~esr/writings/taoup/">The Art of Unix Programming</a> by Eric S. Raymond</li> <li><a href="http://www.eecs.yorku.ca/course_archive/2003-04/W/3311/sectionM/case_studies/money/KentBeck_TDD_byexample.pdf">Test-Driven Development: By Example</a> by Kent Beck (free PDF)</li> <li><a href="http://pragprog.com/book/pad/practices-of-an-agile-developer">Practices of an Agile Developer</a> by Venkat Subramaniam and Andy Hunt</li> <li><a href="http://www.sensible.com/dmmt.html">Don't Make Me Think</a> by Steve Krug</li> <li><a href="http://www.amazon.com/Software-Development-Principles-Patterns-Practices/dp/0135974445">Agile Software Development, Principles, Patterns, and Practices</a> by Robert C. Martin</li> <li><a href="http://www.amazon.com/Domain-Driven-Design-Tackling-Complexity-Software/dp/0321125215">Domain Driven Designs</a> by Eric Evans</li> <li><a href="http://www.jnd.org/books/the-design-of-everyday-things.html">The Design of Everyday Things</a> by Donald Norman</li> <li><a href="http://erdani.com/index.php/books/modern-c-design/">Modern C++ Design</a> by Andrei Alexandrescu</li> <li><a href="http://joelonsoftware.com/articles/BestSoftwareWriting.html">Best Software Writing I</a> by Joel Spolsky</li> <li><a href="http://cm.bell-labs.com/cm/cs/tpop/">The Practice of Programming</a> by Kernighan and Pike</li> <li><a href="http://pragprog.com/press_releases/pragmatic-thinking-and-learning-refactor-your-wetware">Pragmatic Thinking and Learning: Refactor Your Wetware</a> by Andy Hunt</li> <li><a href="http://www.stevemcconnell.com/est.htm">Software Estimation: Demystifying the Black Art</a> by Steve McConnel</li> <li><a href="http://pragprog.com/book/cfcar2/the-passionate-programmer">The Passionate Programmer</a> by Chad Fowler</li> <li><a href="http://www.gutenberg.org/ebooks/729">Hackers: Heroes of the Computer Revolution</a> by Steven Levy (free ebook)</li> <li><a href="http://www.ethoberon.ethz.ch/WirthPubl/AD.pdf">Algorithms + Data Structures = Programs</a> by Niklaus Wirth (free PDF)</li> <li><a href="http://c2.com/cgi/wiki?WritingSolidCode">Writing Solid Code</a> by Steve Maguire</li> <li><a href="http://javascript.crockford.com/">JavaScript - The Good Parts</a> by Douglas Crockford</li> <li><a href="https://gettingreal.37signals.com/">Getting Real</a> by 37 Signals (free PDF)</li> <li><a href="http://openmymind.net/FoundationsOfProgramming.pdf">Foundations of Programming</a> by Karl Seguin (free PDF)</li> <li><a href="http://www.amazon.com/Computer-Graphics-Principles-Practice-Edition/dp/0201848406">Computer Graphics: Principles and Practice in C (2nd Edition)</a> by Foley, Dam, Feiner, Hughes</li> <li><a href="http://www.mindviewinc.com/Books/TIJ4/">Thinking in Java</a> by Bruce Eckel</li> <li><a href="http://www.nand2tetris.org/">The Elements of Computing Systems</a> by Noam Nisan and Shimon Schocken(free ebook)</li> <li><a href="http://industriallogic.com/xp/refactoring/">Refactoring to Patterns</a> by Joshua Kerievsky</li> <li><a href="http://www.cs.vu.nl/~ast/books/mos2/">Modern Operating Systems</a> by Andrew S. Tanenbaum</li> <li><a href="http://www.theannotatedturing.com/">The Annotated Turing</a> by Charles Petzold</li> <li><a href="http://www.jnd.org/books/things-that-make-us-smart-defending-human-attributes-in-the-age-of-the-machine.html">Things That Make Us Smart</a> by Donald Norman</li> <li><a href="http://www.amazon.com/The-Timeless-Building-Christopher-Alexander/dp/0195024028">The Timeless Way of Building</a> by Christopher Alexander</li> <li><a href="http://tomdemarco.com/Books/deadline.html">The Deadline: A Novel About Project Management</a> by Tom DeMarco</li> <li><a href="http://www.stroustrup.com/3rd.html">The C++ Programming Language (3rd edition)</a> by Stroustrup</li> <li><a href="http://csapp.cs.cmu.edu/">Computer Systems - A Programmer's Perspective</a> by Bryant and O'Hallaron</li> <li><a href="http://www.amazon.com/Agile-Principles-Patterns-Practices-C/dp/0131857258">Agile Principles, Patterns, and Practices in C#</a> by Robert C. Martin</li> <li><a href="http://www.growing-object-oriented-software.com/">Growing Object-Oriented Software, Guided by Tests</a> by Steve Freeman and Nat Pryce</li> <li><a href="http://www.amazon.com/Framework-Design-Guidelines-Conventions-Libraries/dp/0321545613">Framework Design Guidelines</a> by Brad Abrams</li> <li><a href="http://www.microsoft.com/learning/en-us/book.aspx?ID=6820">Object Thinking</a> by Dr. David West</li> <li><a href="http://www.cs.stevens.edu/~jschauma/810D/">Advanced Programming in the UNIX Environment</a> by W. Richard Stevens</li> <li><a href="http://www.paulgraham.com/hackpaint.html">Hackers and Painters: Big Ideas from the Computer Age</a> by Paul Graham</li> <li><a href="http://www.tracykidder.com/books/soul/">The Soul of a New Machine</a> by Tracy Kidder</li> <li><a href="http://shop.oreilly.com/product/9780735627048.do">CLR via C#</a> by Jeffrey Richter</li> <li><a href="http://www.amazon.com/Design-Patterns-C-Software/dp/0321718933">Design Patterns in C#</a> by Steve Metsker</li> <li><a href="http://www.amazon.com/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0060589469">Zen and the Art of Motorcycle Maintenance</a> by Robert M. Pirsig</li> <li><a href="http://www.amazon.com/About-Face-Essentials-Interaction-Design/dp/0470084111">About Face - The Essentials of Interaction Design</a> by Alan Cooper</li> <li><a href="http://www.amazon.com/Here-Comes-Everybody-Organizing-Organizations/dp/0143114948">Here Comes Everybody: The Power of Organizing Without Organizations</a> by Clay Shirky</li> <li><a href="http://www.canonical.org/~kragen/tao-of-programming.html">The Tao of Programming</a> by Geoffrey James (free ebook)</li> <li><a href="https://mitpress.mit.edu/books/computational-beauty-nature">Computational Beauty of Nature</a> by Gary William Flake</li> <li><a href="http://philip.greenspun.com/panda/">Philip and Alex's Guide to Web Publishing</a> (free ebook)</li> <li><a href="http://www.amazon.com/Object-Oriented-Analysis-Design-Applications-Edition/dp/020189551X">Object-Oriented Analysis and Design with Applications</a> by Grady Booch</li> <li><a href="http://www.amazon.com/Computability-Introduction-Recursive-Function-Theory/dp/0521294657">Computability</a> by N. J. Cutland</li> <li><a href="http://www.amazon.com/Masterminds-Programming-Conversations-Creators-Languages/dp/0596515170">Masterminds of Programming</a> by Federico Biancuzzi</li> <li><a href="http://acc6.its.brooklyn.cuny.edu/~phalsall/texts/taote-v3.html">The Tao Te Ching</a> by Lao-tzu</li> <li><a href="http://nealford.com/books/productiveprogrammer">The Productive Programmer</a> by Neil Ford</li> <li><a href="http://www.amazon.com/exec/obidos/tg/detail/-/0764569597">The Art of Deception</a> by Kevin Mitnick</li> <li><a href="http://www.christopherduncan.com/thecareerprogrammer.aspx">The Career Programmer: Guerilla Tactics for an Imperfect World</a> by Christopher Duncan</li> <li><a href="http://norvig.com/paip.html">Paradigms of Artificial Intelligence Programming: Case studies in Common Lisp</a> by Peter Norvig (free ebook)</li> <li><a href="http://www.amazon.com/Masters-Doom-Created-Transformed-Culture/dp/0812972155">Masters of Doom</a> by David Kushner</li> <li><a href="http://pragprog.com/book/utc2/pragmatic-unit-testing-in-c-with-nunit">Pragmatic Unit Testing in C# with NUnit</a> by Andy Hunt and Dave Thomas</li> <li><a href="https://notendur.hi.is/hei2/teaching/Polya_HowToSolveIt.pdf">How To Solve It</a> by George Polya (free PDF)</li> <li><a href="http://stephane.ducasse.free.fr/FreeBooks/BlueBook/Bluebook.pdf">Smalltalk-80: The Language and its Implementation</a> by Adele Goldberg</li> <li><a href="http://www.microsoft.com/learning/en-us/book.aspx?ID=5957">Writing Secure Code (2nd Edition)</a> by Michael Howard</li> <li><a href="http://www.cs.ox.ac.uk/publications/publication2613-abstract.html">Introduction to Functional Programming</a> by Richard Bird</li> <li><a href="http://www.amazon.com/No-Bugs-Delivering-Error-Free/dp/0201608901">No Bugs!</a> by David Thielen</li> <li><a href="http://www.manning.com/tahchiev/">JUnit in Action</a> by Petar Tahchiev</li> </ol> https://tedneward.github.io/Research/reading/development/cps/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/cps/index.html <p><a href="https://en.wikipedia.org/wiki/Continuation-passing_style">Wikipedia</a>:<br> ... a style of programming in which control is passed explicitly in the form of a continuation. ... A function written in continuation-passing style takes an extra argument: an explicit "continuation"; i.e., a function of one argument. When the CPS function has computed its result value, it "returns" it by calling the continuation function with this value as the argument. That means that when invoking a CPS function, the calling function is required to supply a procedure to be invoked with the subroutine's "return" value. Expressing code in this form makes a number of things explicit which are implicit in direct style. These include: procedure returns, which become apparent as calls to a continuation; intermediate values, which are all given names; order of argument evaluation, which is made explicit; and tail calls, which simply call a procedure with the same continuation, unmodified, that was passed to the caller.</p> <p>In CPS, each procedure takes an extra argument representing what should be done with the result the function is calculating. This, along with a restrictive style prohibiting a variety of constructs usually available, is used to expose the semantics of programs, making them easier to analyze. This style also makes it easy to express unusual control structures, like catch/throw or other non-local transfers of control.</p> <p>The key to CPS is to remember that (a) every function takes an extra argument known as its continuation, and (b) every argument in a function call must be either a variable or a lambda expression (not a more complex expression). This has the effect of turning expressions "inside-out" because the innermost parts of the expression must be evaluated first, thus CPS makes explicit the order of evaluation as well as the control flow. Some examples of code in direct style and the corresponding CPS appear below. These examples are written in the Scheme programming language; by convention the continuation function is represented as a parameter named "k":</p> <p>Direct style:</p> <pre><code>(define (pyth x y) (sqrt (+ (* x x) (* y y)))) </code></pre> <p>CPS:</p> <pre><code>(define (pyth&amp; x y k) (*&amp; x x (lambda (x2) (*&amp; y y (lambda (y2) (+&amp; x2 y2 (lambda (x2py2) (sqrt&amp; x2py2 k)))))))) </code></pre> <p>Direct style:</p> <pre><code>(define (factorial n) (if (= n 0) 1 ; NOT tail-recursive (* n (factorial (- n 1))))) </code></pre> <p>CPS:</p> <pre><code>(define (factorial&amp; n k) (=&amp; n 0 (lambda (b) (if b ; growing continuation (k 1) ; in the recursive call (-&amp; n 1 (lambda (nm1) (factorial&amp; nm1 (lambda (f) (*&amp; n f k))))))))) </code></pre> <p>Direct style:</p> <pre><code>(define (factorial n) (f-aux n 1)) (define (f-aux n a) (if (= n 0) a ; tail-recursive (f-aux (- n 1) (* n a)))) </code></pre> <p>CPS:</p> <pre><code>(define (factorial&amp; n k) (f-aux&amp; n 1 k)) (define (f-aux&amp; n a k) (=&amp; n 0 (lambda (b) (if b ; unmodified continuation (k a) ; in the recursive call (-&amp; n 1 (lambda (nm1) (*&amp; n a (lambda (nta) (f-aux&amp; nm1 nta k))))))))) </code></pre> <p>Functions using more than one continuation can be defined to capture various control flow paradigms, for example (in Scheme):</p> <pre><code>(define (/&amp; x y ok err) (=&amp; y 0.0 (lambda (b) (if b (err (list "div by zero!" x y)) (ok (/ x y)))))) </code></pre> <h3>Haskell examples</h3> <p>Direct:</p> <pre><code>pow2 :: Float -&gt; Float pow2 a = a ** 2 add :: Float -&gt; Float -&gt; Float add a b = a + b pyth :: Float -&gt; Float -&gt; Float pyth a b = sqrt (add (pow2 a) (pow2 b)) </code></pre> <p>CPS:</p> <pre><code>pow2' :: Float -&gt; (Float -&gt; a) -&gt; a pow2' a cont = cont (a ** 2) add' :: Float -&gt; Float -&gt; (Float -&gt; a) -&gt; a add' a b cont = cont (a + b) -- Types a -&gt; (b -&gt; c) and a -&gt; b -&gt; c are equivalent, so CPS function -- may be viewed as a higher order function sqrt' :: Float -&gt; ((Float -&gt; a) -&gt; a) sqrt' a = \cont -&gt; cont (sqrt a) pyth' :: Float -&gt; Float -&gt; (Float -&gt; a) -&gt; a pyth' a b cont = pow2' a (\a2 -&gt; pow2' b (\b2 -&gt; add' a2 b2 (\anb -&gt; sqrt' anb cont))) </code></pre> <h3>Continuations as objects</h3> <p>Programming with continuations can also be useful when a caller does not want to wait until the callee completes. For example, in user-interface (UI) programming, a routine can set up dialog box fields and pass these, along with a continuation function, to the UI framework. This call returns right away, allowing the application code to continue while the user interacts with the dialog box. Once the user presses the "OK" button, the framework calls the continuation function with the updated fields.</p> <pre><code>function confirmName() { fields.name = name; framework.Show_dialog_box(fields, confirmNameContinuation); } function confirmNameContinuation(fields) { name = fields.name; } </code></pre> <p>A similar idea can be used when the function must run in a different thread or on a different processor. The framework can execute the called function in a worker thread, then call the continuation function in the original thread with the worker's results. This is in Java 8 using the Swing UI framework:</p> <pre><code>void buttonHandler() { // This is executing in the Swing UI thread. // We can access UI widgets here to get query parameters. int parameter = getField(); new Thread(() =&gt; { // This code runs in a separate thread. // We can do things like access a database or a // blocking resource like the network to get data. int result = lookup(parameter); javax.swing.SwingUtilities.invokeLater(() =&gt; { // This code runs in the UI thread and can use // the fetched data to fill in UI widgets. setField(result); }); }).start(); } </code></pre> https://tedneward.github.io/Research/reading/development/clean-architecture/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/clean-architecture/index.html <p><em>(by Robert Martin, aka "Uncle Bob", aka "the most misogynistic man in tech")</em></p> <h1>Part I:</h1> <h1>1: What is Design and Architecture?</h1> <p>"I’ll assert that there is no difference between [design and architecture]. None at all. ... The word “architecture” is often used in the context of something at a high level that is divorced from the lower-level details, whereas “design” more often seems to imply structures and decisions at a lower level. " Does his home have an architecture? It is the shape of the room, etc, but architecture diagrams provide an immense number of details. <em>(Fallacy: database logical schema vs physical schema, as a counter-example)</em> "The low-level details and the high-level structure are all part of the same whole. They form a continuous fabric that defines the shape of the system. You can’t have one without the other; indeed, no clear dividing line separates them." <em>(Disagree: design is the expression of code one level above the first-class citizen level of the language being used. Architecture is the set of decisions made ahead of time for developers, so that they are doing the "right" thing--by the project's standards and influenced by its problems--by default. They overlap, they influence each other, but architecture is clearly apart from design.)</em></p> <p>"<strong>The goal of software architecture is to minimize the human resources required to build and maintain the required system.</strong> The measure of design quality is simply the measure of the effort required to meet the needs of the customer. If that effort is low, and stays low throughout the lifetime of the system, the design is good. If that effort grows with each new release, the design is bad."</p> <p>[Anonymized case study showing engineering growth growing yet productivity declining.]</p> <p>[Tortoise v the Hare; counterexample is a 6-day experiment of one developer writing with TDD, and one day without.] "The only way to go fast, is to go well." <em>(The apples-to-oranges nature of the comparison here is just appalling. Nobody argues that quick-and-dirty yields great results every time--but the arrogance to assert that quick-and-dirty is never acceptable, particularly when executives are agreed on the consequences of doing so, is just beyond credulity.)</em></p> <h1>2: A Tale of Two Values</h1> <p>"Every software system provides two different values to the stakeholders: behavior and structure. Software developers are responsible for ensuring that both those values remain high. Unfortunately, they often focus on one to the exclusion of the other. Even more unfortunately, they often focus on the lesser of the two values, leaving the software system eventually valueless."</p> <p><strong>Behavior</strong>: "Programmers are hired to make machines behave in a way that makes or saves money for the stakeholders. <em>(Not all software is just about saving money for stakeholders.)</em> We do this by helping the stakeholders develop a functional specification, or requirements document. <em>(Fascinating, because this sounds vaguely waterfall-ish, and Bob supposedly embraces agile.)</em> Then we write the code that causes the stakeholder's machines to satisfy those requirements."</p> <p><strong>Architecture</strong>: "The second value of software has to do with the word "software" ... Software was invented to be "soft." It was intended to be a way to easily change the behavior of machines. Software was invented to be “soft.” It was intended to be a way to easily change the behavior of machines. ... When the stakeholders change their minds about a feature, that change should be simple and easy to make. The difficulty in making such a change should be proportional only to the scope of the change, and not to the shape of the change."</p> <p>"If you ask the business managers, they’ll often say that it’s more important for the software system to work. Developers, in turn, often go along with this attitude. But it’s the wrong attitude. I can prove that it is wrong with the simple logical tool of examining the extremes.</p> <ul> <li>"If you give me a program that works perfectly but is impossible to change, then it won’t work when the requirements change, and I won’t be able to make it work. Therefore the program will become useless.</li> <li>"If you give me a program that does not work but is easy to change, then I can make it work, and keep it working as requirements change. Therefore the program will remain continually useful."</li> </ul> <p>[Important/Urgent matrix]</p> <p>"The first value of software—behavior—is urgent but not always particularly important. The second value of software—architecture—is important but never particularly urgent."</p> <p>[Exhortation to "fight for the architecture"]</p> <h1>Part II: Programming Paradigms</h1> <p>"Paradigms are ways of programming, relatively unrelated to languages. A paradigm tells you which programming structures to use, and when to use them. To date, there have been three such paradigms." <em>(Not true, there've been numerous paradigms, many of which overlap. See <a href="/languages/paradigms-for-dummies">VanRoy</a>. And VanRoy really ignores <a href="/reading/languages/aspect-oriented.html">AOP</a> entirely to boot.)</em></p> <h1>3: Paradigm Overview</h1> <p>An overview chapter on structured programming, object-oriented programming, and functional programming. Summarized as follows:</p> <ul> <li>"Structured programming imposes discipline on direct transfer of control."</li> <li>"Object-oriented programming imposes discipline on indirect transfer of control."</li> <li>"Functional programming imposes discipline upon assignment."</li> </ul> <p><em>(All three of these are really incorrect analyses, because he wants to emphasize discipline on the developer. It's a very shallow and pretty unidimensional view of each.)</em></p> <h1>4: Structured Programming</h1> <p>[History of Dijkstra] "The problem that Dijkstra recognized, early on, was that programming is hard, and that programmers don’t do it very well. A program of any complexity contains too many details for a human brain to manage without help. Overlooking just one small detail results in programs that may seem to work, but fail in surprising ways." <em>(Language designers refer to this in the large as "separation of concerns".)</em> "Dijkstra’s solution was to apply the mathematical discipline of proof. His vision was the construction of a Euclidian hierarchy of postulates, theorems, corollaries, and lemmas. Dijkstra thought that programmers could use that hierarchy the way mathematicians do. In other words, programmers would use proven structures, and tie them together with code that they would then prove correct themselves. ... The very control structures that made a module provable were the same minimum set of control structures from which all programs can be built. Thus structured programming was born."</p> <p>["Goto Considered Harmful"]</p> <p>"Structured programming allows modules to be recursively decomposed into provable units, which in turn means that modules can be functionally decomposed. That is, you can take a large-scale problem statement and decompose it into high-level functions." <em>(Or you could take small problems, collect them into a function, and build up that way--top-down or bottom-up.)</em> "But the proofs never came. The Euclidean hierarchy of theorems was never built. And programmers at large never saw the benefits of working through the laborious process of formally proving each and every little function correct. In the end, Dijkstra’s dream faded and died. Few of today’s programmers believe that formal proofs are an appropriate way to produce high-quality software." <em>(Academics are constantly chasing that dream, as are language designers--every language-generation or so has an effort towards provably-correct code, the most recent having found roots in the pure-functional-language world.)</em></p> <p>"Science is fundamentally different from mathematics, in that scientific theories and laws cannot be proven correct. ... They are falsifiable but not provable. ... Science does not work by proving statements true, but rather by proving statements false. Those statements that we cannot prove false, after much effort, we deem to be true enough for our purposes." <em>(Here comes the argument for TDD)</em> "... a program can be proven incorrect by a test, but it cannot be proven correct. All that tests can do, after sufficient testing effort, is allow us to deem a program to be correct enough for our purposes."</p> <h1>5: Object-Oriented Programming</h1> <p>What is O-O? "One answer to this question is “The combination of data and function.” Although often cited, this is a very unsatisfying answer because it implies that <code>o.f()</code> is somehow different from <code>f(o)</code>. This is absurd. Programmers were passing data structures into functions long before 1966, when Dahl and Nygaard moved the function call stack frame to the heap and invented OO." <em>(Bob blithely assumes that the paradigm must be tied to the language--in fact, yes, the union of state and behavior is the principal focus of objects, and you can express that paradigm in languages that don't have objects as first-class citizens. C did it with GUI applications for many years before C++ came along. Paradigms are concepts, and languages are expressions of concepts, making it much easier to write code in a particular paradigm if you use a language that embraces that same paradigm.)</em></p> <p>"Some folks fall back on three magic words to explain the nature of OO: encapsulation, inheritance, and polymorphism. The implication is that OO is the proper admixture of these three things, or at least that an OO language must support these three things." <em>(Warning: Strawman argument coming up.)</em></p> <p>Encapsulation: C had "perfect" encapsulation, then C++ broke it, and Java/C# refuse to allow for separation of declaration and definition. <em>(Not surprisingly, he is skating very fast over lots of details here in his rush to prove a meaningless point.)</em></p> <p>Inheritance: Again with the "we had this in C" discussion, relying on C compiler behavior to not rearrange order of fields in a struct to provide "effective" substitution capabilities of a <code>NamedPoint</code> for a <code>Point</code>. <em>(Never mind the fact that there was zero compiler enforcement or recognition of the relationship, and therefore these two could easily drift out of sync and lead to extremely subtle bugs that would only manifest at runtime.)</em></p> <p><em>(Readers may get the feeling that Bob wants to go back to the days of C. This is common among the elderly.)</em></p> <p>Polymorphism: UNIX and its "everything is a file"/file descriptors design as an example of polymorphic behavior. <em>(He's falling back again on the "because we could do this before by hand, it's not that interesting to have" argument. Ironically, it's the exact opposite of Dijkstra's argument earlier: We could always do structured programming in assembly by using <code>goto</code> as necessary to implement it. It's a hugely inconsistent argument.)</em> "The bottom line is that polymorphism is an application of pointers to functions." <em>(Yup, I heard this argument from C programmers about C++ way back in the day.)</em> "OO languages may not have given us polymorphism, but they have made it much safer and much more convenient. ... Using an OO language makes polymorphism trivial. That fact provides an enormous power that old C programmers could only dream of. On this basis, we can conclude that OO imposes discipline on indirect transfer of control." <em>(That was some pretty fast skating there.)</em></p> <p>Power of polymorphism lies in being able to abstract without details. [Example of a copy program, where the source or target can be any file descriptor]</p> <p>"module HL1 calls the F() function in module ML1. The fact that it calls this function through an interface is a source code contrivance. At runtime, the interface doesn’t exist. HL1 simply calls F() within ML1. ... Note, however, that the source code dependency (the inheritance relationship) between ML1 and the interface I points in the opposite direction compared to the flow of control. This is called dependency inversion, and its implications for the software architect are profound." <em>(That... really isn't what dependency inversion refers to, at least not among the people who coined the term.)</em> "With this approach, software architects working in systems written in OO languages have absolute control over the direction of all source code dependencies in the system. They are not constrained to align those dependencies with the flow of control. No matter which module does the calling and which module is called, the software architect can point the source code dependency in either direction. ... you can rearrange the source code dependencies of your system so that the database and the user interface (UI) depend on the business rules, rather than the other way around. ... This means that the UI and the database can be plugins to the business rules. It means that the source code of the business rules never mentions the UI or the database. As a consequence, the business rules, the UI, and the database can be compiled into three separate components or deployment units (e.g., jar files, DLLs, or Gem files) that have the same dependencies as the source code. The component containing the business rules will not depend on the components containing the UI and database. In turn, the business rules can be deployed independently of the UI and the database. Changes to the UI or the database need not have any effect on the business rules. Those components can be deployed separately and independently. In short, when the source code in a component changes, only that component needs to be redeployed. This is independent deployability. If the modules in your system can be deployed independently, then they can be developed independently by different teams. That’s independent developability."</p> <p><em>(This is all great in theory, but in practice, it never manifests this way. Tellingly, there's no example here, even a simple one.)</em></p> <h1>6: Functional Programming</h1> <p>[Example: Squaring the first 25 integers in Java, Clojure/Lisp] "Variables in functional languages do not vary." <em>(sigh. That's why they usually don't get called variables, but values. Or name-value bindings. And only in pure-functional languages. And...)</em></p> <p>[Concurrency and immutability] "One of the most common compromises in regard to immutability is to segregate the application, or the services within the application, into mutable and immutable components. The immutable components perform their tasks in a purely functional way, without using any mutable variables. The immutable components communicate with one or more other components that are not purely functional, and allow for the state of variables to be mutated. Since mutating state exposes those components to all the problems of concurrency, it is common practice to use some kind of transactional memory to protect the mutable variables from concurrent updates and race conditions." <em>(Not sure where he's getting that STM is 'common practice'; it certainly isn't in any system I'm familiar with....)</em></p> <p>[Event sourcing] <em>(Greg Young strikes again)</em></p> <h1>Part III: Design Principles</h1> <ul> <li><strong>SRP: The Single Responsibility Principle</strong>: An active corollary to Conway’s law: The best structure for a software system is heavily influenced by the social structure of the organization that uses it so that each software module has one, and only one, reason to change.</li> <li><strong>OCP: The Open-Closed Principle</strong>: Bertrand Meyer made this principle famous in the 1980s. The gist is that for software systems to be easy to change, they must be designed to allow the behavior of those systems to be changed by adding new code, rather than changing existing code.</li> <li><strong>LSP: The Liskov Substitution Principle</strong>: Barbara Liskov’s famous definition of subtypes, from 1988. In short, this principle says that to build software systems from interchangeable parts, those parts must adhere to a contract that allows those parts to be substituted one for another.</li> <li><strong>ISP: The Interface Segregation Principle</strong>: This principle advises software designers to avoid depending on things that they don’t use.</li> <li><strong>DIP: The Dependency Inversion Principle</strong>: The code that implements high-level policy should not depend on the code that implements low-level details. Rather, details should depend on policies.</li> </ul> <h1>Part IV: Component Principles</h1> <h1>12: Components</h1> <p>"Components are the units of deployment. They are the smallest entities that can be deployed as part of a system." <em>(Not exactly, but it's not a terrible summation of Szyperski.)</em></p> <p>[PDP history around linking and loading and relocation] <em>(I have really no idea what his point here is with this.)</em></p> <h1>13: Component Cohesion</h1> <ul> <li><strong>REP: The Reuse/Release Equivalence Principle</strong> (The granule of reuse is the granule of release)</li> <li><strong>CCP: The Common Closure Principle</strong></li> <li><strong>CRP: The Common Reuse Principle</strong></li> </ul> <h1>14: Component Coupling</h1> <h1>Part V: Architecture</h1> <h1>15: What is Architecture?</h1> <h1>16: Independence</h1> <h1>17: Boundaries: Drawing Lines</h1> <h1>18: Boundary Anatomy</h1> <h1>19: Policy and Level</h1> <h1>20: Business Rules</h1> <h1>21: Screaming Architecture</h1> <h1>22: The Clean Architecture</h1> <h1>23: Presenters and Humble Objects</h1> <h1>24: Partial Boundaries</h1> <h1>25: Layers and Boundaries</h1> <h1>26: The Main Component</h1> <h1>27: Services: Great and Small</h1> <h1>28: The Test Boundary</h1> <h1>29: Clean Embedded Architecture</h1> <h1>Part VI: Details</h1> <h1>30: The Database is a Detail</h1> <h1>31: The Web is a Detail</h1> <h1>32: Frameworks are Details</h1> <h1>33: Case Study: Video Sales</h1> <h1>34: The Missing Chapter</h1> <h1>Part VII: Appendix</h1> <h1>A: Architecture Archaeology</h1> https://tedneward.github.io/Research/reading/development/beginners/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/beginners/index.html <p><a href="https://github.com/microsoft/Web-Dev-For-Beginners">Microsoft 12-week program</a>: 12-week, 24-lesson; HTML, CSS, JavaScript.</p> https://tedneward.github.io/Research/reading/development/blockchain/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/blockchain/index.html <h2>Reading</h2> <ul> <li><a href="http://bitcoinbook.cs.princeton.edu">Bitcoin and Cryptocurrency Technologies</a> - Arvind Narayanan, Joseph Bonneau, Edward Felten, Andrew Miller, Steven Goldfeder, Jeremy Clark (PDF)</li> <li><a href="https://www.ibm.com/downloads/cas/36KBMBOG">Blockchain for Dummies, 2nd IBM Limited Edition</a> - Manav Gupta (PDF)</li> <li><a href="https://chain.courses">chain.courses</a> - James Gan, Rishub Kumar</li> <li><a href="https://www.ibm.com/downloads/cas/RYWXAR0M">Getting Started with Enterprise Blockchain: A Guide to Design and Development</a> - Michael Bradley, David Gorman, Matt Lucas, Matthew Golby-Kirk (PDF)</li> <li><a href="https://www.ibm.com/downloads/cas/GZPPMWM5">IBM Blockchain: The Founder’s Handbook, Third Edition</a> - Antonio Banda, Matthew Hamilton, Eileen Lowry, John Widdifield et al. (PDF)</li> <li><a href="https://schadokar.dev/ebooks/playtime-with-hyperledger-composer/">Playtime with Hyperledger Composer</a> - Shubham Chadokar (PDF)</li> <li><a href="https://www.distributed-systems.net/my-data/papers/2021.blockchains.pdf">The Difficulty in Scaling Blockchains: A Simple Explanation</a> - Maarten van Steen - " In this paper, we provide a laymen's description of what may turn out to be a fundamental hurdle in attaining blockchains that combine scalability, high transaction processing capabilities, and are indeed fully decentralized."</li> </ul> https://tedneward.github.io/Research/reading/development/browser/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/browser/index.html <h2>Articles</h2> <p><a href="https://nolanlawson.com/2023/12/02/lets-learn-how-modern-javascript-frameworks-work-by-building-one/">"How modern Javascript Frameworks Work, By Building One"</a></p> https://tedneward.github.io/Research/reading/development/algorithms-data-structures/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/algorithms-data-structures/index.html <ul> <li><a href="https://web.archive.org/web/20191020195105/http://www0.cs.ucl.ac.uk/staff/W.Langdon/ftp/papers/poli08_fieldguide.pdf">A Field Guide To Genetic Programming</a> - Riccardo Poli et al. (PDF)</li> <li><a href="https://archive.org/details/AlgorithmDesign1stEditionByJonKleinbergAndEvaTardos2005PDF">Algorithm Design</a> - Jon Kleinberg and Éva Tardos (<a href="./AlgorithmDesign.pdf">PDF</a>)</li> <li>Algorithms and Data Structures - Wirth (<a href="./AlgorithmsDataStructures.pdf">PDF</a>)</li> <li><a href="http://code.google.com/p/graphbook/">Algorithmic Graph Theory</a></li> <li><a href="https://en.wikibooks.org/wiki/Algorithms">Algorithms</a> - Wikibooks</li> <li><a href="./Algorithms.pdf">Algorithms</a> - by Vazirani and Dasgupta</li> <li><a href="http://algs4.cs.princeton.edu/home/">Algorithms, 4th Edition</a> (<a href="./Algorithms-4E.pdf">PDF</a>)- Robert Sedgewick and Kevin Wayne</li> <li><a href="https://archive.org/details/Algorithms_And_Automatic_Computing_Machines">Algorithms and Automatic Computing Machines (1963)</a> - B. A. Trakhtenbrot</li> <li><a href="https://www.math.upenn.edu/~wilf/AlgoComp.pdf">Algorithms and Complexity</a> - Herbert S. Wilf (PDF)</li> <li><a href="http://jeffe.cs.illinois.edu/teaching/algorithms/">Algorithms Course Materials</a> - Jeff Erickson</li> <li><a href="https://goalkicker.com/AlgorithmsBook">Algorithms Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="http://www.cse.iitd.ernet.in/~ssen/csl356/admin356.html">Analysis and Design of Algorithms</a> - Sandeep Sen, IIT Delhi</li> <li><a href="http://visualgo.net">Animated Algorithm and Data Structure Visualization</a> (Resource)</li> <li><a href="https://github.com/mdipierro/nlib">Annotated Algorithms in Python: Applications in Physics, Biology, and Finance</a> - Massimo di Pierro</li> <li><a href="http://cslibrary.stanford.edu/110/BinaryTrees.pdf">Binary Trees</a> (PDF)</li> <li><a href="https://austinhenley.com/blog/challengingalgorithms.html">Challenging algorithms and data structures every programmer should try</a></li> <li><a href="https://adityacse.weebly.com/uploads/2/4/0/7/24078687/data-structures.pdf">Data Structures</a> - Aditya CSE (PDF)</li> <li><a href="https://en.wikibooks.org/wiki/Data_Structures">Data Structures</a> - Wikibooks</li> <li><a href="http://www.cse.iitd.ernet.in/~suban/cs130/index.html">Data Structures</a> - Prof. Subhashis Banerjee, IIT Delhi</li> <li><a href="http://www-inst.eecs.berkeley.edu/~cs61b/fa14/book2/data-structures.pdf">Data Structures (Into Java) - Paul N. Hilfinger</a> (PDF)</li> <li><a href="https://web.archive.org/web/20170715160229/http://dotnetslackers.com/Community/files/folders/data-structures-and-algorithms/entry30283.aspx">Data Structures and Algorithms: Annotated Reference with Examples</a> - G. Barnett and L. Del Tongo</li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/datastructurespart1">Data Structures Succinctly Part 1, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://www.syncfusion.com/resources/techportal/ebooks/datastructurespart2">Data Structures Succinctly Part 2, Syncfusion</a> (PDF, Kindle) (email address <em>requested</em>, not required)</li> <li><a href="https://github.com/liuxinyu95/AlgoXY">Elementary Algorithms</a> (<a href="./ElementaryAlgorithms.pdf">PDF</a>) - Larry LIU Xinyu</li> <li><a href="http://infolab.stanford.edu/~ullman/focs.html">Foundations of Computer Science</a> - Al Aho and Jeff Ullman</li> <li><a href="https://en.wikipedia.org/wiki/Fractal_tree_index">Fractal tree index - Wikipedia</a>: A refinement of the B-epsilon tree.</li> <li><a href="./FundamentalsComputerAlgorithm.pdf">Fundamentals of Computer Algorithms</a> - by Horowitz and Sahni</li> <li><a href="http://geomalgorithms.com">Geometry Algorithms</a> - Dan Sunday</li> <li><a href="http://ianparberry.com/books/free/license.html">Lectures Notes on Algorithm Analysis and Computational Complexity (Fourth Edition)</a> - Ian Parberry (use form at bottom of license)</li> <li><a href="http://people.mpi-inf.mpg.de/~mehlhorn/LEDAbook.html">LEDA: A Platform for Combinatorial and Geometric Computing</a> - K. Mehlhorn et al.</li> <li><a href="http://cslibrary.stanford.edu/103/LinkedListBasics.pdf">Linked List Basics</a> (PDF)</li> <li><a href="http://cslibrary.stanford.edu/105/LinkedListProblems.pdf">Linked List Problems</a> (PDF)</li> <li><a href="http://www.jjj.de/fxt/fxtbook.pdf">Matters Computational: Ideas, Algorithms, Source Code</a> (PDF)</li> <li><a href="http://opendatastructures.org">Open Data Structures: An Introduction</a> - Pat Morin</li> <li><a href="http://planning.cs.uiuc.edu">Planning Algorithms</a></li> <li><a href="http://ianparberry.com/books/free/license.html">Problems on Algorithms (Second Edition)</a> - Ian Parberry (use form at bottom of license)</li> <li><a href="https://web.archive.org/web/20210917054102/http://www.cs.cmu.edu/~rwh/theses/okasaki.pdf">Purely Functional Data Structures (1996)</a> - Chris Okasaki (PDF)</li> <li><a href="http://www.inf.fh-flensburg.de/lang/algorithmen/sortieren/algoen.htm">Sequential and parallel sorting algorithms</a></li> <li><a href="http://igm.univ-mlv.fr/~mac/REC/text-algorithms.pdf">Text Algorithms</a> (PDF)</li> <li><a href="http://www8.cs.umu.se/kurser/TDBAfl/VT06/algorithms/BOOK/BOOK/BOOK.HTM">The Algorithm Design Manual</a> (<a href="./TheAlgorithmDesignManual.pdf">PDF</a>) (<a href="https://github.com/SkienaBooks/Algorithm-Design-Manual-Programs">code</a>) - by Steven S. Skiena</li> <li><a href="http://www.designofapproxalgs.com/book.pdf">The Design of Approximation Algorithms</a> (PDF)</li> <li><a href="http://cslibrary.stanford.edu/109/TreeListRecursion.pdf">The Great Tree List Recursion Problem</a> (PDF)</li> <li><a href="https://www.syncfusion.com/ebooks/kademlia_protocol_succinctly">The Kademlia Protocol Succinctly</a> - Marc Clifton</li> <li><a href="https://greenteapress.com/wp/think-complexity-2e/">Think Complexity</a> - Allen B. Downey (2nd Edition) (PDF, HTML)</li> </ul> <p>Others not linked:</p> <p>Introduction to Algorithms (4th Ed) by Corman</p> <p>Grokking Algorithms (2nd Ed) by Bhargava</p> <p>Algorithms in a Nutshell by Heineman, Police and Selkow</p> <p>Introduction to Algorithms: A Creative Approach by Manber</p> <p>The Design and Analysis of Algorithms by Levitin</p> <p>Data Structures and Algorithms by Aho, Ullman &amp; Hopcroft</p> <p>Python Algorithms: Mastering the Basic Algorithms in the Python Language by Hetland</p> <p>Papers:</p> <ul> <li><a href="http://supertech.csail.mit.edu/papers/BenderFaJa15.pdf">An Introduction to Bε (B-epsilon) trees and Write-Optimization</a>: "A Bε-tree is an example of a write-optimized data structure and can be used to organize on-disk storage for an application such as a database or filesystem. A Bε-tree provides a key-value API, similar to a B-tree, but with better performance, particularly for inserts, range queries, and key-value updates. This article describes the Bε-tree,compares its asymptotic performance to B-trees and Log-Structured Merge trees (LSM-trees), and presents real-world performance measurements. After finishing this article, a reader should have a basic understanding of how a Bε-treeworks, its performance characteristics, how it compares to other key-value stores, and how to design applications to gain the most performance from a Bε-tree."</li> <li><a href="http://essay.utwente.nl/87368/1/Khavrona_BA_EEMCS.pdf">B-epsilon-tree and cache-oblivious lookahead array: a comparative study of two write-optimised data structures</a>: "The ever-growing amounts of data stored in the world require efficient and fast data structures to store and process it. Due to the large size of such massive datasets, the data structures that operate on them grow so large that they can no longer fit in main memory. Thus, the number of I/O operations between fast main memory and slow disk becomes the performance bottleneck of these data structures. To properly assess their performance, these data structures are analysed in the external memory model that puts emphasis on the number of blocks transferred between main memory and disk.Multiple data structures and their variations were developed in the external memory model to optimise the number of block transfers, among which the B-tree is the most well-known one. One of the research areas related to designing data structures in the external memory model has been focused on making data structures that keep the same search performance as the B-tree but asymptotically improve the speed of writes. Despite extensive theoretical results in the area, little experimental data about performance of such write-optimised data structures is available. In this research study,we analysed two write optimised data structures--the B-tree and cache-oblivious lookahead array(COLA)--and performed experiments to determine which data structure performs better under which conditions. As our results show, <strong><em>the COLA has much better write speeds than the B-tree when inserted elements are not sorted, but achieves worse results when the data is sorted. Point queries are faster in the B-tree, which makes it a better choice for workloads that require more querying than updating data.</em></strong> Lastly, <strong><em>the support of an efficient read-and-update operation and more streamlined experience of implementing the B-tree compared to the COLA make it an even more favourable data structure to consider for a use in data storage systems."</em></strong></li> <li><a href="https://github.com/rahulyesantharao/b-epsilon-tree">b-epsilon-tree</a>: A simple implementation of the write-optimized B-epsilon tree</li> <li><a href="https://driftingin.space/posts/you-might-not-need-a-crdt">You might not need a CRDT</a></li> </ul> https://tedneward.github.io/Research/reading/development/analysis/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/analysis/index.html <h1>General</h1> <ul> <li>Program Analysis Reading List - Rolf Rolles - <a href="http://www.msreverseengineering.com/program-analysis-reading-list/">http://www.msreverseengineering.com/program-analysis-reading-list/</a></li> <li>Reading for graduate students in static analysis - <a href="http://matt.might.net/articles/books-papers-materials-for-graduate-students/#analysis">http://matt.might.net/articles/books-papers-materials-for-graduate-students/#analysis</a></li> <li>Program Analysis Resources - <a href="http://reversing.io/resources/">http://reversing.io/resources/</a></li> <li>Conferences on Software Verification and Analysis - <a href="https://github.com/soarlab/conferences">https://github.com/soarlab/conferences</a></li> <li>Static Analysis Roadmap - <a href="https://tpiazza.me/posts/2017-11-01-static-analysis-roadmap.html">https://tpiazza.me/posts/2017-11-01-static-analysis-roadmap.html</a></li> </ul> <h1>Symbolic Execution</h1> <ul> <li>Awesome Symbolic Execution - <a href="https://github.com/ksluckow/awesome-symbolic-execution">https://github.com/ksluckow/awesome-symbolic-execution</a></li> <li>Symbolic execution history timeline: <a href="https://github.com/enzet/symbolic-execution">https://github.com/enzet/symbolic-execution</a></li> <li>Symbolic Execution: Intuition and Implementation - <a href="http://www.usrsb.in/symbolic-execution-intuition-and-implementation.html">http://www.usrsb.in/symbolic-execution-intuition-and-implementation.html</a></li> <li>A bibliography of papers related to symbolic execution - <a href="https://github.com/saswatanand/symexbib">https://github.com/saswatanand/symexbib</a></li> <li>A Survey of Symbolic Execution Techniques <ul> <li>ACM Computing Surveys 51(3) 2018</li> <li>Roberto Baldoni, Emilio Coppa, Daniele Cono D'Elia, Camil Demetrescu, Irene Finocchi</li> <li><a href="https://github.com/season-lab/survey-symbolic-execution">https://github.com/season-lab/survey-symbolic-execution</a></li> <li><a href="https://arxiv.org/abs/1610.00502">https://arxiv.org/abs/1610.00502</a></li> </ul> </li> <li>All You Ever Wanted to Know About Dynamic Taint Analysis and Forward Symbolic Execution (but might have been afraid to ask) <ul> <li>Security and Privacy (SP), 2010</li> <li>Edward J. Schwartz, Thanassis Avgerinos, David Brumley</li> <li>Paper: <a href="https://edmcman.github.io/papers/oakland10.pdf">https://edmcman.github.io/papers/oakland10.pdf</a></li> <li>Slides: <a href="https://edmcman.github.io/pres/oakland10.pdf">https://edmcman.github.io/pres/oakland10.pdf</a></li> </ul> </li> <li>Systematic Comparison of Symbolic Execution Systems: Intermediate Representation and its Generation <ul> <li>Annual Computer Security Applications Conference (ACSAC) 2019</li> <li>Sebastian Poeplau and Aurélien Francillon</li> <li><a href="http://s3.eurecom.fr/tools/symbolic_execution/">http://s3.eurecom.fr/tools/symbolic_execution/</a></li> <li><a href="http://s3.eurecom.fr/docs/acsac19_poeplau.pdf">http://s3.eurecom.fr/docs/acsac19_poeplau.pdf</a></li> <li><a href="https://hal.archives-ouvertes.fr/hal-02305914/">https://hal.archives-ouvertes.fr/hal-02305914/</a></li> </ul> </li> <li>Scaling Symbolic Evaluation for Automated Verification of Systems Code with Serval <ul> <li><a href="https://sosp19.rcs.uwaterloo.ca/program.html">SOSP 2019</a></li> <li>Luke Nelson, James Bornholt, Ronghui Gu, Andrew Baumann, Emina Torlak, Xi Wang</li> <li><a href="https://unsat.cs.washington.edu/projects/serval/">https://unsat.cs.washington.edu/projects/serval/</a></li> <li><a href="https://blog.acolyer.org/2019/11/13/scaling-symbolic-evaluation-serval/">https://blog.acolyer.org/2019/11/13/scaling-symbolic-evaluation-serval/</a></li> </ul> </li> </ul> <h1>Lectures &amp; Courses</h1> <ul> <li>Foundations of Programming Languages <ul> <li>M-PS (WS 2014/2015): Concepts of Programming Languages</li> <li><a href="http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/index.en.html">http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/index.en.html</a></li> <li><a href="http://web.archive.org/web/http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/index.en.html">http://web.archive.org/web/http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/index.en.html</a></li> <li>lectures (videos &amp; slides): <ul> <li><a href="https://www.youtube.com/channel/UCpSoGwyH5yHHvQut3x6c_2g/playlists">https://www.youtube.com/channel/UCpSoGwyH5yHHvQut3x6c_2g/playlists</a></li> <li><a href="http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/sessions.en.html">http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/sessions.en.html</a></li> <li><a href="http://web.archive.org/web/http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/sessions.en.html">http://web.archive.org/web/http://sepl.cs.uni-frankfurt.de/2014-ws/m-ps/sessions.en.html</a></li> </ul> </li> </ul> </li> <li>Static Program Analysis <ul> <li>Anders Møller and Michael I. Schwartzbach</li> <li><a href="https://cs.au.dk/~amoeller/spa/">https://cs.au.dk/~amoeller/spa/</a></li> <li>PLISS 2019 - Anders Møller <ul> <li>Static Program Analysis (part 1/2) - <a href="https://www.youtube.com/watch?v=Lr4cMmaJHrg">https://www.youtube.com/watch?v=Lr4cMmaJHrg</a></li> <li>Static Program Analysis (part 2/2) - <a href="https://www.youtube.com/watch?v=6QQSIIvH-F0">https://www.youtube.com/watch?v=6QQSIIvH-F0</a></li> </ul> </li> </ul> </li> <li>25 Years of Program Analysis <ul> <li>DEF CON 25 (2017) - Yan Shoshitaishvili (Zardus)</li> <li><a href="https://www.youtube.com/watch?v=XL9kWQ3YpLo">https://www.youtube.com/watch?v=XL9kWQ3YpLo</a></li> <li><a href="https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20presentations/DEFCON-25-Zardus-25-Years-of-Program-Analysis-UPDATED.pdf">https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20presentations/DEFCON-25-Zardus-25-Years-of-Program-Analysis-UPDATED.pdf</a></li> </ul> </li> <li>Software Analysis and Testing - <a href="http://www.cis.upenn.edu/~mhnaik/">Mayur Naik</a> <ul> <li><a href="http://rightingcode.org/">http://rightingcode.org/</a></li> <li><a href="http://www.cis.upenn.edu/~mhnaik/edu/cis700/">http://www.cis.upenn.edu/~mhnaik/edu/cis700/</a></li> <li><a href="https://www.youtube.com/channel/UCvwqRhlkE_Wm2FF9qzvHfJw">https://www.youtube.com/channel/UCvwqRhlkE_Wm2FF9qzvHfJw</a></li> </ul> </li> <li>Program Analysis and Reliability - Nick Sumner, CMPT 886, Spring 2015, SFU <ul> <li>Playlist: <a href="https://www.youtube.com/playlist?list=PLNC6lmsIySCOPjY8IwKBtD2cqe-MMgIGM">https://www.youtube.com/playlist?list=PLNC6lmsIySCOPjY8IwKBtD2cqe-MMgIGM</a></li> <li>Schedule &amp; Slides: <a href="http://www.cs.sfu.ca/~wsumner/teaching/886/15/schedule.html">http://www.cs.sfu.ca/~wsumner/teaching/886/15/schedule.html</a></li> </ul> </li> <li>Program analysis for reverse engineers: from T to ⊥ <ul> <li>BSides Canberra 2018; Adrian Herrera</li> <li><a href="https://www.youtube.com/watch?v=vOmGmjbVff4">https://www.youtube.com/watch?v=vOmGmjbVff4</a></li> <li>slides (not displayed in the video, may be a good idea to watch alongside):<br> <a href="https://drive.google.com/file/d/1j9rfMt14pubi6G9PKK3akddyeet5bf0x/view">https://drive.google.com/file/d/1j9rfMt14pubi6G9PKK3akddyeet5bf0x/view</a></li> </ul> </li> <li>CS 252r: Advanced Topics in Programming Languages <ul> <li><a href="http://web-static-aws.seas.harvard.edu/courses/cs252/2011sp/">http://web-static-aws.seas.harvard.edu/courses/cs252/2011sp/</a></li> <li><a href="https://www.seas.harvard.edu/courses/cs252/2015fa/schedule.html">https://www.seas.harvard.edu/courses/cs252/2015fa/schedule.html</a></li> </ul> </li> <li>A Gentle Introduction to Program Analysis <ul> <li>Programming Languages Mentoring Workshop 2014</li> <li>Isıl Dillig (University of Texas)</li> <li><a href="https://www.cis.upenn.edu/~alur/CIS673/isil-plmw.pdf">https://www.cis.upenn.edu/~alur/CIS673/isil-plmw.pdf</a></li> </ul> </li> </ul> <h1>Software</h1> <ul> <li>SPARTA: a library that provides the basic blocks for building high-performance static code analyzers based on Abstract Interpretation <ul> <li><a href="https://github.com/facebookincubator/SPARTA">https://github.com/facebookincubator/SPARTA</a></li> <li>SPARTA: a C++ library of software components for building high-performance static analyzers</li> <li><a href="https://code.fb.com/open-source/sparta/">https://code.fb.com/open-source/sparta/</a></li> </ul> </li> <li>Bill Torpey - <a href="http://btorpey.github.io/blog/categories/static-analysis/">static analysis posts</a> <ul> <li>Static Analysis with Clang - <a href="http://btorpey.github.io/blog/2015/04/27/static-analysis-with-clang/">http://btorpey.github.io/blog/2015/04/27/static-analysis-with-clang/</a></li> <li>Mo' Static - <a href="http://btorpey.github.io/blog/2016/04/07/mo-static/">http://btorpey.github.io/blog/2016/04/07/mo-static/</a></li> <li>Even Mo' Static - <a href="http://btorpey.github.io/blog/2016/11/12/even-mo-static/">http://btorpey.github.io/blog/2016/11/12/even-mo-static/</a></li> <li>Lots o' static - <a href="http://btorpey.github.io/blog/2017/09/17/lotso-static/">http://btorpey.github.io/blog/2017/09/17/lotso-static/</a></li> </ul> </li> </ul> <h2>LLVM</h2> <ul> <li>awesome-llvm: A curated list of awesome LLVM related docs, tools, and other resources <ul> <li><a href="https://github.com/HongxuChen/awesome-llvm">https://github.com/HongxuChen/awesome-llvm</a></li> </ul> </li> <li>LLVM Weekly - <a href="http://llvmweekly.org/">http://llvmweekly.org/</a></li> <li><a href="https://eli.thegreenplace.net/tag/llvm-clang">https://eli.thegreenplace.net/tag/llvm-clang</a></li> <li>Building a Checker in 24 hours <ul> <li>2012 LLVM Developers’ Meeting; Anna Zaks, Jordan Rose</li> <li><a href="https://llvm.org/devmtg/2012-11/Zaks-Rose-Checker24Hours.pdf">https://llvm.org/devmtg/2012-11/Zaks-Rose-Checker24Hours.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=kdxlsP5QVPw">https://www.youtube.com/watch?v=kdxlsP5QVPw</a></li> </ul> </li> <li>Code transformation and analysis using Clang and LLVM <ul> <li>Static and Dynamic Analysis</li> <li>HPC Summer School 2017; Hal Finkel</li> <li><a href="https://llvm.org/devmtg/2017-06/2-Hal-Finkel-LLVM-2017.pdf">https://llvm.org/devmtg/2017-06/2-Hal-Finkel-LLVM-2017.pdf</a></li> <li><a href="https://github.com/hfinkel/llvm-ss-2017">https://github.com/hfinkel/llvm-ss-2017</a></li> </ul> </li> <li>Compiler-assisted Performance Analysis <ul> <li>2016 LLVM Developers’ Meeting; Adam Nemet</li> <li><a href="https://github.com/llvm-mirror/llvm/tree/master/tools/opt-viewer">https://github.com/llvm-mirror/llvm/tree/master/tools/opt-viewer</a></li> <li><a href="https://llvm.org/devmtg/2016-11/Slides/Nemet-Compiler-assistedPerformanceAnalysis.pdf">https://llvm.org/devmtg/2016-11/Slides/Nemet-Compiler-assistedPerformanceAnalysis.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=qq0q1hfzidg">https://www.youtube.com/watch?v=qq0q1hfzidg</a></li> </ul> </li> <li>Dg: LLVM Static Slicer <ul> <li>Dependence graph for programs. A set of generic program analyses and a static slicer for LLVM bitcode</li> <li><a href="https://github.com/mchalupa/dg">https://github.com/mchalupa/dg</a></li> </ul> </li> <li>FPSCEV: Floating-Point Scalar Evolution in LLVM <ul> <li><a href="https://github.com/sheredom/fpscev">https://github.com/sheredom/fpscev</a></li> <li><a href="http://www.duskborn.com/posts/fpscev/">http://www.duskborn.com/posts/fpscev/</a></li> <li><a href="http://www.duskborn.com/posts/fpscev-fast-math-propagation/">http://www.duskborn.com/posts/fpscev-fast-math-propagation/</a></li> <li><a href="http://www.duskborn.com/posts/fpscev-inst-simplify/">http://www.duskborn.com/posts/fpscev-inst-simplify/</a></li> <li><a href="http://www.duskborn.com/posts/fpscev-improved-range/">http://www.duskborn.com/posts/fpscev-improved-range/</a></li> </ul> </li> <li>LLVM Dataflow Info Printer Pass <ul> <li><a href="https://github.com/regehr/llvm-dataflow-info">https://github.com/regehr/llvm-dataflow-info</a></li> <li>Tell us what some of LLVM's dataflow analyses think about the code being compiled.</li> <li>Testing Dataflow Analyses for Precision and Soundness <ul> <li><a href="https://blog.regehr.org/archives/1709">https://blog.regehr.org/archives/1709</a></li> </ul> </li> <li>Testing Static Analyses for Precision and Soundness <ul> <li>CGO (Code Generation and Optimization) 2020</li> <li>Jubi Taneja, Zhengyang Liu, John Regehr</li> <li><a href="http://www.cs.utah.edu/~regehr/cgo20.pdf">http://www.cs.utah.edu/~regehr/cgo20.pdf</a></li> </ul> </li> </ul> </li> <li>LLVM Debugging Tips and Tricks <ul> <li><a href="http://bholt.org/posts/llvm-debugging.html">http://bholt.org/posts/llvm-debugging.html</a></li> </ul> </li> <li>Phasar - A LLVM-based static code analysis framework <ul> <li><a href="http://phasar.org/">http://phasar.org/</a></li> <li><a href="https://github.com/secure-software-engineering/phasar">https://github.com/secure-software-engineering/phasar</a></li> <li>Static Analysis for C++ with Phasar</li> <li><a href="https://pldi18.sigplan.org/event/pldi-2018-pldi-tutorials-static-analysis-for-c-with-phasar">https://pldi18.sigplan.org/event/pldi-2018-pldi-tutorials-static-analysis-for-c-with-phasar</a></li> <li>PhASAR: An Inter-Procedural Static Analysis Framework for C/C++ <ul> <li>TACAS 2019</li> <li>Philipp D. Schubert, Ben Hermann, Eric Bodden</li> <li><a href="http://www.thewhitespace.de/publications/shb19-phasar.pdf">http://www.thewhitespace.de/publications/shb19-phasar.pdf</a></li> </ul> </li> </ul> </li> <li>SVF: Interprocedural Static Value-Flow Analysis in LLVM <ul> <li>Pointer Analysis and Program Dependence Analysis for C and C++ Programs</li> <li><a href="http://svf-tools.github.io/SVF/">http://svf-tools.github.io/SVF/</a></li> <li><a href="https://github.com/unsw-corg/SVF">https://github.com/unsw-corg/SVF</a></li> <li>SVF: Interprocedural Static Value-Flow Analysis in LLVM <ul> <li>Compiler Construction (CC '16)</li> <li>Yulei Sui and Jingling Xue</li> <li><a href="https://yuleisui.github.io/publications/cc16.pdf">https://yuleisui.github.io/publications/cc16.pdf</a></li> <li>2016 EuroLLVM Developers' Meeting: Y. Sui "SVF: Static Value-Flow Analysis in LLVM"</li> <li><a href="https://www.youtube.com/watch?v=nD-i-enA8rc">https://www.youtube.com/watch?v=nD-i-enA8rc</a></li> </ul> </li> </ul> </li> </ul> <h3>LLVM - Symbolic Execution</h3> <ul> <li>haybale: Symbolic execution of LLVM IR with an engine written in Rust <ul> <li>operates on LLVM IR, which allows it to analyze programs written in C/C++, Rust, Swift, or any other language which compiles to LLVM IR</li> <li><a href="https://github.com/PLSysSec/haybale">https://github.com/PLSysSec/haybale</a></li> </ul> </li> <li>KLEE Symbolic Virtual Machine <ul> <li><a href="http://klee.github.io/">http://klee.github.io/</a></li> <li><a href="http://klee.github.io/publications/">http://klee.github.io/publications/</a></li> <li><a href="https://github.com/klee/klee">https://github.com/klee/klee</a></li> </ul> </li> </ul> <h3>LLVM - Verification</h3> <ul> <li>LLBMC: The Low-Level Bounded Model Checker <ul> <li><a href="http://llbmc.org/">http://llbmc.org/</a></li> <li><a href="http://llbmc.org/publications.html">http://llbmc.org/publications.html</a></li> <li>LLBMC: Bounded Model Checking of C and C++ Programs Using a Compiler IR <ul> <li>Verified Software: Theories, Tools and Experiments (VSTTE) 2012</li> <li>Florian Merz, Stephan Falke, Carsten Sinz</li> <li><a href="https://doi.org/10.1007/978-3-642-27705-4_12">https://doi.org/10.1007/978-3-642-27705-4_12</a></li> <li><a href="http://llbmc.org/files/papers/VSTTE12.pdf">http://llbmc.org/files/papers/VSTTE12.pdf</a></li> <li><a href="http://llbmc.org/files/talks/vstte-2012.pdf">http://llbmc.org/files/talks/vstte-2012.pdf</a></li> </ul> </li> </ul> </li> <li>llrêve: Automatic regression verification for LLVM programs <ul> <li>Automatically check programs for equivalence</li> <li><a href="https://github.com/mattulbrich/llreve">https://github.com/mattulbrich/llreve</a></li> <li><a href="https://formal.iti.kit.edu/projects/improve/reve/">https://formal.iti.kit.edu/projects/improve/reve/</a></li> <li>Relational Program Reasoning Using Compiler IR – Combining Static Verification and Dynamic Analysis <ul> <li>Journal of Automated Reasoning 60(3) 2017</li> <li>Moritz Kiefer, Vladimir Klebanov, Mattias Ulbrich</li> <li><a href="http://dx.doi.org/10.1007/s10817-017-9433-5">http://dx.doi.org/10.1007/s10817-017-9433-5</a></li> </ul> </li> </ul> </li> <li>SAW: Software Analysis Workbench <ul> <li><a href="https://saw.galois.com/">https://saw.galois.com/</a></li> <li>Constructing Semantic Models of Programs with the Software Analysis Workbench <ul> <li>Verified Software: Theories, Tools and Experiments (VSTTE) 2016</li> <li>Robert Dockins, Adam Foltzer, Joe Hendrix, Brian Huffman, Dylan McNamee, Aaron Tomb</li> <li><a href="https://doi.org/10.1007/978-3-319-48869-1_5">https://doi.org/10.1007/978-3-319-48869-1_5</a></li> </ul> </li> </ul> </li> <li>SeaHorn Verification Framework: A fully automated analysis framework for LLVM-based languages <ul> <li><a href="http://seahorn.github.io/">http://seahorn.github.io/</a></li> <li><a href="https://github.com/seahorn/seahorn">https://github.com/seahorn/seahorn</a></li> <li><a href="http://seahorn.github.io/#publications">http://seahorn.github.io/#publications</a></li> <li>The SeaHorn Verification Framework <ul> <li>Computer Aided Verification (CAV) 2015</li> <li>A. Gurfinkel, T. Kahsai, A. Komuravelli, J.A.Navas</li> <li><a href="http://seahorn.github.io/papers/cav15.pdf">http://seahorn.github.io/papers/cav15.pdf</a></li> </ul> </li> <li>A Context-Sensitive Memory Model for Verification of C/C++ Programs <ul> <li>Slides: <a href="http://seahorn.github.io/papers/sas17_slides.pdf">http://seahorn.github.io/papers/sas17_slides.pdf</a></li> <li>Paper: <a href="http://seahorn.github.io/papers/sea-dsa-SAS17.pdf">http://seahorn.github.io/papers/sea-dsa-SAS17.pdf</a></li> </ul> </li> <li>Crab-llvm: Abstract Interpretation of LLVM bitcode <ul> <li><a href="https://github.com/seahorn/crab-llvm">https://github.com/seahorn/crab-llvm</a></li> </ul> </li> </ul> </li> <li>SMACK Software Verifier and Verification Toolchain <ul> <li><a href="http://smackers.github.io">http://smackers.github.io</a></li> <li><a href="https://github.com/smackers/smack">https://github.com/smackers/smack</a></li> <li>SMACK: Decoupling Source Language Details from Verifier Implementations <ul> <li>Computer Aided Verification (CAV) 2014</li> <li>Zvonimir Rakamaric, Michael Emmi</li> <li><a href="https://soarlab.org/publications/2014_CAV_SMACK/">https://soarlab.org/publications/2014_CAV_SMACK/</a></li> </ul> </li> <li>SMACK Software Verification Toolchain <ul> <li>International Conference on Software Engineering (ICSE) Companion 2016</li> <li>Montgomery Carter, Shaobo He, Jonathan Whitaker, Zvonimir Rakamaric, Michael Emmi</li> <li><a href="https://soarlab.org/publications/2016_ICSE_SMACK/">https://soarlab.org/publications/2016_ICSE_SMACK/</a></li> </ul> </li> <li>Leveraging Compiler Intermediate Representation for Multi- and Cross-Language Verification <ul> <li>Verification, Model Checking, and Abstract Interpretation (VMCAI) 2020</li> <li>Jack Garzella, Marek S. Baranowski, Shaobo He, Zvonimir Rakamaric</li> <li><a href="https://soarlab.org/publications/2020_vmcai_gbhr/">https://soarlab.org/publications/2020_vmcai_gbhr/</a></li> </ul> </li> </ul> </li> </ul> <h3>Clang</h3> <ul> <li>clang-tutor: A collection of Clang plugins - learn the API through examples <ul> <li><a href="https://github.com/banach-space/clang-tutor/">https://github.com/banach-space/clang-tutor/</a></li> </ul> </li> <li>My First Clang Warning <ul> <li>2019 LLVM Developers’ Meeting</li> <li>Meike Baumgärtner, Dmitri Gribenko</li> <li><a href="https://www.youtube.com/watch?v=FNnKMSkaLkY">https://www.youtube.com/watch?v=FNnKMSkaLkY</a></li> <li><a href="https://llvm.org/devmtg/2019-10/talk-abstracts.html#tut11">https://llvm.org/devmtg/2019-10/talk-abstracts.html#tut11</a></li> </ul> </li> </ul> <h4>Clang Static Analyzer</h4> <ul> <li>Developing the Clang Static Analyzer <ul> <li>2019 LLVM Developers’ Meeting; Artem Dergachev</li> <li><a href="https://www.youtube.com/watch?v=g0Mqx1niUi0">https://www.youtube.com/watch?v=g0Mqx1niUi0</a></li> </ul> </li> <li>Faster, Stronger C++ Analysis with the Clang Static Analyzer <ul> <li>2018 LLVM Developers’ Meeting; George Karpenkov, Artem Dergachev</li> <li><a href="https://www.youtube.com/watch?v=4n3l-ZcDJNY">https://www.youtube.com/watch?v=4n3l-ZcDJNY</a></li> </ul> </li> </ul> <h3>Introduction</h3> <ul> <li>Building Program Reasoning Tools using LLVM and Z3 <ul> <li>POPL 2020 Tutorial</li> <li>Introduction to LLVM and Z3; Static Dataflow Analysis, Dynamic Symbolic Execution, Assertion Verification</li> <li><a href="http://rightingcode.org/tutorials/popl20/">http://rightingcode.org/tutorials/popl20/</a></li> </ul> </li> <li>Getting Started With LLVM: Basics <ul> <li>2019 LLVM Developers’ Meeting; Jessica Paquette, Florian Hahn</li> <li><a href="https://www.youtube.com/watch?v=3QQuhL-dSys">https://www.youtube.com/watch?v=3QQuhL-dSys</a></li> </ul> </li> <li>How to Contribute to LLVM <ul> <li>2019 LLVM Developers’ Meeting; Chris Bieneman, Kit Barton</li> <li><a href="https://www.youtube.com/watch?v=C5Y977rLqpw">https://www.youtube.com/watch?v=C5Y977rLqpw</a></li> </ul> </li> <li>Introduction to LLVM <ul> <li>2019 LLVM Developers’ Meeting</li> <li>Eric Christopher &amp; Johannes Doerfert</li> <li><a href="https://www.youtube.com/watch?v=J5xExRGaIIY">https://www.youtube.com/watch?v=J5xExRGaIIY</a></li> <li><a href="https://llvm.org/devmtg/2019-10/talk-abstracts.html#tut9">https://llvm.org/devmtg/2019-10/talk-abstracts.html#tut9</a></li> </ul> </li> <li>Intrinsics, Metadata, and Attributes: The story continues! <ul> <li>2016 LLVM Developers’ Meeting</li> <li>Hal Finkel, Argonne National Laboratory</li> <li><a href="https://www.youtube.com/watch?v=jII0AcgU_5c">https://www.youtube.com/watch?v=jII0AcgU_5c</a></li> </ul> </li> <li>LLVM IR Tutorial - Phis, GEPs and other things, oh my! <ul> <li>2019 EuroLLVM Developers’ Meeting; Vince Bridgers, Felipe de Azevedo Piovezan (Intel)</li> <li><a href="https://www.youtube.com/watch?v=m8G_S5LwlTo">https://www.youtube.com/watch?v=m8G_S5LwlTo</a></li> </ul> </li> <li>LLVM Seminar - Northeastern+MIT <ul> <li>2019; Mike Shah</li> <li><a href="http://www.mshah.io/#Talks">http://www.mshah.io/#Talks</a></li> <li><a href="https://www.youtube.com/playlist?list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N">https://www.youtube.com/playlist?list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N</a></li> <li>Introduction to LLVM <ul> <li><a href="https://www.youtube.com/watch?v=KTMk45Q0d-8&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N">https://www.youtube.com/watch?v=KTMk45Q0d-8&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N</a></li> <li><a href="http://www.mshah.io/LLVM/NortheasternMITIntroduction%20to%20LLVM.pdf">http://www.mshah.io/LLVM/NortheasternMITIntroduction%20to%20LLVM.pdf</a></li> <li><a href="http://www.mshah.io/LLVM/llvm_6_3_19.zip">http://www.mshah.io/LLVM/llvm_6_3_19.zip</a></li> </ul> </li> <li>Introduction to Clang <ul> <li><a href="https://www.youtube.com/watch?v=RAzre6PA-WI&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N">https://www.youtube.com/watch?v=RAzre6PA-WI&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N</a></li> <li><a href="http://www.mshah.io/LLVM/NortheasternMITIntroductiontoClang.pdf">http://www.mshah.io/LLVM/NortheasternMITIntroductiontoClang.pdf</a></li> </ul> </li> <li>Introduction to Program Analysis using LLVM <ul> <li><a href="http://www.mshah.io/LLVM/NortheasternMITIntroductiontoProgramAnalysisusingLLVM.pdf">http://www.mshah.io/LLVM/NortheasternMITIntroductiontoProgramAnalysisusingLLVM.pdf</a></li> <li><a href="https://www.youtube.com/watch?v=w6SD3ramrwI&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N">https://www.youtube.com/watch?v=w6SD3ramrwI&amp;list=PLvv0ScY6vfd8NDoT7qUab4VVAWV67oH-N</a></li> </ul> </li> </ul> </li> <li>Security Research and Development with LLVM - Andrew Reiter <ul> <li><a href="https://github.com/roachspray/opcde2017">https://github.com/roachspray/opcde2017</a></li> <li><a href="https://github.com/comaeio/OPCDE/tree/master/2017/Security%20Research%20and%20Development%20with%20LLVM%20-%20Andrew%20Reiter">https://github.com/comaeio/OPCDE/tree/master/2017/Security%20Research%20and%20Development%20with%20LLVM%20-%20Andrew%20Reiter</a></li> <li>Examples - <a href="https://github.com/roachspray/opcde2017/tree/master/code">https://github.com/roachspray/opcde2017/tree/master/code</a></li> </ul> </li> <li>clang-llvm-tutorial <ul> <li>Clang and LLVM Tutorial Examples (AST Interpreter, Function Pointer Analysis, Value Range Analysis, Data-Flow Analysis, Andersen Pointer Analysis, LLVM Backend)</li> <li><a href="https://github.com/lijiansong/clang-llvm-tutorial/">https://github.com/lijiansong/clang-llvm-tutorial/</a></li> </ul> </li> <li>Custom Alias Analysis in LLVM <ul> <li><a href="https://blog.tartanllama.xyz/llvm-alias-analysis/">https://blog.tartanllama.xyz/llvm-alias-analysis/</a></li> </ul> </li> <li>Introduction to LLVM: Building simple program analysis tools and instrumentation <ul> <li><a href="https://fosdem.org/2018/schedule/track/llvm_toolchain/">FOSDEM 2018</a> - Mike Shah</li> <li><a href="https://fosdem.org/2018/schedule/event/introduction/">https://fosdem.org/2018/schedule/event/introduction/</a></li> <li><a href="https://www.youtube.com/watch?v=VKIv_Bkp4pk">https://www.youtube.com/watch?v=VKIv_Bkp4pk</a></li> <li>slides &amp; code: <a href="http://www.mshah.io/fosdem18.html">http://www.mshah.io/fosdem18.html</a></li> </ul> </li> <li>LLVMPlayground: Small sample programs that use LLVM and Clang APIs. <ul> <li><a href="https://github.com/modocache/LLVMPlayground">https://github.com/modocache/LLVMPlayground</a></li> </ul> </li> <li>Mapping High Level Constructs to LLVM IR <ul> <li><a href="https://github.com/f0rki/mapping-high-level-constructs-to-llvm-ir">https://github.com/f0rki/mapping-high-level-constructs-to-llvm-ir</a></li> <li><a href="https://mapping-high-level-constructs-to-llvm-ir.readthedocs.io/">https://mapping-high-level-constructs-to-llvm-ir.readthedocs.io/</a></li> </ul> </li> <li><a href="https://www.cs.sfu.ca/~wsumner/">Nick Sumner</a>'s Examples <ul> <li>slides: <a href="https://www.cs.sfu.ca/~wsumner/teaching/886/llvm.pdf">https://www.cs.sfu.ca/~wsumner/teaching/886/llvm.pdf</a></li> <li>llvm-demo: A simple example of how LLVM can be used to gather static or dynamic facts about a program. <ul> <li><a href="https://github.com/nsumner/llvm-demo">https://github.com/nsumner/llvm-demo</a></li> </ul> </li> <li>callgraph-profiler-template: A template for an introductory project on dynamic analysis using LLVM <ul> <li><a href="https://github.com/nsumner/callgraph-profiler-template">https://github.com/nsumner/callgraph-profiler-template</a></li> </ul> </li> <li>clang-plugins-demo: A simple example of defining custom plugins for Clang and the Clang Static Analyzer <ul> <li><a href="https://github.com/nsumner/clang-plugins-demo">https://github.com/nsumner/clang-plugins-demo</a></li> </ul> </li> <li>llvm-dataflow-analysis: (very simple) static intraprocedural dataflow analyses <ul> <li><a href="https://github.com/nsumner/llvm-dataflow-analysis">https://github.com/nsumner/llvm-dataflow-analysis</a></li> </ul> </li> <li>overflower-template: Template for a project to teach basic static dataflow analysis using LLVM <ul> <li><a href="https://github.com/nsumner/overflower-template">https://github.com/nsumner/overflower-template</a></li> </ul> </li> <li>path-profiler-template: A template for a path profiling project using LLVM <ul> <li><a href="https://github.com/nsumner/path-profiler-template">https://github.com/nsumner/path-profiler-template</a></li> </ul> </li> </ul> </li> <li>Quarkslab blog <ul> <li><a href="https://blog.quarkslab.com/global-dead-code-elimination-for-llvm-revisited.html">https://blog.quarkslab.com/global-dead-code-elimination-for-llvm-revisited.html</a></li> <li><a href="https://blog.quarkslab.com/turning-regular-code-into-atrocities-with-llvm-the-return.html">https://blog.quarkslab.com/turning-regular-code-into-atrocities-with-llvm-the-return.html</a></li> <li><a href="https://blog.quarkslab.com/turning-regular-code-into-atrocities-with-llvm.html">https://blog.quarkslab.com/turning-regular-code-into-atrocities-with-llvm.html</a></li> </ul> </li> <li>Writing a basic clang static analysis check <ul> <li><a href="https://bbannier.github.io/blog/2015/05/02/Writing-a-basic-clang-static-analysis-check.html">https://bbannier.github.io/blog/2015/05/02/Writing-a-basic-clang-static-analysis-check.html</a></li> </ul> </li> </ul> <h3>Instrumentation</h3> <ul> <li>Creating an LLVM Sanitizer from Hopes and Dreams <ul> <li><a href="https://blog.trailofbits.com/2019/06/25/creating-an-llvm-sanitizer-from-hopes-and-dreams/">https://blog.trailofbits.com/2019/06/25/creating-an-llvm-sanitizer-from-hopes-and-dreams/</a></li> <li>llvm-sanitizer-tutorial and documentation <ul> <li><a href="https://github.com/trailofbits/llvm-sanitizer-tutorial">https://github.com/trailofbits/llvm-sanitizer-tutorial</a></li> </ul> </li> </ul> </li> <li>Instrew: Leveraging LLVM for High Performance Dynamic Binary Instrumentation <ul> <li>VEE 2020</li> <li>Alexis Engelke, Martin Schulz</li> <li><a href="https://conf.researchr.org/details/vee-2020/vee-2020-papers/5/Instrew-Leveraging-LLVM-for-High-Performance-Dynamic-Binary-Instrumentation">https://conf.researchr.org/details/vee-2020/vee-2020-papers/5/Instrew-Leveraging-LLVM-for-High-Performance-Dynamic-Binary-Instrumentation</a></li> <li><a href="https://github.com/aengelke/instrew">https://github.com/aengelke/instrew</a></li> </ul> </li> <li>Loom: LLVM instrumentation library <ul> <li><a href="https://github.com/cadets/loom">https://github.com/cadets/loom</a></li> </ul> </li> <li>PolyTracker: An LLVM-based instrumentation tool for universal taint analysis. <ul> <li><a href="https://github.com/trailofbits/polytracker">https://github.com/trailofbits/polytracker</a></li> <li><a href="https://blog.trailofbits.com/2019/11/01/two-new-tools-that-tame-the-treachery-of-files/">https://blog.trailofbits.com/2019/11/01/two-new-tools-that-tame-the-treachery-of-files/</a></li> </ul> </li> <li>QBDI (QuarkslaB Dynamic binary Instrumentation): A Dynamic Binary Instrumentation framework based on LLVM <ul> <li><a href="https://qbdi.quarkslab.com">https://qbdi.quarkslab.com</a></li> <li><a href="https://github.com/quarkslab/QBDI">https://github.com/quarkslab/QBDI</a></li> <li>34C3 (2017) Implementing an LLVM based Dynamic Binary Instrumentation framework: <a href="https://events.ccc.de/congress/2017/Fahrplan/events/9006.html">https://events.ccc.de/congress/2017/Fahrplan/events/9006.html</a></li> <li>Example: plugging Triton on top of QBDI - <a href="http://shell-storm.org/repo/Notepad/qbdi_with_triton.txt">http://shell-storm.org/repo/Notepad/qbdi_with_triton.txt</a></li> <li>A Preliminary Test of QBDI - <a href="https://www.johnfxgalea.com/2018/01/13/a-preliminary-test-of-qbdi/">https://www.johnfxgalea.com/2018/01/13/a-preliminary-test-of-qbdi/</a></li> <li>Example: SRAC - a Simple Return Address Checker - <a href="https://github.com/johnfxgalea/SRAC">https://github.com/johnfxgalea/SRAC</a></li> </ul> </li> <li>sbt-instrumentation: Configurable instrumentation of LLVM bitcode <ul> <li><a href="https://github.com/staticafi/sbt-instrumentation">https://github.com/staticafi/sbt-instrumentation</a></li> <li>Instrumentation of LLVM IR - <a href="https://is.muni.cz/th/409920/fi_m/?lang=en">https://is.muni.cz/th/409920/fi_m/?lang=en</a></li> </ul> </li> </ul> <h3>Lifting</h3> <p>Lifting: Disassembly, Decompilation, Recompilation, Reverse Engineering</p> <ul> <li>ANVILL Decompiler Toolchain <ul> <li>ANVILL forges beautiful LLVM bitcode out of raw machine code</li> <li><a href="https://github.com/lifting-bits/anvill">https://github.com/lifting-bits/anvill</a></li> </ul> </li> <li>decomp: Compositional Decompilation using LLVM IR <ul> <li><a href="https://github.com/decomp/decomp">https://github.com/decomp/decomp</a></li> <li>Design documents: <a href="https://github.com/decomp/doc">https://github.com/decomp/doc</a></li> </ul> </li> <li>llvm-mctoll <ul> <li>This tool statically (AOT) translates (or raises) binaries to LLVM IR.</li> <li><a href="https://github.com/Microsoft/llvm-mctoll">https://github.com/Microsoft/llvm-mctoll</a></li> <li>Raising binaries to LLVM IR with MCTOLL <ul> <li>LCTES 2019 (WIP paper)</li> <li>S. Bharadwaj Yadavalli, Aaron Smith</li> <li><a href="https://dl.acm.org/citation.cfm?id=3326354">https://dl.acm.org/citation.cfm?id=3326354</a></li> <li><a href="https://conf.researchr.org/details/LCTES-2019/LCTES-2019-papers/15/Raising-Binaries-to-LLVM-IR-with-MCTOLL-Work-in-progress-">https://conf.researchr.org/details/LCTES-2019/LCTES-2019-papers/15/Raising-Binaries-to-LLVM-IR-with-MCTOLL-Work-in-progress-</a></li> </ul> </li> </ul> </li> <li>McSema: Framework for lifting x86, amd64, and aarch64 program binaries to LLVM bitcode <ul> <li><a href="https://www.trailofbits.com/research-and-development/mcsema/">https://www.trailofbits.com/research-and-development/mcsema/</a></li> <li><a href="https://github.com/lifting-bits/mcsema">https://github.com/lifting-bits/mcsema</a></li> <li>Decompiling Binaries into LLVM IR Using McSema and Dyninst <ul> <li><a href="https://is.muni.cz/th/pxe1j/?lang=en">https://is.muni.cz/th/pxe1j/?lang=en</a></li> </ul> </li> </ul> </li> <li>Rellic: produces goto-free C output from LLVM bitcode <ul> <li>Rellic is an implementation of the pattern-independent structuring algorithm to produce a goto-free C output from LLVM bitcode.</li> <li><a href="https://github.com/lifting-bits/rellic">https://github.com/lifting-bits/rellic</a></li> </ul> </li> <li>Rellume — Lifts x86-64 to LLVM IR <ul> <li><a href="https://github.com/aengelke/rellume">https://github.com/aengelke/rellume</a></li> <li>Rellume is a lifter for x86-64 machine code to LLVM IR with focus on the performance of the lifted code. The generated LLVM IR can be compiled and executed again, for example using LLVM's JIT compiler, ideally having the same (or even better) performance as the original code.</li> </ul> </li> <li>Remill: Library for lifting of x86, amd64, and aarch64 machine code to LLVM bitcode <ul> <li><a href="https://github.com/lifting-bits/remill">https://github.com/lifting-bits/remill</a></li> </ul> </li> <li>RetDec: a retargetable machine-code decompiler based on LLVM <ul> <li><a href="https://retdec.com/">https://retdec.com/</a></li> <li><a href="https://github.com/avast-tl/retdec">https://github.com/avast-tl/retdec</a></li> </ul> </li> <li>revng: a static binary translator <ul> <li>revng is a static binary translator. Given a input ELF binary for one of the supported architectures (currently MIPS, ARM and x86-64) it will analyze it and emit an equivalent LLVM IR. To do so, revng employs the QEMU intermediate representation (a series of TCG instructions) and then translates them to LLVM IR.</li> <li><a href="https://rev.ng/">https://rev.ng/</a></li> <li><a href="https://github.com/revng/revng">https://github.com/revng/revng</a></li> </ul> </li> </ul> <h3>Passes</h3> <ul> <li>Building, Testing and Debugging a Simple out-of-tree LLVM Pass <ul> <li>2016 EuroLLVM Developers' Meeting</li> <li>Serge Guelton &amp; Adrien Guinet</li> <li><a href="http://llvm.org/devmtg/2016-03/Tutorials/Tutorial.pdf">http://llvm.org/devmtg/2016-03/Tutorials/Tutorial.pdf</a></li> <li><a href="https://github.com/quarkslab/llvm-dev-meeting-tutorial-2015">https://github.com/quarkslab/llvm-dev-meeting-tutorial-2015</a></li> <li><a href="https://blog.quarkslab.com/goto-llvm_dev_meeting.html">https://blog.quarkslab.com/goto-llvm_dev_meeting.html</a></li> <li><a href="https://www.youtube.com/watch?v=Z5KcwVaak3s">https://www.youtube.com/watch?v=Z5KcwVaak3s</a></li> </ul> </li> <li>llvm-pass-tutorial: A step-by-step tutorial for building an LLVM sample pass <ul> <li><a href="https://github.com/abenkhadra/llvm-pass-tutorial">https://github.com/abenkhadra/llvm-pass-tutorial</a></li> </ul> </li> <li>llvm-tutor: Basic LLVM passes for learning the API <ul> <li><a href="https://github.com/banach-space/llvm-tutor">https://github.com/banach-space/llvm-tutor</a></li> <li>Writing an LLVM Pass: 101 <ul> <li>2019 LLVM Developers’ Meeting; Andrzej Warzynski</li> <li><a href="https://www.youtube.com/watch?v=ar7cJl2aBuU">https://www.youtube.com/watch?v=ar7cJl2aBuU</a></li> <li><a href="http://llvm.org/devmtg/2019-10/talk-abstracts.html#tut4">http://llvm.org/devmtg/2019-10/talk-abstracts.html#tut4</a></li> </ul> </li> </ul> </li> </ul> <h4>Legacy Pass Manager</h4> <ul> <li>2007 LLVM Developer's Meeting <ul> <li><a href="http://llvm.org/devmtg/2007-05/03-Patel-Passmanager.pdf">http://llvm.org/devmtg/2007-05/03-Patel-Passmanager.pdf</a></li> <li>LLVM Pass Manager Demystified (1 of 3) - <a href="https://www.youtube.com/watch?v=dZOrlikTaik">https://www.youtube.com/watch?v=dZOrlikTaik</a></li> <li>LLVM Pass Manager Demystified (2 of 3) - <a href="https://www.youtube.com/watch?v=PaUWxVLGBg0">https://www.youtube.com/watch?v=PaUWxVLGBg0</a></li> <li>LLVM Pass Manager Demystified (3 of 3) - <a href="https://www.youtube.com/watch?v=4HEy5EtVdCA">https://www.youtube.com/watch?v=4HEy5EtVdCA</a></li> </ul> </li> </ul> <h3>New Pass Manager</h3> <ul> <li>New PM: taming a custom pipeline of Falcon JIT <ul> <li>2018 EuroLLVM Developers’ Meeting - Fedor Sergeev, Azul Systems</li> <li><a href="https://www.youtube.com/watch?v=6X12D46sRFw">https://www.youtube.com/watch?v=6X12D46sRFw</a></li> <li><a href="http://llvm.org/devmtg/2018-04/slides/Sergeev-Taming%20a%20custom%20pipeline%20of%20Falcon%20JIT.pdf">http://llvm.org/devmtg/2018-04/slides/Sergeev-Taming%20a%20custom%20pipeline%20of%20Falcon%20JIT.pdf</a></li> </ul> </li> <li>The LLVM Pass Manager, Part 2 <ul> <li>2014 LLVM Developers' Meeting</li> <li>Chandler Carruth, Google</li> <li><a href="https://llvm.org/devmtg/2014-10/#talk11">https://llvm.org/devmtg/2014-10/#talk11</a></li> <li><a href="https://llvm.org/devmtg/2014-10/Slides/Carruth-TheLLVMPassManagerPart2.pdf">https://llvm.org/devmtg/2014-10/Slides/Carruth-TheLLVMPassManagerPart2.pdf</a></li> <li>Video: <a href="http://web.archive.org/web/20160718071630/http://llvm.org/devmtg/2014-10/Videos/The%20LLVM%20Pass%20Manager%20Part%202-720.mov">http://web.archive.org/web/20160718071630/http://llvm.org/devmtg/2014-10/Videos/The%20LLVM%20Pass%20Manager%20Part%202-720.mov</a></li> </ul> </li> <li>Passes in LLVM, Part 1 <ul> <li>2014 European LLVM Conference</li> <li>Chandler Carruth</li> <li><a href="https://www.youtube.com/watch?v=rY02LT08-J8">https://www.youtube.com/watch?v=rY02LT08-J8</a></li> <li><a href="https://llvm.org/devmtg/2014-04/PDFs/Talks/Passes.pdf">https://llvm.org/devmtg/2014-04/PDFs/Talks/Passes.pdf</a></li> </ul> </li> <li>Writing LLVM Pass in 2018 <ul> <li>Preface - <a href="https://medium.com/@mshockwave/writing-llvm-pass-in-2018-preface-6b90fa67ae82">https://medium.com/@mshockwave/writing-llvm-pass-in-2018-preface-6b90fa67ae82</a></li> <li>Part I: Write a new HelloWorld Pass in new pass manager fashion <ul> <li><a href="https://medium.com/@mshockwave/writing-llvm-pass-in-2018-preface-6b90fa67ae82">https://medium.com/@mshockwave/writing-llvm-pass-in-2018-preface-6b90fa67ae82</a></li> </ul> </li> <li>Part II - AnalysisManager - <a href="https://medium.com/@mshockwave/writing-llvm-pass-in-2018-part-ii-640f680978ec">https://medium.com/@mshockwave/writing-llvm-pass-in-2018-part-ii-640f680978ec</a></li> <li>Part III - In-Tree Pass Integration - <a href="https://medium.com/@mshockwave/writing-llvm-pass-in-2018-part-iii-d44cd0c2c354">https://medium.com/@mshockwave/writing-llvm-pass-in-2018-part-iii-d44cd0c2c354</a></li> </ul> </li> </ul> <h1>Readings</h1> <ul> <li>A Few Billion Lines of Code Later: Using Static Analysis to Find Bugs in the Real World <ul> <li>Communications of the ACM, Vol. 53 No. 2, 2010</li> <li>Al Bessey, Ken Block, Ben Chelf, Andy Chou, Bryan Fulton, Seth Hallem, Charles Henri-Gros, Asya Kamsky, Scott McPeak, Dawson Engler</li> <li><a href="https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext">https://cacm.acm.org/magazines/2010/2/69354-a-few-billion-lines-of-code-later/fulltext</a></li> </ul> </li> <li>Analysing the Program Analyser <ul> <li><a href="https://srg.doc.ic.ac.uk/publications/analyse-analyser-icse-v2025.html">https://srg.doc.ic.ac.uk/publications/analyse-analyser-icse-v2025.html</a></li> </ul> </li> <li>Continuous Reasoning: Scaling the Impact of Formal Methods <ul> <li>Logic in Computer Science (LISC) 2018; Peter O'Hearn</li> <li><a href="https://research.fb.com/publications/continuous-reasoning-scaling-the-impact-of-formal-methods/">https://research.fb.com/publications/continuous-reasoning-scaling-the-impact-of-formal-methods/</a></li> </ul> </li> <li>From Start-ups to Scale-ups: Opportunities and Open Problems for Static and Dynamic Program Analysis <ul> <li>IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) 2018</li> <li>Mark Harman, Peter O'Hearn</li> <li><a href="https://research.fb.com/publications/from-start-ups-to-scale-ups-opportunities-and-open-problems-for-static-and-dynamic-program-analysis/">https://research.fb.com/publications/from-start-ups-to-scale-ups-opportunities-and-open-problems-for-static-and-dynamic-program-analysis/</a></li> </ul> </li> <li>How to Prevent the next Heartbleed <ul> <li>2014; David A. Wheeler</li> <li><a href="https://dwheeler.com/essays/heartbleed.html">https://dwheeler.com/essays/heartbleed.html</a></li> </ul> </li> <li>Lessons from Building Static Analysis Tools at Google (2018) <ul> <li><a href="https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/fulltext">https://cacm.acm.org/magazines/2018/4/226371-lessons-from-building-static-analysis-tools-at-google/fulltext</a></li> </ul> </li> <li>Pointer analysis: haven't we solved this problem yet? <ul> <li>PASTE 2001</li> <li><a href="https://dl.acm.org/citation.cfm?id=379665">https://dl.acm.org/citation.cfm?id=379665</a></li> <li><a href="http://www.cs.trinity.edu/~mlewis/CSCI3294-F01/Papers/p54-hind.pdf">http://www.cs.trinity.edu/~mlewis/CSCI3294-F01/Papers/p54-hind.pdf</a></li> </ul> </li> <li>Righting Software <ul> <li>IEEE Software, May 2004</li> <li><a href="https://www.microsoft.com/en-us/research/publication/righting-software/">https://www.microsoft.com/en-us/research/publication/righting-software/</a></li> </ul> </li> <li>Source Code Analysis: A Road Map <ul> <li>Future of Software Engineering (FOSE) 2007</li> <li>David Binkley</li> <li><a href="https://dl.acm.org/citation.cfm?id=1254713">https://dl.acm.org/citation.cfm?id=1254713</a></li> </ul> </li> <li>Static versus dynamic analysis---an illusory distinction? <ul> <li><a href="https://www.cs.kent.ac.uk/people/staff/srk21/blog/research/static-and-dynamic-analyses.html">https://www.cs.kent.ac.uk/people/staff/srk21/blog/research/static-and-dynamic-analyses.html</a></li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/development/art-of-concurrency/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/art-of-concurrency/index.html <p><em>(by (author), (publisher))</em></p> <h1>8 Simple Rules for Designing Multithreaded Applications</h1> <ol> <li>Identify truly independent computations</li> <li>Implement concurrency at the highest level possible</li> <li>Plan early for scalability</li> <li>Make use of thread-safe libraries</li> <li>Use the right threading model (don't use explicit threads if an implicit model will do)</li> <li>Never assume a particular order of execution</li> <li>Use thread-local storage or associate locks to specific data</li> <li>Dare to change the algorithm for a better chance of concurrency</li> </ol> <h1>Terminology</h1> <p><strong>Barrier</strong> -- sync object that blocks all threads until every thread has arrived; upon arrival of last thread, all threads are released to continue</p> <p><strong>Benign Data Race</strong> -- data race w/no adverse consequences</p> <p><strong>Complete Binary Tree</strong> -- binary tree in which each level (except perhas the deepest) is full; leaf nodes placed as far left as possible</p> <p><strong>CRCW/CREW/ERCW/EREW</strong> -- Concurrent vs Exclusive Read vs Write</p> <p><strong>Data Decomposition</strong> -- identify independent work forcusing on dividing large data sets that can be processed concurrently</p> <p><strong>Data Flow Parallelism</strong> -- execute an instruction when args are ready (as opposed to the original sequence in which instruction was written)</p> <p><strong>Data Race</strong> -- 2+ threads accessing same resource/location where one is attempting to update</p> <p><strong>Deadlock</strong> -- 1+ threads are waiting for an event that will never occur</p> <p><strong>Fork-Join Parallelism</strong> -- threads are forked/spawned, execute, and join before continuing</p> <p><strong>Livelock</strong> -- threads doing same computation are unable to proceed due to the actions of some other thread (Robin Hood &amp; Little John meeting on the log bridge)</p> <p><strong>Modulo Locks</strong> -- associate multiple indexed objects w/ single lock</p> <p><strong>Mutex/Mutual Exclusion</strong> -- only allow one thread to execute given portion of code</p> <p><strong>Race Condition</strong> -- flaw in which result depends on timing or sequence of multiple threads' execution</p> <p><strong>Reduction</strong> -- computation that takes large data set and computes a value from that data; operation performed must be associative &amp; commutative</p> <p><strong>Semaphore</strong> -- Sync object w/ associated non-negative count w/2 operations: post (increment count) &amp; wait (if ct != 0, decrement count, else block until ct &gt; 0)</p> <p><strong>Starvation</strong> -- thread is not allowed to proceed (usually due to higher-priority threads getting scheduled first)</p> <p><strong>Task Decomposition</strong> -- identify independent work focusing on the computations to be performed by the threads</p> https://tedneward.github.io/Research/reading/development/art-of-unix-programming/index.html Mon, 13 Apr 2026 14:04:56 +0000 reading/development/art-of-unix-programming/index.html <p><em>(by Eric S Raymond (Addison-Wesley, 2004, ISBN 0-13-142901-9))</em></p> <h1>Basics of the UNIX Philosophy</h1> <p><strong>Rule of Modularity</strong>: Write simple parts connected by clean interfaces</p> <p><strong>Rule of Clarity</strong>: Clarity is better than cleverness</p> <p><strong>Rule of Composition</strong>: Design programs to be connected with other programs</p> <p><strong>Rule of Separation</strong>: Separate policy from mechanism; separate interfaces from engines</p> <p><strong>Rule of Simplicity</strong>: Design for simplicity; add complexity only where you must</p> <p><strong>Rule of Parsimony</strong>: Write a big program only when it is clear by demonstration that nothing else will do</p> <p><strong>Rule of Transparency</strong>: Design for visibility to make inspection and debugging easier</p> <p><strong>Rule of Robustness</strong>: Robustness is the child of transparency and simplicity</p> <p><strong>Rule of Representation</strong>: Fold knowledge into data, so program logic can be stupid and robust</p> <p><strong>Rule of Least Surprise</strong>: In interface design, always do the least surprising thing</p> <p><strong>Rule of Silence</strong>: When a program has nothing surprising to say, it should say nothing</p> <p><strong>Rule of Repair</strong>: Repair what you can--but when you must fail, fail noisily and as soon as possible</p> <p><strong>Rule of Economy</strong>: Programmer time is expensive; conserve it in preference to machine time</p> <p><strong>Rule of Generation</strong>: Avoid hand-hacking; write programs to write programs when you can</p> <p><strong>Rule of Optimization</strong>: Prototype before polishing. Get it working before you optimize it</p> <p><strong>Rule of Diversity</strong>: Distrust all claims for "one true way"</p> <p><strong>Rule of Extensibility</strong>: Design for the future, because it will be here sooner than you think</p> https://tedneward.github.io/Research/reading/business/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/index/index.html <h2>Reading</h2> <h3>Websites</h3> <p><a href="https://commoncog.com/">Commoncog</a>: Commoncog helps business leaders and investors make better decisions in uncertain times, by expanding the set of business scenarios that they recognise in their heads. We research and publish business cases that help you do better, organised according to business concepts, using a theory of accelerated expertise that’s proven to work.</p> <ul> <li><a href="https://commoncog.com/syllabus/">Commoncog Syllabus</a>: This syllabus is the recommended path to consume Commoncog’s exploration of business expertise. Start Here.</li> <li><a href="https://commoncog.com/goodharts-law-not-useful/">Goodhart's Law Isn't as Useful as you might think</a>: "It is descriptive; it tells you of the existence of a phenomenon, but it doesn’t tell you what to do about it or how to solve it. ... The first step is to turn Goodhart’s Law as a narrower, more actionable formulation. The one that I like the most is from Deming contemporary Donald Wheeler, who writes, in <a href="https://www.goodreads.com/book/show/63859">Understanding Variation</a>: <pre><code>&gt; When people are pressured to meet a target value there are three ways they can proceed: &gt; &gt; 1) They can work to improve the system &gt; 2) They can distort the system &gt; 3) Or they can distort the data </code></pre> <p>Joiner’s list suggests a number of solutions:</p> <ul> <li>Make it difficult to distort the system. <ul> <li>Make it difficult to distort the data, and</li> <li>Give people the slack necessary to improve the system (a tricky thing to do, which we’ve covered elsewhere).</li> </ul> </li> </ul> <p>The third point is really important. Preventing distortions is just one half of the solution. Avoiding Goodhart’s Law requires you to also give people the space to improve the system."</p> </li> </ul> <p><a href="https://newsletter.pragmaticengineer.com/">Pragmatic Engineer</a></p> <p><a href="https://projects.propublica.org/claimfile/">ProPublica</a>: Find out why your health insurer denied your claim. ProPublica’s Claim File Helper lets you customize a letter requesting the notes and documents your insurer used when deciding to deny you coverage. Get your claim file before submitting an appeal.</p> <p><a href="https://theorg.com/">TheOrg</a>: Find the right person to contact in any organization. (Commercial?)</p> <h3>Articles/Blogs</h3> <p><a href="https://apenwarr.ca/log/20230605">Tech debt metaphor maximalism</a>: "I really like the "tech debt" metaphor. A lot of people don't, but I think that's because they either don't extend the metaphor far enough, or because they don't properly understand financial debt." <em>Really</em> good read on how to think about tech debt as a direct extension of financial debt.</p> <p><a href="https://www.linkedin.com/posts/weskao_most-companies-suck-at-communicating-the-activity-6934551400383201280-ONcW">12 Ways to Show Your Value</a>:</p> <ol> <li> <p><strong>Pour salt on the wound</strong>: No pain, no sale. People will only value your solution if they value their problem. If they don’t think their problem is that bad, they won’t pay you to solve it. When in doubt, emphasize the pain. Painkillers sell better than vitamins.</p> </li> <li> <p><strong>Use before-and-after</strong>: An effective before-and-after skips all the boring parts. It’s like a jump cut in a movie. It emphasizes <em>contrast</em>. This heightens drama and interest. The best before-and-afters make you say wow.</p> </li> <li> <p><strong>Don’t jump right to the solution</strong>: Instead, explain how much $ the problem costs and how it’s making your customer’s life hard. Then talk about your product &amp; how you’ll make their problem go away.</p> </li> <li> <p><strong>Show, don’t tell</strong>: I came across AI-driven software that reduces background noise on Zoom calls. Their website has a toggle on/off button where you can hear a sound sample without vs with their tool. That demo was worth 10,000 words.</p> </li> <li> <p><strong>Increase desire, don’t just decrease friction</strong>: People jump through hoops for things they want. Did you pick your spouse because they were the most convenient option? Of course not. You picked them despite obstacles.</p> </li> <li> <p><strong>Explain "why now?"</strong>: Most folks AGREE that your product is great--but they still don’t buy. What they’re really thinking: “Do I need this enough to pay $ right now? Or can I do this later?" Later often means never.</p> </li> <li> <p><strong>Be/Do/Have</strong>: This is a classic framework in leadership coaching but applies to marketing. What does your product let your customer <em>be</em> that they weren’t before? What can they <em>do</em> now that they couldn’t do before? What do they <em>have</em> now that they didn’t have before?</p> </li> <li> <p><strong>Aim for “no-brainer status"</strong>: You need to aim for an instant yes. Not a yes after 100+ touchpoints of nurturing. Think about what would get that reaction. You likely won’t get an instant yes. But if you aim for it, it’ll speed up getting there.</p> </li> <li> <p><strong>Do what makes their eyes light up (ELU)</strong>: You can tell when people listen to be polite. But there are moments when aliveness flashes in their eyes--you just said something they care about. Watch for ELU. It's data on what to do more of.</p> </li> <li> <p><strong>Avoid backstory scope creep</strong>: It's easy to get carried away sharing background info. Remove 90% of it. You need less backstory than you think.</p> </li> <li> <p><strong>Focus on what your customer wants to hear, not what you want to talk about</strong>: This is an easy way to trim from your pitch.</p> </li> <li> <p><strong>Consider overhauling</strong>: If you’re struggling to describe your product’s value, figure out if the problem is (A) the marketing or (B) the offer itself. If your underlying offer isn’t good, no amount of improving the marketing will help.</p> </li> </ol> <p><a href="https://newsletter.pragmaticengineer.com/p/the-scoop-fast">"The Scoop: Inside Fast's Rapid Collapse"</a>:</p> <ul> <li>a well-funded scaleup could be closer to bankruptcy than it seems <ul> <li> <p>How was Fast able to hire hundreds of people of this caliber? Money, storytelling, and equity:</p> <ul> <li> <p>Fast paid very strong base salaries, told a great story, and offered lavish equity. Most engineers joining didn't know much about why the one-click checkout industry has the potential of billions. What they did know is Fast offered at or above base salaries of Big Tech, directors shared an enthusiastic story of the company being a rocketship, and they were granted equity which seemed like a golden ticket to retirement.</p> </li> <li> <p>Fast base salaries were top of the market. For senior software engineers, Fast offered $200-240K/year in base salaries with full remote work. They offered this range not just in the US, but outside it, when hiring in Europe, converting the base salary to Euros (€180-€220K/year).</p> </li> <li> <p>Large sign-on bonuses were common for engineers. Those who asked for it all received sign-on bonuses of $20-50K as a one-off payment. These types of sign-on bonuses are not uncommon at Big Tech but are much more rare at startups that are unprofitable. These large sign-on bonuses added to the feeling that Fast is doing well financially. Sign-on bonuses are typically repayable when the employee quits the company within 12 months. In some ways, the sign-on bonuses made the layoffs more bearable for those who joined recently, as they can keep the full amount.</p> </li> <li> <p>Equity issued was lavish and presented as potentially life-changing. Senior software engineers received anywhere from 15,000 to 80,000 stock options, vesting over 4 years, with a 1-year cliff.</p> </li> </ul> </li> <li> <p>Visualizing the potential stock upside was a major reason many candidates chose Fast over Big Tech offers where the stock was liquid.</p> </li> <li> <p>Exploding offers because of “the inevitability of the Series C closing” was used as a tactic to close candidates at the end of 2021.</p> </li> <li> <p>Warning signs within the company:</p> <ul> <li>A hockey stick of growth presented across the company - but with the Y-axis being employees.</li> <li>Tiny daily sales numbers that leadership and senior employees received</li> <li>Fast did less than $300K worth of sales and below $6K in revenue on most days</li> <li>Not having large clients on their platform</li> <li>Sales vs engineering and sales winning. I have reports of engineering directly raising concerns to the CEO, and suggesting to focus on larger customers, fewer customizations, and bring in more revenue. Sales, however, wanted the opposite: close many deals and hit their targets of signups. In the end, sales got their way, and the strategy of many small merchants stayed in place.</li> <li>The new CFO instituting an immediate hiring freeze</li> <li>A warning sign could have been the annual burn rate just for engineering - for anyone who would have done the math. Most software engineers at Fast were at the senior or above level, making $200K/year or more. Fast had about 150 engineers by early 2022. The burn rate of just the base salaries for engineers would have been around $30M/year (150 x $200,000), not counting sign-on bonuses. Engineering was a third of the company’s headcount, though likely a more expensive third. With this math, it’s not hard to understand how Fast burned through its $100M investment in just over a year.</li> <li>A cofounder going radio silent starting in March was another warning sign.</li> </ul> </li> <li> <p>Advice when joining startups</p> <ul> <li>Do your research on the company and founders. Do your homework on the history of the company, and past red flags. I did this when I got a reachout to potentially interview for Fast in the second half of 2020. I found some unsettling information about the founder of Fast, Dominic Holland. From his previous startup going bust, him firing staff over a text and threatening to sell sensitive personal records. In the early days of Fast, he terminated contract software engineers in Africa with no justification after the raised funding off the prototype they built, never acknowledging their work.</li> <li>Ask for numbers. How much runway does the company have left? What is the burn per month? What is the revenue of the company, and what is the spending? Especially when joining at senior levels - Staff engineer or above, Director or above - it should be a red flag if people don’t share these numbers with you, at least verbally. If you get pushback, you can always use the Fast story as reasoning on why you want to know these numbers.</li> <li>Ask if there is a clear set of key business metrics are and if engineers have access to these close to real-time. Is it the number of users? Revenue? Signups? Percentage of returning daily users? If the answer is “no” to either of the questions, treat it as a red flag. In the case of Fast, these metrics were likely not defined, and the data was not shared with all engineers.</li> <li>Reverse interview your future manager and a founder. Once you have an offer, ask to talk with your future manager and a founder. Collect all your doubts and questions, and ask them on the spot.</li> <li>Ask to talk with lead investors. Investors are almost always happy to jump on a call to talk with potential candidates. Expect that this will mostly be a sell call, where the investors share why they invested in the company. Still, you might find out information.</li> <li>Talk with people who left. Try to find people who left the startup, and who were in a similar position to what you are applying for. Message them on LinkedIn, and ask if they can share why they left. A surprisingly large number of them tend to reply.</li> <li>For executive positions: reach out to VCs. If you are joining as a Director or above, do backchannel checks through the VC network. If you already have VC connections, ask if they can get you information on the startup. If you don’t yet have VC connections, try reaching out to VCs on Twitter or LinkedIn who work at some of the well-known funds I list in the article Finding the next company to work at.</li> <li>Make a plan if the stock grant is worthless.</li> <li>Know that risk and reward are often connected.</li> <li>Not every startup operates like Fast, but many startups face the same types of problems that Fast had.</li> </ul> </li> </ul> </li> </ul> https://tedneward.github.io/Research/reading/business/meeting-the-challenge/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/meeting-the-challenge/index.html <p><em>(by Clayton M Christensen and Michael Overdorf, from "HBR's 10 Must Reads: The Essentials")</em></p> <p>Why do so few established companies innovate successfully? When a new venture captures the established firms' imagination, they get their people working on it within organizational structures (such as functional teams) designed to surmount old challenges--not ones that the new venture is facing. To avoid this mistake, ask:</p> <ul> <li> <p><strong>"Does my organization have the right resources to support this innovation?"</strong> Resources supporting business-as-usual (people, technologies, product designs, brands, customer and supplier relationships) rarely match those required for new ventures.</p> </li> <li> <p><strong>"Does my organization have the right processes to innovate?"</strong> Processes supporting your established business (decision-making protocols, coordination patterns) may hamstring the new venture.</p> </li> <li> <p><strong>"Does my organization have the right values to innovate?"</strong> Consider how you decide whether to commit to a new venture. For example, can you tolerate lower profit margins than your established enterprise demands?</p> </li> <li> <p><strong>"What team and structure will best support our innovation effort?"</strong> Should you use a team dedicated to the project within your company? Create a separate spin-off organization?</p> </li> </ul> <p>By selecting the right team and organizational structure for your innovation--and infusing it with the right resources, processes, and values--you heighten your chances of innovating successfully.</p> <p><strong>Selecting the Right Structure for Your Innovation</strong></p> <table> <thead> <tr> <th>If your innovation... </th> <th> Select this type of team... </th> <th> To operate... </th> <th> Because...</th> </tr> </thead> <tbody> <tr> <td>Fits <em>well</em> with your existing values and processes </td> <td> <strong>Functional teams</strong> who work sequentially on issues, or <strong>lightweight teams</strong>--ad-hoc cross-functional teams who work simultaneously on multiple issues </td> <td> Within your existing organization </td> <td> Owing to the good fit with existing processes and values, no new capabilities or organizational structures are called for.</td> </tr> <tr> <td>Fits <em>well</em> with existing values but <em>poorly</em> with existing processes </td> <td> <strong>Heavyweight team</strong> dedicated exclusively to the innovation project, with complete responsibility for its success </td> <td> Within your existing organization </td> <td> The poor fit with existing processes requires new types of coordination among groups and individuals.</td> </tr> <tr> <td>Fits <em>poorly</em> with existing values but <em>well</em> with existing processes </td> <td> <strong>Heavyweight team</strong> dedicated exclusively to the innovation project, with complete responsibility for its success </td> <td> Within your existing organization for development, followed by a spin-off for commercialization </td> <td> In-house development capitalizes on existing processes. A spin-off for the commercialization phase facilitates new values--such as a different cost structure with lower profit margins.</td> </tr> <tr> <td>Fits <em>poorly</em> with your existing processes and values </td> <td> <strong>Heavyweight team</strong> dedicated exclusively to the innovation project, with complete responsibility for its success </td> <td> In a separate spin-off or acquired organization </td> <td> A spin-off enables the project to be governed by different values and ensures that new processes emerge.</td> </tr> </tbody> </table> <h3>In more depth</h3> <p>What managers lack is a habit of thinking about their organization's capabilities as carefully as they think about individual people's capabilities.</p> <p>Where capabilities reside: its resources, its processes, and its values.</p> <ul> <li><strong>Resources</strong>: (tangible) people, equipment, technologies, cash, (less tangible) product designs, information, brands, relationships with suppliers/distributors/customers.</li> <li><strong>Processes</strong>: patterns of interaction, coordination, communication, and decision-making employees use. Processes, by their very nature, are set up so that employees perform tasks in a consistent way time after time--they are meant not to change. When people use a process to do the task it was designed for, the process will perform well; when using it for a very different task, it is likely to perform sluggishly. The most important capabilities and concurrent disabilities aren't necessarily embodied in the most visible processes; they are more likely to be in the less visible, background processes that support decisions about where to invest resources--those that define how market research is habitually done, how such analysis is translated into financial projections, how plans and budgets are negotiated internally, and so on. It is in those processes that many organizations' most serious disabilities in coping with change reside.</li> <li><strong>Values</strong>: An organization's values are the standards by which employees set priorities that enable them to judge whether an order is attractive or unattractive/whether a customer is more important or less important/whether an idea for a new product is attractive or marginal/etc. Prioritization decisions are made by employees at every level. The larger/more complex a company becomes, the more important it is for senior managers to train employees throughout the organization to make independent decisions about priorities that are consistent with the strategic direction and the business model of the company. A key metric is whether such clear, consistent values have permeated the organization.</li> </ul> <p>We want to focus on two sets of values in particular that tend to evolve in most companies in very predictable ways. The inexorable evolution of these two values is what makes companies progressively less capable of addressing disruptive change successfully.</p> <ul> <li>The way the company judges acceptable gross margins.</li> <li>How big a business opportunity has to be before it can be interesting. It follows that an opportunity that excites a small company isn't big enough to be interesting to a large company. One of the bittersweet results of success, in fact, is that as companies become large, they lose the ability to enter into small, emerging markets. This disability is not caused by a change in resources within the companies--rather it's caused by an evolution in values.</li> </ul> <h3>The Migration of Capabilities</h3> <p>In the early stages of an organization, much of what gets done is attributable to resources--people, in particular. The addition/departure of a few key people can profoundly influence its success. Over time, the locus of the organization's capabilities shifts towards its processes and values. (One reason that many soaring young companies flame out after an IPO based on a single hot product is that their initial success is grounded in resources--often the founding engineers--and they fail to develop processes that can create a sequence of hot products.) At some successful companies, the processes and values have become so powerful that it almost doesn't matter which people get assigned to which project teams--the company is able to crank out high-quality work year after year because its core capabilities are rooted in its processes and values rather than its resources.</p> <p>The founder has early influence, and employees experience for themselves the validity of the founder's problem-solving and decision-making methods. Thus processes become defined. Likewise, if the company becomes financially successful by allocating resources according to criteria that reflect the founder's priorities, the company's values coalesce around those criteria. (From here,these processes and values become enshrined as culture, enabling employees to act autonomously but causes them to act consistently.) As long as the organization continues to face the same sorts of problems that its processes and values were designed to address, managing the organization can be straightforward. But because those factos also define what an organization cannot do, they consititute disabilities when the problems facing the company change fundamentally.</p> <h3>Sustaining vs Disruptive Innovation</h3> <p>Sustaining innovations are evolutionary changes in markets, and successful companies are pretty good at responding to them. Disruptive innovations are those that are revolutionary, that create an entirely new market through the introduction of a new kind of product or service, one that's actually worse, as judged by the performance metrics that mainstream customers value.</p> <p>Sustaining innovations are nearly always developed and introduced by established industry leaders. Those same companies never introduce--or cope well with--disruptive innovations, because the evolution from people to processes and values means that the organization has (through repeated execution) optimized for solving a particular kind of problem--and disruptive innovations are, by definition, not that particular kind of problem. Large companies often surrender emerging growth markets, and smaller, disruptive companies are actually more capable of pursuing them. Startups lack resources, but that doesn't matter--their values can embrace small markets and their cost structures can embrace low margins. Their market research and resource allocation processes allow managers to proceed intuitively; every decision need not be backed by careful research and analysis. All these advantages add up to the ability to embrace and even initiate disruptive change. But how can a large company develop these capabilities?</p> <h3>Creating Capabilities to Cope with Change</h3> <p>Processes are not nearly as flexible or adaptable as resources are--and values are even less so. So when an org needs new processes and values--because it needs new capabilities--managers must create new organizational space where those capabilities can be developed. There are three ways to do that:</p> <ul> <li> <p>create new organizational structures within corporate boundaries in which new processes can be developed.</p> </li> <li> <p>spin out an independent organization from the existing organization and develop within it the new processes and values required to solve the new problem.</p> </li> <li> <p>acquire a different organization whose processes and values close match the requirements of the new task.</p> </li> </ul> https://tedneward.github.io/Research/reading/business/product-roadmaps/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/product-roadmaps/index.html <p><em>(by Lombardo, McCarthy, Ryan, Connors)</em></p> <h3>Chapter 1: Relaunching Roadmaps</h3> <p>Roadmap is a strategic communication tool, a statement of intent and direction; describes how you intend to achieve your product vision. It focuses on the value you propose to deliver for your customer and your organization in order to rally support and coordinate effort among stakeholders.</p> <p>Product: how you deliver value to your organization's customers.</p> <p>Stakeholder: all the internal and external colleagues and partners who are involved with the product being developed, marketed, sold, and serviced.</p> <p>Customer: the recipient of the value your product provides. Buyer and user are often the same, but not always.</p> <p>Product roadmap should:</p> <ul> <li>Put the organization's plans in a strategic context <ul> <li>Problem: Nobody understands why things are on the roadmap</li> <li>Solution: Tie the roadmap to a compelling vision of the future</li> </ul> </li> <li>Focus on delivering value to customers and the organization <ul> <li>Problem: You are shipping a lot but not making progress</li> <li>Solution: Focus the roadmap on delivering value</li> </ul> </li> <li>Embrace learning as part of a successful product development process <ul> <li>Problem: Executives and customers demand commitments</li> <li>Solution: Commit to outcomes rather than output</li> </ul> </li> <li>Rally the organization around a single set of priorities <ul> <li>Problem: Marketing and sales are not selling what you are making</li> <li>Solution: Align everyone around common goals and priorities</li> </ul> </li> <li>Get customers excited about the product's direction <ul> <li>Problem: Customers aren't excited about your new features</li> <li>Solution: Use the roadmap to reality-check your direction with customers</li> </ul> </li> </ul> <p>Product roadmap should NOT:</p> <ul> <li>Make promises a team cannot deliver <ul> <li>Problem: Your stakeholders and customers expect a firm commitment on dates for your product releases</li> <li>Solution: Prioritization is critical to deliver on your commitments</li> </ul> </li> <li>Require wasteful up-front design and estimation <ul> <li>Problem: Time spent estimating design and development efforts takes time away from actually implementing them</li> <li>Solution: Let the teams determine the solutions, and allow them to solve the problem</li> </ul> </li> <li>Should not be confused with a release plan or a project plan <ul> <li>Problem: Your team looks at the roadmap as if it were a project plan when features will be released</li> <li>Solution: Commit to outcomes rather than output</li> </ul> </li> </ul> <h3>Chapter 2: Components of a Roadmap</h3> <p>Primary components:<br> * Product vision is your guiding principle<br> * Business objectives help to measure progress<br> * Themes focus on outcomes rather than output; "What would need to be true for our product to realize its vision and attain its business objectives?"<br> * Broad timeframes avoid overcommitment<br> * Disclaimer protects you (and your customer)</p> <p>Secondary components to answer the concerns of certain stakeholders:<br> * Features and solutions show how you intend to deliver on your themes<br> * Confidence<br> * Stage of development<br> * Target customers (if your product serves more than one customer)<br> * Product areas</p> <p>Complementary information to provide context for your roadmap:<br> * Project information<br> * Platform considerations<br> * Financial context<br> * External drivers</p> <h3>Chapter 3: Gathering Inputs</h3> <h3>Chapter 4: Establishing the Why with Product Vision and Strategy</h3> <p>Mission defines your intent. Four key elements: value (what value does your mission bring to the world), inspiration (how does your mission inspire your team to make the vision a reality), plausibility (is your mission realistic and achievable), and specificity (is your mission specific to your business, industry and/or sector).</p> <p>Vision is the outcome you seek.</p> <p>Values are beliefs and ideals.</p> <p>Product vision: Why your product exists.</p> <p>Value proposition template:</p> <p>For: (target customer)</p> <p>Who: (target customer's needs)</p> <p>The: (product name)</p> <p>Is a: (product category)</p> <p>That: (product benefit/reason to buy)</p> <p>Unlike: (competitors)</p> <p>Our product: (differentiator)</p> <p>Supports our: (objective(s))</p> <p>Universal business objectives:</p> <ul> <li>Sustainable value:<br> * Support the product's core value<br> * Create barriers to competition <ul> <li>Growth <ul> <li>Grow market share</li> <li>Fulfill more demand</li> <li>Develop new markets</li> <li>Improve recurring revenue</li> </ul> </li> <li>Profit <ul> <li>Support higher prices</li> <li>Improve lifetime value</li> <li>Lower costs</li> <li>Leverage existing assets</li> </ul> </li> </ul> </li> </ul> <h3>Chapter 5: Uncovering Customer Needs Through Themes</h3> <p>Identifying customer needs is the most important aspect of your roadmapping process. Draw a clear distinction between the product roadmap and the release plan. Themes are an organizational construct for defining what's important to your customers at the present time. A theme is a high-level customer need. A subtheme is a more specific need. Themes are about outcomes, not outputs.</p> <p>Opportunity-Solution Trees (<a href="https://www.producttalk.org/opportunity-solution-tree">https://www.producttalk.org/opportunity-solution-tree</a>)</p> <p>Expected Outcome (Objective)<br> * Opportunity (Themes)<br> * Opportunity<br> * Opportunity<br> * Solution (Features)<br> * Solution<br> * Experiment<br> * Experiment</p> <p>Job stories (similar to user stories):<br> When (situation/motivation)<br> I need (desire)<br> So I can (result)</p> <p>Themes:<br> Ensure (result) for (stakeholder)</p> <p>Tie the themes back to your strategic objective. (Product vision -&gt; Objectives -&gt; Themes and subthemes)</p> <h3>Chapter 6: Deepening Your Roadmap</h3> <p>Add secondary information to flesh out the picture.</p> <p>Features and Solutions: How They Can Work with Themes<br> * Probable solutions (with a relatively high degree of confidence)<br> * Infrastructure solutions<br> * Carryover solutions</p> <p>Stages of Development: discovery, market research, R&amp;D, design, testing, prototyping, development, pre-production, alpha, beta, early access, and so on. Spotify uses "think it", "ship it" or "tweak it".</p> <p>Confidence: Nothing should ever be 100% confident. (0.2 to 0.8 confidence levels)</p> <p>Identifying Target Customers.</p> <p>Strive for Balance--don't bury the roadmap in secondary details.</p> <h3>Chapter 7: Prioritizing--with Science!</h3> <p>Why Prioritization is Crucial<br> * Opportunity Cost<br> * Shiny Object Syndrome<br> * Exponential Test Matrix Growth<br> Bad Ways to Prioritize<br> * Your (or someone else's) gut<br> * Analyst opinions<br> * Popularity<br> * Sales requests<br> * Support requests<br> * Competitive me-too features<br> Prioritization frameworks<br> * Critical path: What is the one thing (or set of things) our solution needs to get right?<br> * Use to: Identify the "one thing" that will drive a customer to buy<br> * Choose When: Designing an MVP or making a major expansion in product scope<br> * Downsides: Does not take into account effort, risk, or business goals; does not rank needs finer than "critical" or "noncritical"<br> * Kano (from Dr Noriaki Kano): classifying customer expectations into three broad categories: expected needs, normal needs, and exciting needs. Expected needs not met lead to dissatisfiers. Normal needs are satisfiers. Exciting needs are delighters and go beyond the customer's experience.<br> * Use to: Understand how customers perceive relative value<br> * Choose When: Identifying possible add-ons or enhancements<br> * Downsides: Does not take into account effort, risk or business goals<br> * Desirability (Do customers want it?), Feasibility (Can we--and how easily can we--build it?), Viability (Can we make money on it?); use a 1 (low), 2 (medium), 3 (high) scale for each possibility<br> * Use to: Identify opportunities that meet all key criteria for success<br> * Choose When: Prioritizing a relatively small set of initiatives or solutions to a particular problem<br> * Downsides: Categories are not clearly defined in terms of customer needs, organization goals, or different types of effort or risk<br> * ROI scorecard: Value/Effort = Priority<br> * Strategy defines value; Value = Customer's Needs + Organization's Goals<br> * Deliberate imprecision; be wary of complexity here<br> * Estimating effort; don't scare engineers away; T-shirt sizing<br> * Think cross-functionally<br> * Risks and unknowns<br> * Full Formula: ((CN1 + CN2 + CNn + … + BO1 + BO2 + BOn…) / (E1 + E2 + En…)) x C = P<br> § CN: Customer's Need<br> § BO: Business Objective<br> § E: Effort<br> § C: Confidence<br> § P: Priority<br> * Use to: Rank ideas according to return-on-investment criteria<br> * Choose When: Weighing multiple factors and/or a long list of possible initiatives, problems to solve, features, or solutions<br> * Downsides: More complex model requires alignment on different components of value and effort<br> * MoSCoW method: Must-have, Should-have, Could-have, Won't-have<br> * Use to: Communicate launch criteria<br> * Choose When: Feeling uncertain about what must be included in a product/service/release<br> * Downsides: Does not help set priorities, only communicate them</p> <h3>Chapter 8: Achieving Alignment and Buy-in</h3> <p>Alignment (concerted effort to help people understand the issues and what their respective roles are), consensus (a group of people reaching a mutually agreed-upon decision; in practice, a potential for a passive-aggressive mess), collaboration (individuals cooperate to accomplish a common goal or outcome; individuals work together for a shared purpose--they may not concur on everything each step of the way, but they do agree on the final outcome).</p> <p>Shuttle diplomacy: meeting with each party individually to reach decisions that require compromise and trade-offs. Politics (and strong voices drowning out quieter ones) can ruin attempts at alignment--much more manageable in one-on-one meetings. You can focus the conversation on common goals if it's just you and them, with nobody else in the room for them to answer to or try to impress. Entice stakeholders with a draft of your roadmap and ask for their input; always tie the meeting to goals and objectives. Use GROW (Goals/Reality/Options/Way forward) to guide the conversation.</p> <p>Co-creation workshop. Needs to have clear plan and outcomes before you hold it. Outcome is to finalize the roadmap for the upcoming period. Loose agenda:<br> * Intro and rules (5 min)<br> * Hopes and fears (5 min): each individual writes their hopes for the product's future on one color of Post-It/index card, and fears on another color. Objective is to draw out everyone's emotional aspirations and fears and see how they align with each other and with the established business objectives.<br> * Vision and goals (10 min): articulate the product vision (if it's not already determined). Follow this prompt: "A world where the (target customer) no longer suffers from the (identified problem) because of (product differentiator) they (benefit)." Pare this down to: "To (benefit realized) by (product differentiator)."<br> * Back-plan (15 min): start with the end result of the cover story (PR/FAQ) and work gradually backward to the present. Each step backward must be realistic, so teams are forced to think through all the different changes necessary.<br> * Sizing and prioritizing (40 min): Example: spend $100 on particular customer needs.<br> * Wrap-up (10 min)</p> <h3>Chapter 9: Presenting and Sharing Your Roadmap</h3> <p>(Why, risks, different types of presentations to different kinds of audiences, etc)</p> <h3>Chapter 10: Keeping It Fresh</h3> <p>(When, where and how to incorporate change into the roadmap)</p> <h3>Chapter 11: Relaunching Roadmaps in Your Organization</h3> <p>(Steps for re-launching roadmaps into an organization that has gotten away from them)</p> https://tedneward.github.io/Research/reading/business/art-of-the-start/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/art-of-the-start/index.html <p><em>(by Guy Kawasaki, ISBN ...)</em></p> <h3>Chapter 1: The Art of Starting</h3> <ul> <li>pg 8: Create a mantra for your employees as a guideline; a tagline is for your customers.</li> <li>pg 9: Don't write up a business plan, craft a pitch, or start financial projections until you have a prototype.</li> <li>pg 13: Don't revise your product to get prospective customers to love it; revise it because customers already love it.</li> <li>pg 14: Be sure you can describe your business model in ten words or less, and don't be afraid to copy one.</li> <li>pg 16: Important milestones are prove a concept, complete design specifications, build a prototype, raise capital, ship a testable version, ship a final version, and achieve break-even.</li> <li>pg 17: List the assumptions about the business, and link them to the milestones above so that you can test them as you reach milestones.</li> <li>pg 24: Never admit you're scared to other employees -- a CEO can never have a bad day.</li> <li>pg 24: There is much more to gain by freely discussing your idea than there is to lose.</li> <li>pg 24: If three close friends tell you to give up, you should listen.</li> <li>pg 25: Make a web site, business cards, and a letterhead immediately for customers, partners, and investors to look at.</li> </ul> <h3>Chapter 2: The Art of Positioning</h3> <ul> <li>pg 29: Positioning states why an organization was started, why customers should patronize it, and why good people should work at it.</li> <li>pg 31: Ensure that your positioning is long-lasting, and that it does not sound like your competitors'.</li> <li>pg 33: Target a small niche -- from there you can establish a beachhead and expand your companies into different areas.</li> <li>pg 35: When choosing a company name, avoid numbers, try for verb potential, ensure it sounds like nothing else, and it should match what you do.</li> <li>pg 38: Make your positioning personal so a potential customer can clearly see how your product fills a need.</li> <li>pg 40: Avoid overfamiliar adjectives and find unique language or offer scientific proof points to describe your product.</li> <li>pg 41: New hires can provide a rich vein for a positioning statement because it's from a fresh, outward-facing perspective.</li> <li>pg 41: Communicating your positioning is every employee's job; provide a refresh every six months.</li> <li>pg 43: Don't lie when projecting your image, but don't go out of your way to look undercapitalized, premature, and weak.</li> </ul> <h3>Chapter 3: The Art of Pitching</h3> <ul> <li>pg 45: Explain what your organization does in the first minute; don't make it an auto-biographical narrative.</li> <li>pg 47: Before pitching, learn from your sponsor what the audience would like to learn, and what attracted them to the idea and gave them an opportunity to meet.</li> <li>pg 48: Brainstorm to find connections, hooks, and angles to make the pitch more powerful and meaningful to your audience.</li> <li>pg 49: The pitch is to simulate interest, not to close a deal, so only have ten slides or so; don't include a stupid slide about liquidity.</li> <li>pg 50: Be able to give your pitch in 20 minutes, to allow time for discussion, or in case the previous meeting ran late.</li> <li>pg 56: Get there early to make sure everything works, but bring your own projector, laptop, and printouts of the presentation just in case.</li> <li>pg 56: At the beginning of the pitch, ask how much time you have, and if there is anything in particular they would like to know.</li> <li>pg 59: If bold, catalyze fantasy, where you provide a line of reasoning that says everyone needs your product, and market size numbers are irrelevant.</li> <li>pg 61: Have someone take notes at the pitch, and play it back to make sure you got the correct information. Then follow through, and quickly.</li> <li>pg 62: Patch your pitch after each meeting in response to the most recent questions and objections, but throw away your presentation after ten or so pitches.</li> <li>pg 63: Use a dark background with sans serif fonts, with your logo on every slide, and with diagrams and graphs. Never nest bullets, and reveal them one at a time.</li> <li>pg 65: Don't send your presentation ahead of time; if you hand it out at the start of the meeting, ask them not to skip ahead.</li> </ul> <h3>Chapter 4: The Art of Writing a Business Plan</h3> <ul> <li>pg 68: A business plan won't win over investors, but it's a necessity; use it to find holes in your strategy and in your team.</li> <li>pg 68: The plan is a detailed vision of a pitch, so only begin working it once you have the pitch right.</li> <li>pg 70: The executive summary, which takes the place of the title slide in your pitch, is the most important part of the plan.</li> <li>pg 70: Don't let it exceed 20 pages, include the key metrics, simplify financial projections to two pages, and state the assumptions behind those projections.</li> <li>pg 71: Investors require five years of projections to help them understand your scale, the capital required, and consider your inherent assumptions.</li> <li>pg 73: Write as if you know exactly what the future holds, but react opportunistically when you encounter reality.</li> <li>pg 74: To make your plan stand out, provide a list of customers the reader can call to discuss how much they need or already love your service.</li> </ul> <h3>Chapter 5: The Art of Bootstrapping</h3> <ul> <li>pg 80: Managing for cash flow means passing up sales that take a long time to collect, and stretching out payments on expenses.</li> <li>pg 83: Don't worry about testing; if you would let your mother or father use the product or service in its current state, ship it.</li> <li>pg 84: When building a team, focus on affordability; young, unproven people don't know what they don't know, so they're willing to try anything.</li> <li>pg 87: When choosing service providers, hire the right people with good references, even if that means paying more, but always negotiate.</li> <li>pg 90: Using a reseller isolates you from customers, requires volume to compensate for decreased profit margins, and extends time to delivery.</li> <li>pg 91: By positioning yourself against a market leader, you can utilize their brand awareness to save marketing, promotion, and advertising dollars.</li> <li>pg 93: You need someone grounded with ten years of operating experience; typically a CFO, COO, controller, or accountant who can make hard decisions like firing.</li> <li>pg 94: Intentionally understaff to bootstrap; this may slow your ascent, but its' better than laying people off or running out of money.</li> <li>pg 95: A great board of directors can provide guidance and help you get money, but can't be hard to build without funding in the first place.</li> <li>pg 97: Set and communicate goals, designate people responsible for them, measure the execution of them every 30 days, and praise execution.</li> </ul> <h3>Chapter 6: The Art of Recruiting</h3> <ul> <li>pg 101: The CEO must hire a management team that is better than he is; the management team must hire employees better than it is.</li> <li>pg 102: If you're not getting a reference that's superlative, you're getting a negative one.</li> <li>pg 103: A startup needs kamikazes who work long hours, implementers who turn their work into infrastructure, and operators who upkeep the infrastructure.</li> <li>pg 106: Ignore functional weaknesses; be attracted to candidates with major strengths than those who lack major weaknesses.</li> <li>pg 107: Ask the candidate who all his important decision makers are, and then work with the candidate to answer their concerns too.</li> <li>pg 111: Check references early; checking after the interview will lead you to only hear comments that affirm your decision.</li> <li>pg 113: Establish an initial review period where after 90 days you go over what's going right, what's going wrong, and how to improve performance.</li> <li>pg 114: Reference checking is not to disqualify a candidate, but to look for consistency in how he represented himself.</li> <li>pg 116: Don't recruit to make an investory happy; instead, recruit to build a great organization.</li> </ul> <h3>Chapter 7: The Art of Raising Capital</h3> <ul> <li>pg 121: Current investors, other entrepreneurs, and professors can serve as a credible third party to get you an introduction.</li> <li>pg 124: Your company is flawed if you have friends or roomates in CXO positions, or former employees claiming your technology belongs to them.</li> <li>pg 126: Openly acknowledge your competitors and evaluate your strengths and weaknesses; if needed, zoom out until you have competitors.</li> <li>pg 128: Don't tell investors about deals that will be signed to bring in revenue. Only talk about deals after they're done.</li> <li>pg 129: Investors all know one another, so don't try to play them against one another.</li> <li>pg 130: The optimal number of times to mention your technology is patentable is one; zero is bad because then you have nothing proprietary.</li> <li>pg 134: Courting one investor will make it easier to court others.</li> <li>pg 134: To help court an investor, check-in with them afterward and show improvement and progress after they don't commit, but don't nag.</li> <li>pg 135: When you take outside money, you're obligated to all shareholders even if they own a minority position.</li> <li>pg 138: You need people with two kinds of experience on your board: company-building and deep market knowledge.</li> <li>pg 139: Send reports to your board two days before a meeting, so you spend little time communicating the facts and more time trying to improve upon them.</li> <li>pg 140: When you have bad news or need to make a key decision, meet privately with each board member in advance.</li> <li>pg 143: A law firm recognized for its finance/VC work shows what you're doing, and will assist with the paperwork of a financing.</li> <li>pg 144: More investors means more people helping you, multiple sources of capital, and can give you allies if you disagree with one investor.</li> <li>pg 146: When a VC offers a valuation, ask for a 25 percent higher valuation because you're expected to push back.</li> <li>pg 147: Freely circulate your executive summary and PowerPoint pitch; only ask for an NDA if an investor wants to learn more at the bits-and-bytes level.</li> <li>pg 148: Don't be intimidated by your board of directors; instead, give them assignments and hold them accountable.</li> </ul> <h3>Chapter 8: The Art of Partnering</h3> <ul> <li>pg 152: Never form a partnership to make the press happy or silence critics; do it only to increase revenue.</li> <li>pg 153: When you form a partnership, always define deliverables and objectives.</li> <li>pg 155: A partnership requires an internal champion at each company who is empowered and whose sole goal is to make the partnership a success.</li> <li>pg 156: Form partnerships that accentuate the strengths of each partner, rather than covering their weaknesses.</li> <li>pg 158: Never use a legal document as a straw man to get the discussion rolling; discuss on the whiteboard, define a framework, then draft a document.</li> <li>pg 159: Add an "out" clause within some defined number of days to ensure both parties won't be trapped in an untenable predicament.</li> <li>pg 161: When schmoozing, ask questions and then shut up; follow up quickly while making it easy to get in touch; give favors, return them, and ask for the return of favors.</li> <li>pg 165: Ensure your emails have a signature, containing your name, organization, postal address, phone and fax numbers, and e-mail and web addresses.</li> </ul> <h3>Chapter 9: The Art of Branding</h3> <ul> <li>pg 171: Design a product that delivers basic functionality without requiring the user to refer to an instruction manual. Test it on your parents.</li> <li>pg 172: Make it easy to switch from competitors to your product, but also make it easy to switch away; making it difficult will scare potential customers away.</li> <li>pg 174: From your customers, recruit evangelists. Assign tasks, keep communication open, give them the required tools, respond to their desires, and reward them.</li> <li>pg 178: Per dollar, building a community is the cheapest way to create and maintain a brand, so don't wait for a community to form on its own.</li> <li>pg 178: To achieve humanness, try to target the young, make fun of yourself, and feature your customers. Don't be impersonal.</li> <li>pg 180: Press coverage doesn't convert readers to customers; instead, lower the barrier to entry, get customers, and then the press will write.</li> <li>pg 181: No matter what the weather, maintain good relations and contact with the press. You establish your credibility when times are bad, so don't lie.</li> <li>pg 185: If given the opportunity to give a speech, cut the sales propaganda, don't denigrate the competition, and just say something interesting.</li> <li>pg 186: When on a panel, be entertaining, never look bored, and don't look at the moderator but the audience instead.</li> <li>pg 189: You might insult an evangelist by trying to pay them; instead, make your product better, offer them information, and honor them publicly.</li> <li>pg 190: If you want to change your logo, look and feel, mantra, or tag line, consider that it may only be now that they're entering the public's mind.</li> </ul> <h3>Chapter 10: The Art of Rainmaking</h3> <ul> <li>pg 193: If your product succeeds in an unexpected market, find out why and adjust your business accordingly.</li> <li>pg 197: Ignore titles and find the true influencers in the middle and bottom of companies.</li> <li>pg 198: When you’ve found those to “suck down” to, don’t try to buy them, empathize with them, and never go over their head and complain about them.</li> <li>pg 200: Enable someone to do something they simply could not do before, as opposed to having to displace an entrenched product or service.</li> <li>pg 201: If your product doesn’t resonate with a sales prospect, ask why they won’t buy your product and take good notes.</li> <li>pg 203: Offer a smooth, gentle adoption curve; if your product or service really is great, this will help with the hardest step of getting in the door.</li> <li>pg 205: Don’t rely on the sales types to do all the work for you; encourage everyone to make it rain.</li> </ul> <h3>Chapter 11: The Art of Being a Mensch</h3> <ul> <li>pg 211: The three foundations of menschhood are helping lots of people, doing what’s right, and paying back society.</li> <li>pg 213: A mensch does the right thing, not the easy thing, the expedient thing, or the money-saving thing.</li> <li>pg 214: A mensch pays back for goodness already received, as opposed to paying it forward.</li> </ul> https://tedneward.github.io/Research/reading/business/job-search/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/job-search/index.html <p><a href="https://blog.nindalf.com/posts/tech-interview/">"Lessons From a Tech Job Search"</a></p> https://tedneward.github.io/Research/reading/business/metrics/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/metrics/index.html <p>Types of startup metrics (<a href="https://media.licdn.com/dms/image/C4E22AQEoJLGjQkZDng/feedshare-shrink_2048_1536/0/1672738383273?e=1675900800&amp;v=beta&amp;t=6eA_lYLH1lIxZbhiBNlJh4fWfAfDN8tuWtSucmeh3_k">Source</a> from <a href="https://www.linkedin.com/feed/update/urn:li:activity:7015973294453465088">LinkedIN Alan Fecamp</a>):</p> <ul> <li>Monthly Recurring Revenue (MRR), Annual Recurring Revenue (ARR): monthly or annual total of paid customer fees</li> <li>Average Revenue Per Account (ARPA): Per Account MRR/Total # of Customers</li> <li>Gross Profit: Total revenue minus cost of goods sold</li> <li>Total Contract Value (TCV): value of one-time and recurring charges</li> <li>Annual Contract Value (ACV): value of a contract over a year</li> <li>Lifetime Value (LTV): prediction of the net profit from the entire future relationship with a customer</li> <li>Deferred Revenue: amount that was received by a company in advance of earning it</li> <li>Billings: current quarter revenue + deferred revenue from previous quarter</li> <li>Customer Acquisition Cost (CAC): full cost of acquiring one user</li> <li>Concentration Risk: revenue from largest customer/total revenue</li> <li>Daily Active Users (DAU), Monthly Active Users: Users other than one-time users per day.month</li> <li>Number of logins: sign-ins per month</li> <li>Activation rate: nubmer of users taking a specific action to get value out of a product</li> <li>MoM Growth Rate: month-on-month growth: average of monthly growth rates</li> <li>Compounded Monthly Growth Rate (CMGR): (Latest Month/First Month) (1 / # of Months) - 1</li> <li>Monthly Churn Rate: lost customers this month / prior month total</li> <li>Retention Rate: % of original installed base (1st month) that are still transacting</li> <li>Gross Churn Rate: MRR lost in a given month / MRR at the beginning of the month</li> <li>Net Churn: (MRR lost - MRR from upsells) this month / MRR at the beginning of the month</li> <li>Burn Rate: Monthly cash burn rate</li> <li>Total Addressable Market (TAM): revenue opportunity for a product</li> <li>MRR Projection: projection of the current MRR into the future, annualized</li> </ul> https://tedneward.github.io/Research/reading/business/rework/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/rework/index.html <p><em>(by Jason Fried and David Heinemeier Hansson)</em></p> <p><em>I, <a href="http://omgitsmgp.com/">Michael Parker</a>, own this book and took these notes to further my own learning. If you enjoy these notes, please <a href="http://www.amazon.com/Rework-Jason-Fried/dp/0307463745">purchase the book</a>!</em></p> <h3>Takedowns</h3> <ul> <li>pg 19: Make decisions right before you do something, not far in advance; long term plans are just guesses.</li> <li>pg 23: It's okay to stay small; once big, it's hard to shrink without firing people and damaging morale.</li> </ul> <h3>Go</h3> <ul> <li>pg 35: Build what you need, so you'll know the right answers, assess its quality directly, and love your work.</li> <li>pg 43: Have a point of view and strong opinions; otherwise everything is debatable, and decisions are hard.</li> <li>pg 50: Taking outside money relinquishes control to VCs, deprioritizes customers, and is distracting.</li> <li>pg 56: A business without a path to profit isn't a business, it's a hobby.</li> <li>pg 62: Avoid mass, like excess staff, long-term contracts, lock-ins, road maps, etc. to stay lean and agile.</li> </ul> <h3>Progress</h3> <ul> <li>pg 72: Prioritize working on the most important part of your business so it's the best it can be; defer the rest.</li> <li>pg 77: Make decisions, don't wait for the perfect solution; decisions build momentum and boost morale.</li> <li>pg 90: When you make something, you always make something else; sell those byproducts.</li> </ul> <h3>Productivity</h3> <ul> <li>pg 102: Don't throw good time after bad work; ensure that what you're working on matters.</li> <li>pg 108: Meeting suck: they convey little information, drift off-subject, require preparation, and procreate.</li> <li>pg 115: Momentum fuels motivation; build it through constant small victories that generate enthusiasm.</li> <li>pg 121: Don't forego sleep: you'll be stubborn, uncreative, inefficient, irritable, and lose motivation.</li> <li>pg 127: You'll feel guilty over long lists, then stress out; divide and conquer into small, manageable lists.</li> </ul> <h3>Competitors</h3> <ul> <li>pg 136: If you're a copycat, you're in a passive position where you're always behind the times.</li> <li>pg 141: If a competitor sucks, pick a fight; you'll stand out, earn followers, and ignite passions of users.</li> </ul> <h3>Evolution</h3> <ul> <li>pg 153: Get into the habit of saying no; saying yes is easy, but you often wind up regretting it.</li> <li>pg 157: Let customers outgrow you; focus on features that attract new customers, not please old ones.</li> <li>pg 164: Feature requests that matter you'll hear over and over, so no need to write them down.</li> </ul> <h3>Promotion</h3> <ul> <li>pg 174: Teach your customers; even if people don't use your product, they'll still be your fans.</li> <li>pg 188: Niche bloggers are hungry for finding the new thing; stories that start there can go mainstream.</li> <li>pg 191: Give away something for free; if your product is strong enough, people will come back for more.</li> </ul> <h3>Hiring</h3> <ul> <li>pg 201: Don't hire anyone until you've tried the job yourself; then you'll know what and who to look for.</li> <li>pg 206: Hiring more people than needed leads to invented projects, which adds cost and complexity.</li> <li>pg 210: A cover letter lets you hear someone's actual voice; you can tell if it's in tune with your company.</li> <li>pg 222: Great writers are great candidates, for clear writing is clear thinking.</li> <li>pg 227: Hire potential employees for a mini-project; in a real work environment, the truth comes out.</li> </ul> <h3>Damage Control</h3> <ul> <li>pg 235: Responding to customers quickly with a personal response is the best customer service.</li> <li>pg 238: A good apology accepts responsibility, provides details of the problem, and a solution.</li> </ul> <h3>Culture</h3> <ul> <li>pg 251: Don't make up problems you don't have yet; decisions don't lost forever anyway.</li> <li>pg 262: Don't sound big and formal; communicate simply and directly, without legal or PR oversight.</li> <li>pg 268: Saying ASAP devalues any request without it; use emergency language when really needed.</li> </ul> https://tedneward.github.io/Research/reading/business/auctions/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/auctions/index.html <p><a href="http://www.sci.brooklyn.cuny.edu/~parsons/projects/mech-design/publications/bluffers-final.pdf">Paper</a> | <a href="https://speakerdeck.com/paperswelove/daniel-doubrovkine-on-auctions-and-bidding-a-guide-for-computer-scientists">Slides</a> | <a href="https://www.youtube.com/watch?v=v502KYHa3aE">Talk video</a></p> https://tedneward.github.io/Research/reading/business/licensing/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/licensing/index.html <ul> <li><a href="http://www.aliprandi.org/cc-user-guide/">Creative Commons: a user guide</a> - Simone Aliprandi</li> <li><a href="http://rosenlaw.com/oslbook/">Open Source Licensing Software Freedom and Intellectual Property Law</a> - Lawrence Rosen</li> <li><a href="http://www.thepublicdomain.org/download/">The Public Domain: Enclosing the Commons of the Mind</a> - James Boyle</li> </ul> https://tedneward.github.io/Research/reading/business/negotiation/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/negotiation/index.html <p><a href="https://haseebq.com/my-ten-rules-for-negotiating-a-job-offer/">"My 10 Rules for Negotiation a Job Offer"</a> and <a href="https://haseebq.com/how-not-to-bomb-your-offer-negotiation/">Part 2</a>: "The rules, in order of appearance, are:<br> 1. Get everything in writing<br> 1. Always keep the door open<br> 1. Information is power<br> 1. Always be positive<br> 1. Don’t be the decision maker<br> 1. Have alternatives<br> 1. Proclaim reasons for everything<br> 1. Be motivated by more than just money<br> 1. Understand what they value<br> 1. Be winnable</p> https://tedneward.github.io/Research/reading/business/to-sell-is-human/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/to-sell-is-human/index.html <p><em>(by Daniel Pink)</em></p> <h3>Introduction</h3> <ul> <li>The ability to move others to exchange what they have for what we have is crucial to our survival and our happiness.</li> </ul> <h3>Part 1: Rebirth of a Salesman</h3> <h4>Ch 1: We're All in Sales Now</h4> <ul> <li>One out of every nine American workers works in sales, and by 2020 the Bureau of Labor Statistics expects 2 million new sales jobs.</li> <li>The same holds elsewhere around the world, even after the global financial implosion and the Internet.</li> <li>Beyond producing and consuming, we now focus on moving other people to part with their resources so that we both get what we want.</li> <li>People now spend 40% of their time at work engaged in non-sales selling, and they consider this aspect crucial to their professional success.</li> <li>The older someone is, and so the more experience that person has, the more moving others occupies her days and determines her success.</li> <li>The existing data shows that 1 in 9 Americans work in sales; but the other 8 in 9 engage in non-sales selling.</li> </ul> <h4>Ch 2: Entrepreneurship, Elasticity, and Ed-Med</h4> <ul> <li>Very large enterprises and very small ones not only have differences in degree, but differences in kind.</li> <li>Entrepreneurs cannot specialize, they must wear many hats; and that is the first reason why more of us find ourselves in sales.</li> <li>30% of American workers now work on their own; this may grow by 65 million and become the majority of the US workforce in 2020.</li> <li>Technology was supposed to make salespeople obsolete but transformed more people into sellers -- see Etsy, Square, Kickstarter, etc.</li> <li>Elasticity, or the new breadth of skills demanded by established companies, is the second reason why we're all in sales now.</li> <li>A world of flat organizations and tumultuous business conditions punishes fixed skills and prizes elastic ones.</li> <li>As elasticity of skills becomes more common, one particular category of skill it seems always to encompass is moving others.</li> <li>Education and health services, or Ed-Med, is the largest job sector in the U.S. economy, and the fastest growing sector in the world.</li> <li>Teachers and health care professionals sell and convince others to part with time, attention, and effort, for a better future.</li> <li>Irritation is challenging people to do something you want to do; agitation is challenging them to do something they want to do.</li> <li>Non-sales selling requires influencing, persuading, and changing behavior while balancing what others want and what you can provide them.</li> </ul> <h4>Ch 3: From Caveat Emptor to Caveat Venditor</h4> <ul> <li>Adjectives and interjections can reveal people's attitudes, since they often contain an emotional component that nouns lack.</li> <li>Selling makes us uncomfortable and disgusted because we believe its practice revolves around duplicity, dissembling, and double-dealing.</li> <li>The presence of people who wish to pawn bad wares as good wares tends to drive out the legitimate business.</li> <li>In a world of information asymmetry, the guiding principle is caveat emptor, or buyer beware.</li> <li>If you have just as much information as the seller, and a means to talk back, the new guiding principle is caveat venditor, or seller beware.</li> <li>When buyers can know more than sellers, the sellers are not the protectors and purveyors of information, but the curators and clarifiers of it.</li> <li>Our feelings about sales derive not from the inherent nature of selling, but the information asymmetry that long defined its context.</li> <li>But as long as there are complicated products where the potential for lucre is enormous, abide by caveat emptor.</li> <li>The low road is now harder to pass and the high road has become the better, more pragmatic, long-term route.</li> <li>There are no "natural" salespeople, in part because we're all naturally salespeople with a selling instinct.</li> </ul> <h3>Part 2: How to Be</h3> <h4>Ch 4: Attunement</h4> <ul> <li>Attunement is the ability to bring one's actions and outlook into harmony with other people and with the context you're in.</li> <li>Power leads individuals to anchor too heavily on their own vantage point, insufficiently adjusting to others' perspective.</li> <li>Start your encounters assuming that you have less power; this will help you see other side's perspective, and in turn move them.</li> <li>Perspective-taking is a cognitive capacity, mostly about thinking; empathy is an emotional response, mostly about feeling.</li> <li>Perspective-taking is better than empathy, because you can submerge your own interests.</li> <li>People belong to groups, situations, and contexts, so pay attention to the relationships and connections of a person.</li> <li>Syncing our mannerisms and vocal patterns to someone else so that we both understand and can be understood is fundamental to attunement.</li> <li>Related to mimicry, touching someone on the upper arm or shoulder has positive results.</li> <li>The correlation between extraversion and sales in one study was only 0.07, or virtually non-existent.</li> <li>Extraverts can talk too much and listen too little, and fail to balance between asserting and holding back.</li> <li>Intraverts can be too shy to initiate and too timid to close.</li> <li>Ambiverts are the best movers because they're the most skilled attuners, knowing when to speak up and when to shut up.</li> </ul> <p><em>Sample Case</em></p> <ul> <li>Jim Collins' favorite opening question is "Where are you from?" because it opens someone up easily.</li> <li>To master strategic mimicry: watch, then wait before applying, and finally wane, or become less conscious of what you're doing.</li> <li>Jeff Bezos uses an empty chair in meetings to represent the customer; let the chair represent whom you must be attuned to.</li> <li>A discussion map of a meeting has an X next to someone's name when they talk, and has arrows for directed comments.</li> <li>Finding similarities can help you attune yourself to others and help them attune themselves to you.</li> <li>Similarity is a key form of human connection, as people are more likely to move together when they share common ground.</li> </ul> <h4>Ch 5: Buoyancy</h4> <ul> <li>How to stay afloat in the ocean of rejection, or buoyancy, is the second essential quality in moving others.</li> <li>Positive self-talk is better than negative self-talk, but most effective is to ask questions instead of make statements.</li> <li>The interrogative elicits answers, in which are strategies for actually carrying out the task.</li> <li>Moreover, interrogative self-talk inspires thoughts about autonomous and intrinsically motivated reasons to pursue a goal.</li> <li>Negative emotions narrow our vision and help us survive in the moment, namely fight or flight.</li> <li>Positive emotions broaden our ideas about actions, opening us to a wider range of thoughts, making us more receptive and creative.</li> <li>Believing in an offering leads to a deeper understanding of it, allowing sellers to better match what they have with what others need.</li> <li>A ratio of positive to negative emotions between 3:1 and 11:1 is ideal.</li> <li>Negative emotions offer us feedback on our performance, information on what's not working, and hints on how to do better.</li> <li>Unchecked levity makes you flighty, ungrounded, and unreal, while unchecked gravity leaves you in a heap of misery.</li> <li>Learned helplessness is a habit of people explaining negative events to themselves as permanent, pervasive, or personal.</li> <li>They believe that the negative conditions will last a long time, the causes are universal, and that they're the ones to blame.</li> <li>Optimism can stir persistence, steady us during challenges, and stoke the confidence that we can influence our surroundings.</li> </ul> <p><em>Sample Case</em></p> <ul> <li>Next time you're getting ready to persuade others, ask yourself "Can I move these people?" Answer it, directly and in writing.</li> <li>Be more positive by displaying positive emotions of joy, gratitude, serenity, interest, hope, pride, amusement, inspiration, awe, and love.</li> <li>Explain bad events as temporary, specific, and external, and find ways to "dispute" and "de-catastrophize" negative explanations.</li> <li>Thinking through the gloom-and-doom scenarios and mentally preparing for the very worst can help some people manage their anxieties.</li> <li>Writing yourself a fake rejection letter can make consequences seem less dire, and even reveal soft spots in what you're presenting.</li> </ul> <h4>Ch 6: Clarity</h4> <ul> <li>Partly because our brains evolved when the future itself was perilous, we are bad at wrapping our minds around far-off events.</li> <li>Clarity is the capacity to help others see their problems in fresh and revealing ways, and to identify problems they didn't realize they had.</li> <li>The ability to move others hinges on problem finding, not solving, when one's mistaken, confused, or clueless about the true problem.</li> <li>People most disposed to creative breakthroughs tend to be problem finders, and not just problem solvers.</li> <li>Non-sales selling depends more on the problem-finding because information symmetry enables us to problem solve.</li> <li>We must now be adept at curating data, not accessing it; we must also be adept at asking questions, not answering questions.</li> <li>The contrast principle says we understand something when compared to something else than when we see it in isolation.</li> <li>The less frame: Restricting people's choices can help them see those choices more clearly instead of overwhelming them.</li> <li>The experience frame: Frame a sale as a purchase of an experience, which is more satisfying than the sale of a good.</li> <li>The label frame: Simply changing the label of an activity can favorably alter behavior.</li> <li>The blemished frame: Adding a minor negative detail in an otherwise positive description can give that description more impact.</li> <li>The blemishing effect only works if the subject is in a "low effort" state, and if the negative information comes last.</li> <li>The potential frame: Highlighting one's potential, not achievements, causes deeper thought into why that person is a good choice.</li> <li>Clarity on how to think without clarity on how to act can leave people unmoved, so provide a clear path of action.</li> </ul> <p><em>Sample Case</em></p> <ul> <li>Irrational questions work better than rational questions when trying to motivate resistant people.</li> <li>To become an efficient curator, first seek, or gather information; then sense, or create meaning out if it; and finally share.</li> <li>To ask better questions, write down as many as you can, classify them as open-ended or closed-ended, and choose the best three.</li> <li>The Five Whys technique forces us to examine and express the underlying reasons for our behaviors and attitudes.</li> <li>Always focus on the "one percent," or the essence of what you're exploring, that gives life to the other ninety-nine.</li> </ul> <h3>Part 3: What To Do</h3> <h4>Ch 7: Pitch</h4> <ul> <li>Pitching is the ability to distill one's point to its persuasive essence.</li> <li>A pitch isn't to move others to immediately adopt your idea, but to begin a conversation, collaborate, and arrive at a mutually beneficial outcome.</li> <li>The elevator pitch is a bit threadbare, because organizations today are more democratic and because CEOs have more distractions.</li> <li>A one-word pitch reduces a point to a single word, and helps you be heard when attention spans are nearly disappearing.</li> <li>The question pitch asks a question, which compels people to respond and is more effective than statements unless the backing arguments are weak.</li> <li>The rhyming pitch relies on rhymes increasing "processing fluency," and we equate this ease of processing with accuracy.</li> <li>The subject-line pitch relies on utility when one is busy and curiosity when one is bored, but not both; as well as specificity.</li> <li>The Twitter pitch values short pitches that ask questions, convey information with links, and to self-promotion.</li> <li>The Pixar pitch follows the structure "One upon a time ___. Every day, ___. One day, ___. Because of that, ___. Because of that, ___. Until finally, ___."</li> </ul> <p><em>Sample Case</em></p> <ul> <li>Ensure that after hearing your pitch, you can answer what someone should know, someone should feel, and someone should do.</li> <li>You can enliven question pitches, one-word pitches, rhyming pitches, and Twitter pitches with visuals.</li> <li>A pecha-kucha presentation contains exactly twenty slides, each of which appears on the screen for exactly twenty seconds.</li> <li>When pitching, go first if you're the incumbent and last if you're the challenger; avoid the middle.</li> <li>In a pitch, granular numbers are more credible than coarse ones.</li> <li>Ask people to describe in three words what your organization, your product, and you are about, and then look for patterns.</li> </ul> <h4>Ch 8: Improvise</h4> <ul> <li>Sales scripts perform nicely in an environment where buyers have minimal choices and sellers have maximal information.</li> <li>The stable, simple, and certain conditions that favored scripts have given way to the dynamic, complex, and unpredictable conditions that favor improvisation.</li> <li>To move others, follow the three essential rules of improv: hear others, say "Yes and," and make your partner look good.</li> <li>Listening without some degree of intimacy isn't really listening; it's passive and transactional, not active and engaged.</li> <li>Don't listen <em>for</em> anything; instead, take in anything and everything someone says as an offer you can do something with.</li> <li>While "Yes, but" spirals down into frustration, "Yes, and" spirals up toward positivity.</li> <li>Under conditions of information asymmetry, results are often win-lose; but with information parity, we cannot push for win-lose.</li> <li>Making your partner look good calls for, and enables, clarity, the capacity to develop solutions that nobody previously imagined.</li> <li>Ask questions, because when both sides view an encounter as an opportunity to learn, the desire to defeat the other side wanes.</li> <li>Never argue, for to win an argument is to lose a sale.</li> </ul> <p><em>Sample Case</em></p> <ul> <li>If your conversation partner isn't finishing his sentence, or can't speak without you interrupting, then you need to slow down.</li> <li>Ask questions, but don't ask yes-no questions, don't ask questions that are veiled opinions.</li> </ul> <h4>Ch 9: Serve</h4> <ul> <li>For service to cause people to achieve something greater and more enduring than an exchange of resources, make it personal and purposeful.</li> <li>In circumstances in which we move others, we not adopt a stance that is abstract and distant, but concrete and personal.</li> <li>This lets you recognize the person you're trying to serve, and also puts you personally behind whatever you're trying to sell.</li> <li>Many of us like to say "I'm accountable," but few of us are so deeply committed to serving other that we'll say, "Call my cell."</li> <li>We assume everyone is driven by self-interest, but we all do things for "prosocial" or "self-transcending" reasons.</li> <li>We should also be tapping others' innate desire to serve by making it purposeful, not just personal.</li> <li>A servant-leader begins with the natural feeling that one wants to serve; then conscious choice brings one to aspire to lead.</li> <li>The best test of this philosophy is whether those who are served grow as persons, and in turn more likely themselves to become servants.</li> <li>Servant selling asks if the person you're selling to agrees to buy, will his or her life, and the state of the world, improve?</li> </ul> <p><em>Sample Case</em></p> <ul> <li>Upserving means doing more than the other person expects or you initially intended, transforming a mundane interaction into a memorable experience.</li> <li>Really good salespeople want to solve problems and serve customers, becoming part of something larger than themselves.</li> <li>Emotionally intelligent signage either expresses empathy with the person viewing it, or tries to trigger empathy in that person.</li> <li>To make an encounter personal and encourage you to genuinely serve, imagine that the person you're dealing with is you grandmother.</li> <li>If the buyer isn't improving her life, or the world isn't a better place after the interaction, you're doing something wrong.</li> </ul> https://tedneward.github.io/Research/reading/business/competing-on-analytics/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/competing-on-analytics/index.html <p><em>(by Thomas H Davenport, from "HBR's 10 Must Reads: The Essentials")</em></p> <p>TBD</p> https://tedneward.github.io/Research/reading/business/marketing-myopia/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/marketing-myopia/index.html <p><em>(by Theodore Levitt, from "HBR's 10 Must Reads: The Essentials")</em></p> <p>What business are you really in? A seemingly obvious question--but one we should all ask before demand for our competitors' products or services dwindles.</p> <p>The railorads failed to ask this same question--and stopped growing. Why? Not because people no longer needed transportation. And not because other innovations (cars, airplanes) filled transportation needs. Rather, railroads stopped growing because railroads didn't move to fill those needs. Their executives incorrectly thought that they were in the railroad business, not the transportation business. They viewed themselves as providing a product instead of serving customers. Too many other industries make the same mistake--putting themselves at risk of obsolescence.</p> <p>How to ensure continued growth for your company? Concentrate on meeting customers' needs rather thna selling products. Chemical powerhouse DuPont kept a close eye on its customers' most pressing concerns--and deployed its technical know-how to create an ever-expanding array of products that appealed to customers and continuously enlarged its market. If DuPont had merely found more uses for its flagship innovation (nylon), it might not be around today.</p> <p>We put our businesses at risk of obsolescence when we accept any of the following myths:</p> <p><strong>Myth 1: An ever-expanding and more affluent population will ensure our growth.</strong> When markets are expanding, we often assume we don't have to think imaginatively about our businesses. Instead, we seek to outdo rivals simply by improving on what we're already doing. The consequence: We incrase the efficiency of <em>making</em> our products, rather than boosting the value those products deliver to customers.</p> <p><strong>Myth 2: There is no competitive substitute for our industry's major product.</strong> Believing that our products have no rivals makes our companies vulnerable to dramatic innovations from outside our industries--often by smaller, newer companies that are focusing on customer needs rather than the products themselves.</p> <p><strong>Myth 3: We can protect ourselves through mass production.</strong> Few of us can resist the prospect of the incresaed profits that come with steeply declining unit costs. But focusing on mass production emphasizes our <em>company</em>'s needs--when we should be emphasizing our <em>customers</em>'.</p> <p><strong>Myth 4: Technical research and development will ensure our growth.</strong> When R&amp;D produces breakthrough products, we may be tempted to organize our companies around the technology rather than the consumer. Instead, we should remain focused on satisfying customer needs.</p> <p>(Examples: petroleum, automobiles, electronics industries.)</p> <p>No organization can achieve greatness without a vigorous leader who is driven onward by a pulsating <em>will to succeed</em>. A leader has to have a vision of grandeur, a vision that cna produce eager followers in vast numbers. In business, the followers are the customers.</p> <p>In order to produce these customers, the entire corporation must be viewed as a customer-creating and customer-satisfying organism. Management must think of itself not as producing products but as providing customer-creating value satisfactions. It must push this idea (and everything it means and requires) into every nook and cranny of the organization. It has to do this continuously and with the kind of flair that excites and stimulates the people in it. Otherwise, the company will be merely a series of pigeonholed parts, with no consolidating sense of purpose or direction.</p> <p>In short, the organization must learn to think of itself not as producing goods or services, but as <em>buying customers</em>, as doing the things that will make people <em>want</em> to do business with it. And the chief executive has the inescapable responsibility for creating this environment, this viewpoint, this attitude, this aspiration. The chief executive must set the company's style, its direction, and its goals. This means knowing precisely where he/she wants to go and making sure the whole organization is enthusiasically aware of where that is. This is a first requisite of leadership, for <em>unless a leader knows where he/she is going, any road will take him/her there.</em> (And if any road is OK, the chief executive may as well go fishing--if the organization does not know or care where it is going, it does not need to advertise that fact with a ceremonial figurehead, for everybody will notice it soon enough.)</p> https://tedneward.github.io/Research/reading/business/platform-revolutions/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/platform-revolutions/index.html <p><em>(by Geoffrey G Parker, Marshall W Van Alstyne, and Sangeet Paul Choudary)</em></p> <h1>1. Today</h1> <p><em>Welcome to the Platform Revolution</em></p> <p><strong><em>Platform</em></strong>: "A business based on enabling value-creating interactions between external producers and consumers. Platform provides open, participative infrastructure for these interactions and sets governance conditions for them. Purpose: to consummate matches among users and facilitate the exchange of goods, services, or social currency, thereby enabling value creation for all participants."</p> <p>Traditional system: pipeline/linear value chain (single-track, assembly line)</p> <p><strong><em>Platforms beat pipelines because platforms scale more efficiently by eliminating gatekeepers.</em></strong> Until recently, most business were built around products--designed/made at one end of the pipeline and delivered to consumers at the other end. Pipelines rely on inefficient gatekeepers to manage the flow of value from the producer to the consumer; the elimination of gatekeepers allow consumers greater freedom to select products that fit their needs.</p> <p><strong><em>Platforms beat pipelines because platforms unlock new sources of value creation and supply.</em></strong></p> <p><strong><em>Platforms beat pipelines by using data-based tools to create community feedback loops.</em></strong></p> <p><strong><em>Platforms invert the firm.</em></strong> Strategy has moved from controlling the unique resources and erecting competitive barriers to orchestrating external resources and engaging vibrant communities.</p> <h1>2. Network Effects</h1> <p><em>The Power of the Platform</em></p> <p>(Uber as a "matching"--matching ride-providers to ride-seekers--service)</p> <p>Network effects: the impact the number of users of a platform has on the value created for each user.</p> <p>Positive network effects: the ability of a large, well-managed platform community to produce significant value for each user of the platform</p> <p>Negative network effects: the ability to reduce value for each user</p> <p>Uber's "virtuous circle" (flowing clockwise)</p> <pre><code>Lower prices ---&gt; More demand \ ^ ^ \ | \ More drivers | Faster pickups / | ^ / | \ / | ---- More geographic | / coverage/saturation | v Less driver downtime (More demand -&gt; More drivers -&gt; More geographic coverage/satuation -&gt; Faster pickups and Less driver downtime -&gt; Lower prices -&gt; More demand....) </code></pre> <p><strong><em>Demand economies of scale</em></strong><br> 20th-century monopolies were created based on supply-side economies of scale, driven by production efficiencies, which reduce unit cost of good/service delivered. 21st-century driven by demand economies of scale, which take advantage of technology improvements on the demand side--efficiencies in social networks, demand aggregation, app development, and other phenomena that make bigger networks more valuable to users. (Metcalfe's Law: The value of a telephone network.)</p> <p><strong><em>Two-sided network effects</em></strong><br> (The cycle--both sides of the market are engaged.)</p> <p><strong><em>Network effects vs other growth-building tools</em></strong><br> Price effects are evanescent: they disappear the moment the discounts end or another firm offers a better price. Typically only 1-2% convert.<br> Brand effects are stickier, but difficult to sustain.<br> Virality effects attract, but network effects keep people there.</p> <p><strong><em>Scaling network effects: Frictionless entry (and other tools)</em></strong><br> Effective platforms are able to expand in size quickly and easily, thereby scaling the value that derives from network effects. Netorks that permit frictionless entry are able to grow organically without limits.<br> Sideswitching: when users of one side of the platform can join the opposite side.</p> <p><strong><em>Negative network effects</em></strong><br> Growth adds to difficulty of finding matches; offset by curation.<br> Data-driven network effects</p> <p><strong><em>Four kinds of network effects</em></strong><br> Two-by-two grid of "same-side/cross-side" and "positive/negative" network effects:<br> Same-side positive network effects<br> Same-side negative network effects<br> Cross-side positive network effects<br> Cross-side negative network effects</p> <p><strong><em>Structural change: Network effects turn firms inside out</em></strong><br> Four broad categories of companies:</p> <ul> <li>Asset builders</li> <li>Service providers</li> <li>Technology creators (develop/sell intellectural property)</li> <li>Network orchestrators (develop networks)</li> </ul> <p>In the world of network effects, ecosystems of users are the new source of competitive advantage/market dominance.</p> <h1>3. Architecture</h1> <p><em>Principles for Designing a Successful Platform</em></p> <p>A platform connects producers w/consumers and allows them to exchange value, one of 3 things:</p> <ul> <li>Information</li> <li>Goods/services</li> <li>Currency</li> </ul> <p><strong><em>The core interaction: The Way of Platform Design</em></strong><br> 3 key components:</p> <ul> <li>Particpants: Producer creates value, consumer consumes it</li> <li>Value unit: users are provided w/basis for deciding whether they want to proceed to further exchange</li> <li>Filter: Software-based tool used by the platform to enable the exchange of appropriate value units between users</li> </ul> <blockquote> <p>When designing a platform, your first and most important job is to decide what your core interaction will be, then define the participants, the value units, and the filters to make sure such core interaction is possible.</p> <p>Platforms don't create value units--they are created by the producers in the platform; thus the platforms are "information factories" that have no control over inventory. They create the 'factory floor'; they can foster a culture of quality control, and they develop filters that are designed to deliver valuable units while blocking others, but they have no control over the production process itself.</p> </blockquote> <p><strong><em>Pull, facilitate, match: The How of Platform Design</em></strong></p> <ul> <li>Pull producers/consumers to the paltform</li> <li>Facilitate their interactions by providing tools/rules that make it easy for them to connect &amp; enhance</li> <li>Match producers and consumers effectively by using info about each to connect them in ways they will find mutually rewarding</li> </ul> <p>Pull challenges:</p> <ul> <li>attracting users</li> <li>keeping the interest of users</li> <li>tool: feedback loop(s): single-user and multi-user</li> </ul> <p><strong><em>Behind the core interaction</em></strong><br> Successful platforms tend to scale by layering new inteactions on top of the core interaction</p> <ul> <li>by changing the value unit exchanged between existing users</li> <li>by introducing a new category of users as either producers or consumers</li> <li>by allowing users to exchange new kinds of value units</li> <li>by curating membrs of an existing user group to create a new category of users</li> </ul> <p><strong><em>Applying the end-to-end principle to platform design</em></strong><br> End-to-End Principle: in a general-purpose network, application-specific functions out to reside in the end hosts of a network rather than in intermediary nodes. Only the highest-volume, highest-value features that cut across apps should become part of the core platform.</p> <ul> <li>When specific new features are incorporated into the core platform, apps that don't use them appear slow/inefficient; when app-specific features are run by an app itself, UX is cleaner</li> <li>Platform ecosystem can evolve faster when the core platform is a clean, simple system; stable core restricting variety, but underneath an evolving layer that enables variety</li> </ul> <p><strong><em>Tower of Modularity</em></strong><br> Successful platform must have a modular approach; "designers achieve modularity by partitioning information into visible design rules and hidden design parameters; modularity is beneficial only if the partition is precide, unambiguous, and complete." ... "Fundamental architecture behind all systems is ... the system is partitioned into a set of 'core' components w/low variety and a complementary set of 'peripheral' components w/high variety. The low-variety components consititute the platform. They are the long-lived elements of the system and thus implicitly/explicitly establish the system's interfaces and the rules governing interactions among the parts."</p> <p><strong><em>Re-architecting the platform</em></strong><br> First step: analyze the degree of modularity already achieved.</p> <p><strong><em>Iterative improvement: The Anti-Design principle</em></strong><br> Platforms cannot be entirely planned--they also emerge; platforms characterized by activity controlled by the users, not bythe owners/managers. Platform designers should always leave room for serendipitous discoveries, as suers often lead the way to where design should evolve. Close monitoring of user behavior reveals unexpected patterns, some of which may suggest fruitful new ideas for value creation. Best platforms allow for user quirks &amp; are open enough to gradually incorporate such quirks.</p> <h1>4. Disruption</h1> <p><em>How Platforms Conquer and Transform Traditional Industries</em></p> <p>Platform concept is not new--farmer's markets, stock markets, etc. What's new is digital technology enabling reach.</p> <p><strong><em>Capsule history of digitial disruption</em></strong><br> Stage 1: Efficient pipelines are inefficient pipelines<br> Stage 2: Platforms eat pipelines<br> In the world of platforms, the internet no longer acts merely as a distribution channel; it also acts as a creation infrstructure &amp; coordination mechanism</p> <p><strong><em>Impacts of platform distribution on value creation, value consumption, and quality control</em></strong><br> Reconfiguring value creation to tape new supply sources<br> Reconfiguring value consumption by enabling new forms of consumer behavior<br> Reconfiguring quality control through community-driven curation</p> <p><strong><em>Structural impacts of platform disruption</em></strong><br> De-linking assets from value<br> Re-intermediation<br> Market aggregation</p> <p><strong><em>The incumbents fight back: Pipelines becoming platforms</em></strong><br> They'll need to ask questions:</p> <ul> <li>Which processes that we curently manage in-house can be delegated to outside partners, whether suppliers or consumers?</li> <li>How can we empower outside partners to create products/services that will generate new forms of value for our existing customers?</li> <li>Are there way swe can network w/current competitors to produce valuable new services for customers?</li> <li>How can the value of the goods &amp; services we currently provide be enhanced through new data streams, interpersonal connection, and curation tools?</li> </ul> <h1>5. Launch: Chicken or Egg?</h1> <p><em>Eight Ways to Launch a Successful Platform</em></p> <h4>The heart of platform marketing: Designing for growth</h4> <p>Pull strategies are most effective/important over push; goods/services must be designed to be so attractive they naturally pull customers into their orbit</p> <p><strong><em>The incumbents' advantage: Reality or illusion?</em></strong><br> Large enterprises have advantages, which can lead to complacency</p> <h4>Platform launch strategies</h4> <ol> <li><strong><em>The Follow-the-Rabbit Strategy.</em></strong> Use a non-platform demonstration project to model success, probably attracting both users and producers to a new platform erected on your project's proven infrastructure. How to overcome the chicken/egg problem: <ul> <li>Staging value creation: The platform managers arrange for the creation of value units that will attract one or more sets of users and demonstrate the potential benefits of participating in the platform.</li> <li>Designing the paltform to attract one set of users: platform is designed to provide tools, products, services, or other benefits that will attract one set of users--consumers or producers. The existence of a critical mass of users on one side of a marketplace attracts users on the other side, leading to a positive feedback loop.</li> <li>Simultanous on-boarding: platform creates conditions such that value units can be created that are relevant to users even when the network size is too small. It then strives to stimulate a burst of activity that will simultanously attract both consumers and producers in sufficient numbers to create larger numbers of value units and value-producing interactions so that network effects begin to kick in.</li> </ul> </li> <li><strong><em>The Piggyback Strategy.</em></strong> Connect with an existing user base from a different platform and stage the creation of value units in order to recruit those users to participate.</li> <li><strong><em>The Seeding Strategy.</em></strong> Create value units that will be relevant to at least one set of potential users. When those users are attracted to the platform, other sets of users who want to engage in interactions with them will follow. In many cases, the platform company takes the task of value creation upon itself by acting as the first producer. In addition to jump-starting the platform, the platform owner can define kind and quality of value units they want to see.</li> <li><strong><em>The Marquee Strategy.</em></strong> Provide incentives to attract members of a key user set onto your platform. A variation of this is to choose to purchase a marquee participant in order to obtain exclusive access to the seeds it produces.</li> <li><strong><em>The Single-Side Strategy.</em></strong> Create a business around products or services that benefit a single set of users; later, convert the business into a platform business by attracting a second set of users who want to engage in interactions with the first set.</li> <li><strong><em>The Producer Evangelism Strategy.</em></strong> Design your platform to attract producers, who can induce their customers to become users of the platform. Platforms that provide businesses w/tools for customer relationship management can often solve the chicken/egg problem by attracting one set of users (producers) who then take on the task of bringing along the other set (consumers) from their own customer base. The platform helps the producers cater to their existing set of consumers, and over time, the producers benefit from data-driven cross-pollination as other consumers on the network become interested.</li> <li><strong><em>The Big-Bang Adoption Strategy.</em></strong> Use one or more traditional push marketing strategies to attract a high volume of interest and attention to your platofrm. This triggers a simultaneous on-boarding effect.</li> <li><strong><em>The Micromarket Strategy.</em></strong> Start by targeting a tiny market that comprises members who are already engaging in interactions. This enables the platform to provide the effective matchmaking characteristic of a large market even in the earliest stages of growth.</li> </ol> <h4>Viral growth: The user-to-user launch mechanism</h4> <p>Four key elements:</p> <ul> <li>The sender: getting senders to spread value untis is not the same as word-of-mouth; word of mouth happens when users like your platform so much, they can't stop talking about it. When users become senders and spread value units, they aren't talking about your platform--they are spreading their own creations, and indirectly generating awareness of/interest in your platform. Users do this to get social feedback (fun, fame, fulfillment, fortune, etc); avoid discouraging the spread of value units</li> <li>The value unit: The fundamental unit of virality, but not all value units are spreadable. Design value units to be spreadable as a crucial step.</li> <li>The recipient: Platform managers have no control over them.</li> <li>The external network: External networks often impose restrictions.</li> </ul> <h1>6. Monetization</h1> <p><em>Capturing the Value Created by Network Effects</em></p> <h1>7. Openness</h1> <p><em>Defining What Platform Users and Partners Can and Cannot Do</em></p> <h1>8. Governance</h1> <p><em>Policies to Increase Value and Enhance Growth</em></p> <h1>9. Metrics</h1> <p><em>How Platform Managers Can Measure What Really Matters</em></p> <h1>10. Strategy</h1> <p><em>How Platforms Change Competition</em></p> <h1>11. Policy</h1> <p><em>How Platforms Should (and Should Not) Be Regulated</em></p> <h1>12. Tomorrow</h1> <p><em>The Future of the Platform Revolution</em></p> https://tedneward.github.io/Research/reading/business/yegge/index.html Mon, 13 Apr 2026 14:04:55 +0000 reading/business/yegge/index.html <p><a href="https://gist.github.com/chitchcock/1281611">Stevey's Google Platforms Rant</a>: <em>(This is too good not to capture a copy so I make sure I always have it on-hand. For personal use only--go link to the original.)</em></p> <h1>Stevey's Google Platforms Rant (2011)</h1> <p>I was at Amazon for about six and a half years, and now I've been at Google for that long. One thing that struck me immediately about the two companies -- an impression that has been reinforced almost daily -- is that Amazon does everything wrong, and Google does everything right. Sure, it's a sweeping generalization, but a surprisingly accurate one. It's pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn't let me show it to anyone, even though recruiting loved it.</p> <p>I mean, just to give you a very brief taste: Amazon's recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they've made to level it out. And their operations are a mess; they don't really have SREs and they make engineers pretty much do everything, which leaves almost no time for coding - though again this varies by group, so it's luck of the draw. They don't give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don't have any of our perks or extras -- they just try to match the offer-letter numbers, and that's the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place.</p> <p>To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn't take most of it even if it were free.</p> <p>I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google.</p> <p>I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it's given them some competitive advantages in the marketplace, it's created enough other problems to make it something less than a slam-dunk.</p> <p>But there's one thing they do really really well that pretty much makes up for ALL of their political, philosophical and technical screw-ups.</p> <p>Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon's retail site. He hired Larry Tesler, Apple's Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally -- wisely -- left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn't let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they're all still there, and Larry is not.</p> <p>Micro-managing isn't that third thing that Amazon does better than us, by the way. I mean, yeah, they micro-manage really well, but I wouldn't list it as a strength or anything. I'm just trying to set the context here, to help you understand what happened. We're talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people "who runs the company" when they disagree with him. The guy is a regular... well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don't get me wrong. He just makes ordinary control freaks look like stoned hippies.</p> <p>So one day Jeff Bezos issued a mandate. He's doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion -- back around 2002 I think, plus or minus a year -- he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.</p> <p>His Big Mandate went something along these lines:</p> <p>1) All teams will henceforth expose their data and functionality through service interfaces.</p> <p>2) Teams must communicate with each other through these interfaces.</p> <p>3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team's data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.</p> <p>4) It doesn't matter what technology they use. HTTP, Corba, Pubsub, custom protocols -- doesn't matter. Bezos doesn't care.</p> <p>5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.</p> <p>6) Anyone who doesn't do this will be fired.</p> <p>7) Thank you; have a nice day!</p> <p>Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.</p> <p>#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word "hardened interface" a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.</p> <p>Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon's vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon's dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:</p> <ul> <li> <p>pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.</p> </li> <li> <p>every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.</p> </li> <li> <p>monitoring and QA are the same thing. You'd never think so until you try doing a big SOA. But when your service says "oh yes, I'm fine", it may well be the case that the only thing still functioning in the server is the little component that knows how to say "I'm fine, roger roger, over and out" in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it's indistinguishable from automated QA. So they're a continuum.</p> </li> <li> <p>if you have hundreds of services, and your code MUST communicate with other groups' code via these services, then you won't be able to find any of them without a service-discovery mechanism. And you can't have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.</p> </li> <li> <p>debugging problems with someone else's code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.</p> </li> </ul> <p>That's just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they're not supposed to trust external developers.</p> <p>This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.</p> <p>At this point they don't even do it out of fear of being fired. I mean, they're still afraid of that; it's pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they've come to understand that it's the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it's the right thing because SOA-driven design enables Platforms.</p> <p>That's what Bezos was up to with his edict, of course. He didn't (and doesn't) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.</p> <p>You wouldn't really think that an online bookstore needs to be an extensible, programmable platform. Would you?</p> <p>Well, the first big thing Bezos realized is that the infrastructure they'd built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel' o' other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.</p> <p>The other big realization he had was that he can't always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn't use the goddamn website. It's not even super clear whose mom he was talking about, and doesn't really matter, because nobody's mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I've just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.</p> <p>I'm not really sure how Bezos came to this realization -- the insight that he can't build one product and have it be right for everyone. But it doesn't matter, because he gets it. There's actually a formal name for this phenomenon. It's called Accessibility, and it's the most important thing in the computing world.</p> <p>The. Most. Important. Thing.</p> <p>If you're sorta thinking, "huh? You mean like, blind and deaf people Accessibility?" then you're not alone, because I've come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn't been able to get through to you yet. It's not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software -- or idea-ware for that matter -- fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.</p> <p>Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there's more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.</p> <p>But I'll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.</p> <p>So yeah. In case you hadn't noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I've worked at. But I will never get this little rant published, and you'll never get it read, unless I start to wrap up.</p> <p>That one last thing that Google doesn't do well is Platforms. We don't understand platforms. We don't "get" platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.</p> <p>But no. No, it's like our tenth or eleventh priority. Or fifteenth, I don't know. It's pretty low. There are a few teams who treat the idea very seriously, but most teams either don't think about it all, ever, or only a small percentage of them think about it in a very small way.</p> <p>It's a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they're building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I'm concerned, it's none of them. Stubby's great, but it's like parts when you need a car.</p> <p>A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.</p> <p>Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don't get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: "So is it the Stalker API?" She got all glum and said "Yeah." I mean, I was joking, but no... the only API call we offer is to get someone's stream. So I guess the joke was on me.</p> <p>Microsoft has known about the Dogfood rule for at least twenty years. It's been part of their culture for a whole generation now. You don't eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.</p> <p>Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that's not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there's something there for everyone.</p> <p>Our Google+ team took a look at the aftermarket and said: "Gosh, it looks like we need some games. Let's go contract someone to, um, write some games for us." Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.</p> <p>You can't do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don't have a Steve Jobs here. I'm sorry, but we don't.</p> <p>Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn't need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.</p> <p>I apologize to those (many) of you for whom all this stuff I'm saying is incredibly obvious, because yeah. It's incredibly frigging obvious. Except we're not doing it. We don't get Platforms, and we don't get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.</p> <p>So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don't "get" much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you've never seen it before, prepare to be amazed. Because it's staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can't design for squat, but at least they're doing it.</p> <p>Amazon gets it. Amazon's AWS (aws.amazon.com) is incredible. Just go look at it. Click around. It's embarrassing. We don't have any of that stuff.</p> <p>Apple gets it, obviously. They've made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft's, and have been since time immemorial.</p> <p>Facebook gets it. That's what really worries me. That's what got me off my lazy butt to write this thing. I hate blogging. I hate... plussing, or whatever it's called when you do a massive rant in Google+ even though it's a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it'd be pretty easy to just go. But Google is home, so I'm insisting that we have this little family intervention, uncomfortable as it might be.</p> <p>After you've marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn't look because I didn't want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It's like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.</p> <p>Please don't get me wrong here -- I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They're kicking ass as far as I'm concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.</p> <p>I'm just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where's the Maps APIs in there for Christ's sake? Some of the things in there are labs projects. And the APIs for everything I clicked were... they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it's all snouts and horse hooves.</p> <p>And also don't get me wrong about Google+. They're far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.</p> <p>Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs -- Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it's hard for them to get funding for it because it's not part of our culture. Maestro's funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it's a fluffy rabbit versus a T-Rex. The Docs team knows they'll never be competitive with Office until they can match its scripting facilities, but they're not getting any resource love. I mean, I assume they're not, given that Apps Script only works in Spreadsheet right now, and it doesn't even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.</p> <p>Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook -- that is, the stock service they offer with walls and friends and such -- is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.</p> <p>You know how people are always saying Google is arrogant? I'm a Googler, so I get as irritated as you do when people say that. We're not arrogant, by and large. We're, like, 99% Arrogance-Free. I did start this post -- if you'll reach back into distant memory -- by describing Google as "doing everything right". We do mean well, and for the most part when people say we're arrogant it's because we didn't hire them, or they're unhappy with our policies, or something along those lines. They're inferring arrogance because it makes them feel better.</p> <p>But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we're being fools. You can attribute it to arrogance, or naivete, or whatever -- it doesn't matter in the end, because it's foolishness. There IS no perfect product for everyone.</p> <p>And so we wind up with a browser that doesn't let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I'm actually going blind. For real. I've been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they're quite brazen about it, and Fuck You if you're blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.</p> <p>It's not just them. It's everyone. The problem is that we're a Product Company through and through. We built a successful product with broad appeal -- our search, that is -- and that wild success has biased us.</p> <p>Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers... if only they could be monetized somehow... you can see how he arrived at AWS, in hindsight.</p> <p>Microsoft started out as a platform, so they've just had lots of practice at it.</p> <p>Facebook, though: they worry me. I'm no expert, but I'm pretty sure they started off as a Product and they rode that success pretty far. So I'm not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.</p> <p>Maybe they just looked at us and asked: "How can we beat Google? What are they missing?"</p> <p>The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don't do internal service-oriented platforms, and we just as equally don't do external ones. This means that the "not getting it" is endemic across the company: the PMs don't get it, the engineers don't get it, the product teams don't get it, nobody gets it. Even if individuals do, even if YOU do, it doesn't matter one bit unless we're treating it as an all-hands-on-deck emergency. We can't keep launching products and pretending we'll turn them into magical beautiful extensible platforms later. We've tried that and it's not working.</p> <p>The Golden Rule of Platforms, "Eat Your Own Dogfood", can be rephrased as "Start with a Platform, and Then Use it for Everything." You can't just bolt it on later. Certainly not easily at any rate -- ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it'll be ten times as much work as just doing it correctly up front. You can't cheat. You can't have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.</p> <p>I'm not saying it's too late for us, but the longer we wait, the closer we get to being Too Late.</p> <p>I honestly don't know how to wrap this up. I've said pretty much everything I came here to say today. This post has been six years in the making. I'm sorry if I wasn't gentle enough, or if I misrepresented some product or team or person, or if we're actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I'm sorry.</p> <p>But we've gotta start doing this right.</p> <p><a href="https://plus.google.com/110981030061712822816/posts/WugKtXSp7We">Original URL</a></p> <p><a href="https://plus.google.com/110981030061712822816/posts/bwJ7kAELRnf">Follow-up URL</a></p> <hr> https://tedneward.github.io/Research/presentation/webcomponents/lit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/lit/index.html <p><a href="https://lit.dev/">Website</a> | <a href="https://github.com/lit/lit/">Source</a> | <a href="https://lit.dev/tutorial/">Tutorial</a></p> <p>Builds on top of Web Components.</p> <pre><code class="language-ts">import {html, css, LitElement} from 'lit'; import {customElement, property} from 'lit/decorators.js'; @customElement('simple-greeting') // (1) export class SimpleGreeting extends LitElement { static styles = css`p { color: blue }`; // (2) @property() // (3) name = 'Somebody'; render() { return html`&lt;p&gt;Hello, ${this.name}!&lt;/p&gt;`; // (4) } } </code></pre> <ol> <li> <p><strong>Custom Elements</strong> Lit components are standard custom elements, so the browser treats them exactly like built-in elements. Use them in hand-written HTML or framework code, output them from your CMS or static site builder, even create instances in JavaScript — they just work!</p> </li> <li> <p><strong>Scoped styles</strong> Lit scopes your styles by default, using Shadow DOM. This keeps your CSS selectors simple and ensures that your component’s styles don't affect — and aren't affected by — any other styles on the page.</p> </li> <li> <p><strong>Reactive properties</strong> Declare reactive properties to model your component’s API and internal state. A Lit component efficiently re-renders whenever a reactive property (or corresponding HTML attribute) changes.</p> </li> <li> <p><strong>Declarative templates</strong> Lit templates, based on tagged template literals, are simple, expressive and fast, featuring HTML markup with native JavaScript expressions inline. No custom syntax to learn, no compilation required.</p> </li> </ol> <pre><code>&lt;simple-greeting name="World"&gt;&lt;/simple-greeting&gt; </code></pre> https://tedneward.github.io/Research/presentation/webgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webgui/index.html <p><a href="https://jnmaloney.github.io/WebGui/imgui.html">Website</a> | <a href="https://github.com/jnmaloney/WebGui">Source</a></p> https://tedneward.github.io/Research/presentation/wiggle-ui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wiggle-ui/index.html <p><a href="https://wigggle-ui.vercel.app/">Website</a> | <a href="https://github.com/wigggle-ui/ui">Source</a></p> https://tedneward.github.io/Research/presentation/winui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/winui/index.html <p>"By incorporating the Fluent Design System into all experiences, controls, and styles, WinUI provides consistent, intuitive, and accessible experiences using the latest user interface (UI) patterns. With support for both desktop and UWP apps, you can build with WinUI from the ground up, or gradually migrate your existing MFC, WinForms, or WPF apps using familiar languages such as C++, C#, Visual Basic, and Javascript (via React Native for Windows)."</p> <p><a href="https://microsoft.github.io/microsoft-ui-xaml/">Website</a> | <a href="https://docs.microsoft.com/en-us/windows/apps/winui/">Docs</a> | <a href="https://github.com/microsoft/microsoft-ui-xaml">Source</a></p> <hr> <p><a href="https://dotmorten.github.io/WinUIEx/">WinUIEx</a>: A set of extension methods and classes to fill some gaps in WinUI 3, mostly around windowing and unit testing.</p> https://tedneward.github.io/Research/presentation/xforms/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/xforms/index.html <p><a href="https://en.wikipedia.org/wiki/XForms">Wikipedia</a> | <a href="http://www.w3.org/TR/xforms/">XForms 1.1</a> | <a href="https://www.w3.org/community/xformsusers/wiki/XForms_2.0">XForms 2.0 draft</a></p> https://tedneward.github.io/Research/presentation/zotonic/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/zotonic/index.html <p><a href="http://zotonic.com/">Website</a> | <a href="https://github.com/zotonic/zotonic">Source</a></p> https://tedneward.github.io/Research/presentation/webcomponents/polymer/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/polymer/index.html <p><a href="https://www.polymer-project.org/">Website</a></p> https://tedneward.github.io/Research/presentation/webix/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webix/index.html <p><a href="https://webix.com/">Website</a> | Commercial</p> https://tedneward.github.io/Research/presentation/wikijs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wikijs/index.html <p><a href="https://js.wiki/">Website</a> | <a href="https://github.com/Requarks/wiki">Source</a></p> https://tedneward.github.io/Research/presentation/wiredjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wiredjs/index.html <p><a href="https://wiredjs.com/">Website</a> | <a href="https://github.com/rough-stuff/wired-elements">Source</a> | <a href="https://wiredjs.com/showcase">Component Showcase</a></p> <p>Sandbox:</p> <ul> <li><a href="https://codesandbox.io/s/react-wrapper-for-wired-elements-vid1j">React</a></li> <li><a href="https://codesandbox.io/s/wired-elements-svelte-4hfkb">Svelte</a></li> <li><a href="https://codesandbox.io/s/wired-elements-vanilla-4bpny">Vanilla</a></li> <li><a href="https://codesandbox.io/s/vj389y9375">Vue</a></li> </ul> https://tedneward.github.io/Research/presentation/xmlui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/xmlui/index.html <p><a href="https://www.xmlui.org/">Website</a> | <a href="https://github.com/xmlui-org/xmlui">Source</a> | <a href="https://demo.xmlui.org/">Demos</a> | <a href="https://docs.xmlui.org/">Docs</a></p> <h2>Reading</h2> <ul> <li><a href="https://blog.jonudell.net/2025/07/18/introducing-xmlui/">Introducing XMLUI</a></li> <li><a href="https://blog.xmlui.org/blog/improving-the-xmlui-mcp-server">Improving the XMLUI MCP Server</a> - <a href="https://github.com/xmlui-org/xmlui-mcp">Source</a></li> </ul> <h2>Example</h2> <h3>Getting Started</h3> <p>The XMLUI engine is a single JS file. Here is the minimal XMLUI setup.</p> <pre><code>&lt;your app folder&gt; index.html Main.xmlui xmlui 0.9.90.js </code></pre> <p>index.html</p> <pre><code>&lt;!doctype html&gt; &lt;html lang="en"&gt; &lt;head&gt; &lt;meta charset="UTF-8" /&gt; &lt;meta name="viewport" content="width=device-width, initial-scale=1.0" /&gt; &lt;script src="xmlui/0.9.90.js"&gt;&lt;/script&gt; &lt;/head&gt; &lt;body&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>Main.xmlui</p> <pre><code>&lt;App name="XMLUI Hello World"&gt;Hello World&lt;/App&gt; </code></pre> <p>Automatic reactivity</p> <pre><code>&lt;App&gt; &lt;Select id="lines" initialValue="bakerloo"&gt; &lt;Items data="https://api.tfl.gov.uk/line/mode/tube/status"&gt; &lt;Option value="{$item.id}" label="{$item.name}" /&gt; &lt;/Items&gt; &lt;/Select&gt; &lt;DataSource id="tubeStations" url="https://api.tfl.gov.uk/Line/{lines.value}/Route/Sequence/inbound" resultSelector="stations"/&gt; &lt;Table data="{tubeStations}" height="280px"&gt; &lt;Column bindTo="name" /&gt; &lt;Column bindTo="modes" /&gt; &lt;/Table&gt; &lt;/App&gt; </code></pre> <h2>Components</h2> <p>APICall, App, AppHeader, AppState, AutoComplete, Avatar, Backdrop, Badge, BarChart, Bookmark, Breakout, Button, CHStack, CVStack, Card, ChangeListener, Checkbox, ColorPicker, Column, ContentSeparator, DataSource, DatePicker, DonutChart, DropdownMenu, EmojiSelector, FileInput, FileUploadDropZone, FlowLayout, Footer, Form, FormItem, FormSection, Fragment, H1, H2, H3, H4, H5, H6, HSplitter, HStack, Heading, Icon, Image, Items, LabelList, Legend, LineChart, Link, List, Logo, Markdown, MenuItem, MenuSeparator, ModalDialog, NavGroup, NavLink, NavPanel, NoResult, NumberBox, Option, Page, PageMetaTitle, Pages, PasswordInput, PieChart, ProgressBar, Queue, RadioGroup, RealTimeAdapter, Redirect, Select, Slider, SpaceFiller, Spinner, Splitter, Stack, StickyBox, SubMenuItem, Switch, TabItem, Table, TableOfContents, Tabs, Text, TextArea, TextBox, Theme, ToneChangerButton, VSplitter, VStack</p> <p>See the <a href="https://docs.xmlui.org/components/_overview">whole catalog</a>.</p> https://tedneward.github.io/Research/presentation/webcomponents/shoelace/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/shoelace/index.html <p><a href="https://shoelace.style/">Website</a> | <a href="https://github.com/shoelace-style/shoelace">Source</a></p> <p>Builds on top of <a href="/presentation/webcomponents">Web Components</a>.</p> https://tedneward.github.io/Research/presentation/websharper/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/websharper/index.html <p><a href="https://websharper.com/">Website</a> | <a href="https://github.com/dotnet-websharper">Source</a></p> <h3>From the Documentation/Basics:</h3> <h4>Hello World!</h4> <p>With WebSharper you can develop pure JS/HTML, and single- and multi-page web applications with an optional server side, all in F#. Unless you are looking for low-level control, we recommend that you start by creating a sitelet.</p> <p>The simplest sitelet serves text on a single endpoint at the root of the application:</p> <pre><code>module YourApp open WebSharper open WebSharper.Sitelets [&lt;Website&gt;] let Main = Application.Text (fun ctx -&gt; "Hello World!") </code></pre> <p><img src="http://i.imgur.com/fZgqeKjm.png" alt=""></p> <h4>Single Page Applications</h4> <p>While serving text is fun and often useful, going beyond isn’t any complicated. For instance, you can easily construct single-page applications:</p> <pre><code>module YourApp open WebSharper open WebSharper.Sitelets open WebSharper.UI.Html open WebSharper.UI.Server [&lt;Website&gt;] let Main = Application.SinglePage (fun ctx -&gt; Content.Page( h1 [] [text "Hello World!"] ) ) </code></pre> <p>This code creates an empty HTML document and inserts a header node.</p> <p><img src="http://i.imgur.com/xYITvCqm.png" alt=""></p> <h4>HTML responses</h4> <p>Pages are a special type of content responses, and you can easily finetune them by specifying where you want content to be added, by using an optional Title, Head, Body, and Doctype.</p> <pre><code> ... Application.SinglePage (fun ctx -&gt; Content.Page( Title = "My Hello World app", Body = [ h1 [text "Hello World!"] ], ... ) ) </code></pre> <p>You can construct HTML via the (soon legacy) WebSharper 3.x markup combinators in WebSharper.Html.Server and WebSharper.Html.Client (for client-side markup, see the section below), or using the next generation reactive HTML language from WebSharper UI (as above and in the examples on this page; formerly called UI.Next).</p> <h4>Custom responses</h4> <p>Content responses are asynchronous. Next to full HTML pages, you can return:</p> <ul> <li>Plain text with <code>Content.Text</code>:</li> </ul> <pre><code>Content.Text "Hello World!" </code></pre> <ul> <li>JSON values with <code>Content.Json</code> (visit JSON documentation or JSON cheatsheet for more info):</li> </ul> <pre><code>type Person = { First: string; Last: string; Age: int} Content.Json { First="John"; Last="Smith"; Age=30 } </code></pre> <ul> <li>Files with <code>Content.File</code>:</li> </ul> <pre><code>Content.File("Main.fs", ContentType="text/plain") </code></pre> <ul> <li>Various error codes:</li> </ul> <pre><code>Content.Unauthorized (401) Content.Forbidden (403) Content.NotFound (404) Content.MethodNotAllowed (405) Content.ServerError (500) </code></pre> <ul> <li>You can also create your own custom error code response:</li> </ul> <pre><code>Content.Custom(Status=Http.Status.Custom 402 (Some "Payment Required")) </code></pre> <ul> <li>Any other custom content with <code>Content.Custom</code>.</li> </ul> <h4>Multi-page applications</h4> <p>Multi-page applications have multiple endpoints: pairs of HTTP verbs and paths, and are represented as an annotated union type we typically call Endpoints (or Action in previous terminology). The endpoints, as defined by this union type - given the various annotations on each union case - are mapped to content to be served using Application.MultiPage. Links to endpoints in your site can be calculated from the serving context, so you will never have invalid URLs.</p> <pre><code>module YourApp open WebSharper open WebSharper.Sitelets open WebSharper.UI open WebSharper.UI.Html open WebSharper.UI.Server type Endpoints = | [&lt;EndPoint "GET /"&gt;] Home | [&lt;EndPoint "GET /about"&gt;] About [&lt;Website&gt;] let Main = Application.MultiPage (fun ctx endpoint -&gt; let (=&gt;) label endpoint = a [attr.href (ctx.Link endpoint)] [text label] match endpoint with | Endpoints.Home -&gt; Content.Page( Body = [ h1 [] [text "Hello world!"] "About" =&gt; Endpoints.About ] ) | Endpoints.About -&gt; Content.Page( Body = [ p [] [text "This is a simple app"] "Home" =&gt; Endpoints.Home ] ) ) </code></pre> <p><img src="http://i.imgur.com/WMnmzIPm.png" alt=""></p> <h4>Adding client-side functionality</h4> <p>WebSharper applications can easily incorporate client-side content, expressed in F#, giving an absolute edge over any web development library. Just mark your client-side functions or modules with [<javascript> ] and embed them into server side markup using client. Server-side RPC functions are annotated with [ <rpc> ]. </rpc> </javascript></p> <p>The example below is reimplemented from the blog entry Deploying WebSharper apps to Azure via GitHub, also available in the main WebSharper templates, and although it omits the more advanced templating in that approach (which is straightforward to add to this implementation), it should give you an recipe for adding client-side functionality to your sitelets easily.</p> <pre><code>module YourApp open WebSharper open WebSharper.Sitelets open WebSharper.UI open WebSharper.UI.Html open WebSharper.UI.Client module Server = [&lt;Rpc&gt;] let DoWork (s: string) = async { return System.String(List.ofSeq s |&gt; List.rev |&gt; Array.ofList) } [&lt;JavaScript&gt;] module Client = open WebSharper.JavaScript let Main () = let input = input [attr.value ""] [] let output = h1 [] [] div [ input button [ on.click (fun _ _ -&gt; async { let! data = Server.DoWork input.Value output.Text &lt;- data } |&gt; Async.Start ) ] [text "Send"] hr [] [] h4A [attr.``class`` "text-muted"] [text "The server responded:"] div [attr.``class`` "jumbotron"] [output] ] open WebSharper.UI.Server [&lt;Website&gt;] let MySite = Application.SinglePage (fun ctx -&gt; Content.Page( Body = [ h1 [] [text "Say Hi to the server"] div [] [client &lt;@ Client.Main() @&gt;] ] ) ) </code></pre> <p><img src="http://i.imgur.com/9sPa4lzm.png" alt=""></p> <h4>Using JavaScript libraries</h4> <p>WebSharper extensions bring JavaScript libraries to WebSharper. You can download extensions or develop your own using WIG, among others. Below is an example using WebSharper.Charting and chart.js underneath.</p> <p>Note that these and any other dependencies you may be using will be automatically injected into a Content.Page or other sitelet HTML response, and you will never have to deal with them manually.</p> <pre><code>module YourApp open WebSharper open WebSharper.Sitelets open WebSharper.UI open WebSharper.UI.Html [&lt;JavaScript&gt;] module Client = open WebSharper.JavaScript open WebSharper.UI.Client open WebSharper.Charting let RadarChart () = let labels = [| "Eating"; "Drinking"; "Sleeping"; "Designing"; "Coding"; "Cycling"; "Running" |] let data1 = [|28.0; 48.0; 40.0; 19.0; 96.0; 27.0; 100.0|] let data2 = [|65.0; 59.0; 90.0; 81.0; 56.0; 55.0; 40.0|] let ch = Chart.Combine [ Chart.Radar(Seq.zip labels data1) .WithFillColor(Color.Rgba(151, 187, 205, 0.2)) .WithStrokeColor(Color.Rgba(151, 187, 205, 1.)) .WithPointColor(Color.Rgba(151, 187, 205, 1.)) Chart.Radar(Seq.zip labels data2) .WithFillColor(Color.Rgba(220, 220, 220, 0.2)) .WithStrokeColor(Color.Rgba(220, 220, 220, 1.)) .WithPointColor(Color.Rgba(220, 220, 220, 1.)) ] Renderers.ChartJs.Render(ch, Size = Size(400, 400)) open WebSharper.UI.Server [&lt;Website&gt;] let MySite = Application.SinglePage (fun ctx -&gt; Content.Page( Body = [ h1 [] [text "Charts are easy with WebSharper Warp!"] div [] [client &lt;@ Client.RadarChart() @&gt;] ]) ) </code></pre> <p><img src="http://i.imgur.com/9o7x2b1m.png" alt=""></p> https://tedneward.github.io/Research/presentation/wikmd/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wikmd/index.html <p><a href="https://linbreux.github.io/wikmd/">Website</a> | <a href="https://github.com/Linbreux/wikmd">Source</a></p> <h2>Installation</h2> <h3>Docker-Compose</h3> <pre><code>--- version: "2.1" services: wikmd: image: linbreux/wikmd:latest container_name: wikmd environment: - PUID=1000 - PGID=1000 - TZ=Europe/Paris volumes: - /path/to/wiki:/wiki ports: - 5000:5000 restart: unless-stopped </code></pre> <h3>Docker (CLI)</h3> <pre><code>docker run -d \ --name wikmd \ -e TZ=Europe/Paris \ -e PUID=1000 \ -e PGID=1000 \ -e WIKMD_LOGGING=1 `#optional` \ -p 5000:5000 \ -v /path/to/wiki:/wiki \ --restart unless-stopped \ linbreux/wikmd:latest </code></pre> https://tedneward.github.io/Research/presentation/wxwidgets/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wxwidgets/index.html <p><a href="https://www.wxwidgets.org/">Website</a> | <a href="https://github.com/wxWidgets/wxWidgets">Source</a>: "When installing wxWidgets on Windows or macOS, we always recommend building the library from source yourself, and only provide the source package for most platforms. On some platforms, we have provided a few pre-built binaries for convenience, but wxWidgets supports so many compilers on so many platforms, that we can’t provide binaries for all of them. On Linux, we recommend using the official wxGTK packages provided by each distribution, but newer packages are available."</p> <p>It has popular language bindings for Python, Perl, Ruby and many other languages,</p> https://tedneward.github.io/Research/presentation/yaml/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/yaml/index.html <p><a href="http://www.yaml.de/">Website</a> | <a href="https://github.com/yamlcss/yaml">Source</a></p> https://tedneward.github.io/Research/presentation/webcomponents/kickstand/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/kickstand/index.html <p><a href="https://kickstand-ui.com/">Website</a> | <a href="https://github.com/break-stuff/kickstand-ui">Source</a></p> https://tedneward.github.io/Research/presentation/webcomponents/stencil/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/stencil/index.html <p><a href="https://stenciljs.com/docs/introduction">Website</a> | <a href="https://github.com/ionic-team/stencil">Source</a></p> <p>Stencil uses TypeScript, JSX, and CSS to create standards-based Web Components that can be used to craft high quality component libraries, design systems, and applications.</p> <p>Stencil generates standards-compliant Web Components that can work with popular frameworks right out of the box. In addition, Stencil can be used to generate framework native components that can be used just like any other components in your framework of choice. Stencil accomplishes this by wrapping your Web Components via Stencil's Output Target feature.</p> <p>Example:</p> <pre><code>import { Component, Prop, h } from '@stencil/core'; @Component({ tag: 'my-first-component', }) export class MyComponent { // Indicate that name should be a public property on the component @Prop() name: string; render() { return ( &lt;p&gt; My name is {this.name} &lt;/p&gt; ); } } </code></pre> <p>Once compiled, this component can be used in HTML just like any other tag.</p> <pre><code>&lt;my-first-component name="Max"&gt;&lt;/my-first-component&gt; </code></pre> https://tedneward.github.io/Research/presentation/wicket/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/wicket/index.html <p><a href="https://wicket.apache.org/">Website</a> | <a href="http://www.apache.org/dyn/closer.cgi/wicket/9.7.0">Source</a></p> <ul> <li><a href="http://wicket.apache.org/learn/#guide">Official Free Online Guide for Apache Wicket framework</a></li> </ul> https://tedneward.github.io/Research/presentation/winformswasm/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/winformswasm/index.html <p><a href="https://github.com/roozbehid/WasmWinforms">Source</a></p> https://tedneward.github.io/Research/presentation/xamarin/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/xamarin/index.html <p><a href="https://dotnet.microsoft.com/en-us/apps/xamarin">Website</a> | <a href="https://docs.microsoft.com/en-us/xamarin/">Docs</a> | <a href="https://www.github.com/xamarin">GitHub</a></p> <p>Xamarin has both native bindings to the underlying ecosystem (Xamarin.iOS for iOS and Xamarin.Android for Android) as well as a portable layer that abstracts away the underlying platform (Xamarin.Forms).</p> <h2>Reading</h2> <ul> <li><a href="https://developer.xamarin.com/guides/xamarin-forms/creating-mobile-apps-xamarin-forms/">Creating Mobile Apps with Xamarin.Forms C#</a></li> <li><a href="https://www.packtpub.com/free-ebook/mastering-xamarin-ui-development-second-edition/9781788995511">Mastering Xamarin UI Development, Second Edition</a> (account required)</li> <li><a href="https://www.syncfusion.com/ebooks/xamarin-forms-succinctly">Xamarin.Forms Succinctly</a>, <a href="https://www.syncfusion.com/ebooks/xamarin_forms_for_mac_os_succinctly">Xamarin.Forms for macOS Succinctly</a></li> </ul> <h2>Xamarin iOS</h2> <h2>Xamarin Android</h2> <h2>Xamarin Forms</h2> https://tedneward.github.io/Research/presentation/yoga/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/yoga/index.html <p><a href="https://www.yogalayout.dev/">Website</a></p> <p>Yoga is an embeddable layout system used in popular UI frameworks like React Native. Yoga itself is not a UI framework, and does not do any drawing itself. Yoga's only responsibility is determining the size and position of boxes.</p> <p>Yoga supports a familiar subset of CSS, mostly focused on Flexbox. This gives users a familiar model, and enables sharing code between native platforms and the browser.</p> <p>Yoga is written in C++, with a public C API. This allows Yoga to be used by a wide variety of languages, via both official and unofficial bindings.</p> https://tedneward.github.io/Research/presentation/python/kivy/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/kivy/index.html <p><a href="https://kivy.org/">Website</a> | <a href="https://github.com/kivy">Source</a> | <a href="https://kivy.org/doc/stable/">Docs</a></p> <p>Uses a "design language" (<a href="https://kivy.org/doc/stable/guide/lang.html">Kv Design Language</a>) to separate code from layout:</p> <pre><code>#:kivy 1.0 &lt;Controller&gt;: label_wid: my_custom_label BoxLayout: orientation: 'vertical' padding: 20 Button: text: 'My controller info is: ' + root.info on_press: root.do_action() Label: id: my_custom_label text: 'My label before button press' </code></pre> <p>... matched up with code...</p> <pre><code>import kivy kivy.require('1.0.5') from kivy.uix.floatlayout import FloatLayout from kivy.app import App from kivy.properties import ObjectProperty, StringProperty class Controller(FloatLayout): '''Create a controller that receives a custom widget from the kv lang file. Add an action to be called from the kv lang file. ''' label_wid = ObjectProperty() info = StringProperty() def do_action(self): self.label_wid.text = 'My label after button press' self.info = 'New info text' class ControllerApp(App): def build(self): return Controller(info='Hello world') if __name__ == '__main__': ControllerApp().run() </code></pre> https://tedneward.github.io/Research/presentation/qt/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/qt/index.html <p><a href="https://www.qt.io/">Website</a> | <a href="https://www.qt.io/download-open-source">Open Source Website</a> | <a href="https://wiki.qt.io/Building_Qt_5_from_Git#Getting_the_source_code">Source</a> | <a href="https://www.qt.io/download-qt-installer">Qt Installer</a></p> https://tedneward.github.io/Research/presentation/railo/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/railo/index.html <p><a href="https://github.com/getrailo/railo">Source</a></p> <p>Appears to be abandoned--no new development since 2013 or so.</p> https://tedneward.github.io/Research/presentation/react/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/react/index/index.html <p><a href="https://reactjs.org/">Website</a> | <a href="https://reactjs.org/docs/getting-started.html">Docs</a> | <a href="https://github.com/facebook/react/">Source</a> | <a href="https://github.com/facebook/react/tree/main/compiler">Compiler</a></p> <h3>Reading</h3> <ul> <li><a href="https://airbnb.io/javascript/react/">Airbnb React/JSX Style Guide</a> - Airbnb</li> <li><a href="http://www.hackingwithreact.com">Hacking with React</a></li> <li><a href="https://www.digitalocean.com/community/books/how-to-code-in-react-js-ebook">How To Code in React.js</a> - Joe Morgan</li> <li><a href="http://code.tutsplus.com/tutorials/intro-to-the-react-framework--net-35660">Intro to the React Framework</a></li> <li><a href="https://scotch.io/tutorials/learning-react-getting-started-and-concepts">Learning React.js: Getting Started and Concepts</a></li> <li><a href="https://github.com/vasanthk/react-bits">React-Bits</a></li> <li><a href="https://github.com/softchris/react-book/">React Book, your beginner guide to React</a> - Chris Noring</li> <li><a href="https://www.reactenlightenment.com">React Enlightenment</a></li> <li><a href="https://developmentarc.gitbooks.io/react-indepth/content/">React In-depth: An exploration of UI development</a></li> <li><a href="https://krasimir.gitbooks.io/react-in-patterns/content">React in patterns</a> - Krasimir Tsonev</li> <li><a href="https://goalkicker.com/ReactJSBook/">React JS Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="https://github.com/mikechau/react-primer-draft">React Primer Draft</a></li> <li><a href="https://www.syncfusion.com/ebooks/react-succinctly">React Succinctly</a> - Samer Buna</li> <li><a href="https://reactjs.org/tutorial/tutorial.html">React Tutorial</a></li> <li>React Tutorial by Josh Finnie</li> <li><a href="http://www.joshfinnie.com/blog/reactjs-tutorial-part-1/">Part 1</a></li> <li><a href="http://www.joshfinnie.com/blog/reactjs-tutorial-part-2/">Part 2</a></li> <li><a href="http://www.joshfinnie.com/blog/reactjs-tutorial-part-3/">Part 3</a></li> <li><a href="https://reactjs.net/getting-started/aspnetcore.html">React with ASP.NET Core Tutorial</a></li> <li><a href="https://codegeekz.com/react-js-tutorial/">React.js Tutorial: Now is Your Time to Try It, Right in Your Browser</a></li> <li><a href="http://survivejs.com">SurviveJS - Webpack and React</a></li> <li><a href="https://flaviocopes.com/page/react-handbook/">The React Beginner's Handbook</a> - Flavio Copes (PDF, EPUB, Kindle) <em>(email address requested)</em></li> </ul> <h2>Supplementals</h2> <p><a href="https://mobx.js.org/README.html">MobX.js</a>: Simple, scalable state management.</p> https://tedneward.github.io/Research/presentation/reactiveui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/reactiveui/index.html <p><a href="https://www.reactiveui.net/">Website</a> | <a href="https://github.com/reactiveui">Source</a></p> <blockquote> <p>An advanced, composable, functional reactive model-view-viewmodel framework for all .NET platforms that is inspired by functional reactive programming. ReactiveUI allows you to abstract mutable state away from your user interfaces, express the idea around a feature in one readable place and improve the testability of your application.</p> </blockquote> <p>Includes the <a href="../storage/akavache">Akavache</a> project</p> <p><a href="https://github.com/reactiveui/refit">Refit</a>: The automatic type-safe REST library for .NET Core, Xamarin and .NET. Heavily inspired by Square's Retrofit library, Refit turns your REST API into a live interface.</p> https://tedneward.github.io/Research/presentation/retejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/retejs/index.html <p><a href="https://retejs.org/">Website</a> | <a href="https://github.com/retejs/rete">Source</a> | <a href="https://retejs.org/docs/development/rete-kit">Docs</a></p> https://tedneward.github.io/Research/presentation/sciter/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/sciter/index.html <p><a href="https://sciter.com/">Website</a> | <a href="https://github.com/c-smile/sciter-sdk">Github</a></p> <p>C-based API. <a href="https://sciter.com/hello-cpp-tutorial/">C++ Tutorial</a></p> <p>Used to be built with <a href="/languaes/tiscript">TIScript</a></p> <h2>Apps built with Sciter</h2> <ul> <li><a href="https://notes.sciter.com/">Sciter Notes</a></li> </ul> https://tedneward.github.io/Research/presentation/sequential-workflow-designer/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/sequential-workflow-designer/index.html <p><a href="https://nocode-js.com/">Website</a> | <a href="https://github.com/nocode-js/sequential-workflow-designer">Source</a></p> <p>Works with <a href="https://github.com/nocode-js/sequential-workflow-machine">Sequential Workflow Machine</a></p> https://tedneward.github.io/Research/presentation/sitejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/sitejs/index.html <p><a href="https://sitejs.org/">Website</a> | <a href="https://github.com/small-tech/site.js">Source</a></p> <blockquote> <p>The Small Web is for people (not startups, enterprises, or governments). One person. One server. One site. Develop, test, sync, and deploy (using a single tool that comes in a single binary). Comes bundled with <a href="../hugo">Hugo</a>. Dynamic sites use DotJS ("PHP-like simplicity in JavaScript using .js files"), and includes WebSockets support as well. Comes bundled with <a href="../../storage/jsdb">JavaScript Database (JSDB)</a> built into it.</p> </blockquote> https://tedneward.github.io/Research/presentation/solidjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/solidjs/index.html <p><a href="https://www.solidjs.com/">Website</a></p> https://tedneward.github.io/Research/presentation/stilljs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/stilljs/index.html <p><a href="https://stilljs.dev/">Website</a> | <a href="https://github.com/still-js">Source</a> | <a href="https://github.com/still-js/samples">Samples</a></p> <h2>Reading</h2> <ul> <li><a href="https://thenewstack.io/tired-of-javascript-framework-complexity-meet-still-js/">https://thenewstack.io/tired-of-javascript-framework-complexity-meet-still-js/</a></li> </ul> https://tedneward.github.io/Research/presentation/taipy/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/taipy/index.html <p><a href="https://taipy.io/">Website</a> | <a href="https://github.com/Avaiga/taipy">Source</a></p> <p><a href="https://taipy.io/enterprise">Taipy Designer</a> (Commercial)</p> https://tedneward.github.io/Research/presentation/tiddlywiki/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tiddlywiki/index.html <p><a href="https://tiddlywiki.com/">Website/Demo</a> | <a href="https://github.com/TiddlyWiki/TiddlyWiki5">Source</a> | <a href="https://tiddlywiki.com/#TiddlyDesktop">Desktop</a></p> <p>TiddlyWiki is a complete interactive wiki in JavaScript. It can be used as a single HTML file in the browser or as a powerful Node.js application. It is highly customisable: the entire user interface is itself implemented in hackable WikiText.</p> <p><a href="https://kookma.github.io/TW-Scripts/">Scripts in TiddlyWiki</a></p> <p>It can be run client-server mode in nodejs:</p> <h1><a href="https://github.com/TiddlyWiki/TiddlyWiki5?tab=readme-ov-file#installing-tiddlywiki-on-nodejs">Installing TiddlyWiki on Node.js</a></h1> <p>TiddlyWiki is a <a href="https://tiddlywiki.com/static/SingleFileApplication.html">SingleFileApplication</a>, which is easy to use. For advanced users and developers there is a possibility to use a Node.js client / server configuration. This configuration is also used to build the TiddlyWiki <a href="https://tiddlywiki.com/static/SinglePageApplication.html">SinglePageApplication</a></p> <ol> <li>Install <a href="https://tiddlywiki.com/static/Node.js.html">Node.js</a> <ul> <li>Linux: <blockquote> <p><em>Debian/Ubuntu</em>:<br> <code>apt install nodejs</code><br> May need to be followed up by:<br> <code>apt install npm</code></p> <p><em>Arch Linux</em><br> <code>yay -S tiddlywiki</code><br> (installs node and tiddlywiki)</p> </blockquote> </li> <li>Mac <blockquote> <p><code>brew install node</code></p> </blockquote> </li> <li>Android <blockquote> <p><a href="https://tiddlywiki.com/static/Serving%2520TW5%2520from%2520Android.html">Termux for Android</a></p> </blockquote> </li> <li>Other <blockquote> <p>See <a href="http://nodejs.org/">http://nodejs.org</a></p> </blockquote> </li> </ul> </li> <li>Open a command line terminal and type: <blockquote> <p><code>npm install -g tiddlywiki</code></p> <p>If it fails with an error you may need to re-run the command as an administrator:</p> <p><code>sudo npm install -g tiddlywiki</code> (Mac/Linux)</p> </blockquote> </li> <li>Ensure TiddlyWiki is installed by typing: <blockquote> <p><code>tiddlywiki --version</code></p> </blockquote> <ul> <li>In response, you should see <a href="https://tiddlywiki.com/static/TiddlyWiki.html">TiddlyWiki</a> report its current version (eg "5.3.8". You may also see other debugging information reported.)</li> </ul> </li> <li>Try it out: <ol> <li><code>tiddlywiki mynewwiki --init server</code> to create a folder for a new wiki that includes server-related components</li> <li><code>tiddlywiki mynewwiki --listen</code> to start <a href="https://tiddlywiki.com/static/TiddlyWiki.html">TiddlyWiki</a></li> <li>Visit <a href="http://127.0.0.1:8080/">http://127.0.0.1:8080/</a> in your browser</li> <li>Try editing and creating tiddlers</li> </ol> </li> <li>Optionally, make an offline copy: <ul> <li>click the <strong>save changes</strong> button in the sidebar, <strong>OR</strong></li> <li><code>tiddlywiki mynewwiki --build index</code></li> </ul> </li> </ol> <p>The <code>-g</code> flag causes <a href="https://tiddlywiki.com/static/TiddlyWiki.html">TiddlyWiki</a> to be installed globally. Without it, <a href="https://tiddlywiki.com/static/TiddlyWiki.html">TiddlyWiki</a> will only be available in the directory where you installed it.</p> <p><strong>Warning</strong></p> <p>If you are using Debian or Debian-based Linux and you are receiving a <code>node: command not found</code> error though node.js package is installed, you may need to create a symbolic link between <code>nodejs</code> and <code>node</code>. Consult your distro's manual and <code>whereis</code> to correctly create a link. See github <a href="http://github.com/TiddlyWiki/TiddlyWiki5/issues/1434">issue 1434</a>.</p> <p>Example Debian v8.0: <code>sudo ln -s /usr/bin/nodejs /usr/bin/node</code></p> <p><strong>Tip</strong></p> <p>You can also install prior versions like this:<br> <code>npm install -g tiddlywiki@5.1.13</code></p> <hr> <h2><a href="https://tiddlywiki.com/#Community">Community Resources</a></h2> <h3><a href="https://tiddlywiki.com/#Saving%20to%20a%20Git%20service">Saving to a Git service</a></h3> <p>Saving to a Git service is configured in the $:/ControlPanel in the Git Service Saver tab under the Saving tab. The following settings are supported:</p> <ul> <li>Type - (mandatory) the type of the service (e.g. GitHub, GitLab)</li> <li>Username - (mandatory) the username for the Git service account used for saving changes</li> <li>Password - (mandatory) the OAUTH token or personal access token for the specified account. Note that GitHub deprecated password authentication, permitted authentication methods are shown in the API documentation.</li> <li>Repository - (mandatory) the name of the Git repository. Both the owner name and the repository name must be specified. For example Jermolene/TiddlyWiki5</li> <li>Branch - (optional) the name of the branch to be used within the Git repository. Defaults to main (GitHub) or master (GitLab)"</li> <li>Path - (optional) the path to the target file. Defaults to /</li> <li>Filename - (mandatory) the filename of the target file</li> </ul> <h3><a href="https://tiddlywiki.com/#GitHub%20Saver%20Tutorial%20by%20Mohammad">GitHub Saver Tutorial by Mohammad</a></h3> <p>GitHub Saver is a step by step tutorial that shows how to integrate Tiddlywiki 5 and GitHub Pages to create websites hosted on <a href="https://github.com/">https://github.com/</a>.</p> <p><a href="https://kookma.github.io/TW5-GitHub-Saver/">https://kookma.github.io/TW5-GitHub-Saver/</a></p> <p>This instruction is based on Tiddlywiki single html file model, while it can use subfolder for extra materials like images, audios, videos, pdfs,... in separate folders.</p> <p>Other tutorials</p> <ul> <li> <p>Tiddlywiki, Travis-CI and GitHub Pages</p> <p><a href="https://kookma.github.io/Tiddlywiki-Travis-CI/">https://kookma.github.io/Tiddlywiki-Travis-CI/</a></p> <p>This wiki shows how to set up websites hosted on GitHub Pages using Travis-CI and Tiddlywiki 5 on Node.js.</p> </li> <li> <p>Tiddlywiki and GitHub Pages</p> <p><a href="https://kookma.github.io/Tiddlywiki-and-GitHub-Pages/">https://kookma.github.io/Tiddlywiki-and-GitHub-Pages/</a></p> <p>This instruction is based on local edit, save and push to GitHub. It does NOT use the new GitHub Saver mechanism (requires TW 5.1.20+) which lets edit and save directly from Tiddlywiki!</p> </li> </ul> <h3><a href="https://tiddly.packett.cool/">TiddlyPWA</a></h3> <p><a href="https://codeberg.org/valpackett/tiddlypwa">Source</a></p> <p>Runs TW as a progressive web app. Secure offline storage and cross-device synchronization solution for TiddlyWiki 5.</p> https://tedneward.github.io/Research/presentation/togetherjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/togetherjs/index.html <p><a href="https://togetherjs.com/">Website</a> | <a href="https://github.com/mozilla/togetherjs">Source</a></p> <p>Some built=in features:</p> <ul> <li>Audio chat: TogetherJS uses Web RTC technology to enhance communication for your users.</li> <li>User focus: Your users see each other's mouse cursors and clicks.</li> <li>User presence: TogetherJS enables your users to see each other in real time.</li> <li>Text chat: Your users can chat with each other with familiar instant messaging.</li> <li>Co-browsing: Your users can follow each other to different pages on the same domain.</li> <li>Real time content sync: Your users can see content on a site or app dynamically change together.</li> </ul> https://tedneward.github.io/Research/presentation/ultralight/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ultralight/index.html <p><a href="https://ultralig.ht/">Website</a> | Commercial/closed-source</p> <p>Pros: battle tested / production ready, html/css/js, good documentation and multiple samples, supports both application and game scenarios (render to texture)<br> Cons: non-platform theme (expected with css), usual overhead of js &lt;-&gt; c++ integration</p> https://tedneward.github.io/Research/presentation/vanilla/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/vanilla/index.html <p><a href="https://vanillaforums.com/">Website</a> | <a href="https://github.com/vanilla/vanilla">Source</a></p> https://tedneward.github.io/Research/presentation/vue/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/vue/index/index.html <ul> <li><a href="https://www.newline.co/30-days-of-vue">30 Days Of Vue</a> - Hassan Djirdeh (HTML; <em>email required for PDF</em>)</li> <li><a href="https://riptutorial.com/Download/vue-js.pdf">Learning Vue.js</a> (PDF)</li> <li><a href="https://flaviocopes.nyc3.digitaloceanspaces.com/vue-handbook/vue-handbook.pdf">The Vue.js Handbook</a> - Flavio Copes (PDF)</li> </ul> https://tedneward.github.io/Research/presentation/python/nicegui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/nicegui/index.html <p><a href="https://nicegui.io/">Website</a> | <a href="https://github.com/zauberzeug/nicegui">Source</a></p> <p>Create web-based user interfaces with Python. "The nice way."</p> <p>NiceGUI is an easy-to-use, Python-based UI framework, which shows up in your web browser. You can create buttons, dialogs, Markdown, 3D scenes, plots and much more.</p> <p>It is great for micro web apps, dashboards, robotics projects, smart home solutions and similar use cases. You can also use it in development, for example when tweaking/configuring a machine learning algorithm or tuning motor controllers.</p> <h4>Installation</h4> <p><code>python3 -m pip install nicegui</code></p> <h4>Usage</h4> <p>Write your nice GUI in a file main.py:</p> <pre><code>from nicegui import ui ui.label('Hello NiceGUI!') ui.button('BUTTON', on_click=lambda: ui.notify('button was pressed')) ui.run() </code></pre> <p>Launch it with:</p> <p><code>python3 main.py</code></p> <p>The GUI is now available through <a href="http://localhost:8080/">http://localhost:8080/</a> in your browser. Note: NiceGUI will automatically reload the page when you modify the code.</p> https://tedneward.github.io/Research/presentation/quarto/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/quarto/index.html <p><a href="https://quarto.org/">Website</a> | <a href="https://quarto.org/docs/get-started/">Getting Started</a></p> <ul> <li>Author using Jupyter notebooks or with plain text markdown in your favorite editor.</li> <li>Create dynamic content with Python, R, Julia, and Observable.</li> <li>Publish reproducible, production quality articles, presentations, dashboards, websites, blogs, and books in HTML, PDF, MS Word, ePub, and more.</li> <li>Share knowledge and insights organization-wide by publishing to Posit Connect, Confluence, or other publishing systems.</li> <li>Write using Pandoc markdown, including equations, citations, crossrefs, figure panels, callouts, advanced layout, and more.</li> </ul> https://tedneward.github.io/Research/presentation/ratchet/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ratchet/index.html <p><a href="http://goratchet.com/">Website</a> | <a href="https://github.com/twbs/ratchet">Source</a></p> <p>Built by Twitter?</p> https://tedneward.github.io/Research/presentation/react/native/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/react/native/index.html <p><a href="https://reactnative.dev/">Website</a> | <a href="https://github.com/facebook/react-native">Source</a></p> <h3>Reading</h3> <ul> <li><a href="https://leanpub.com/programming-react-native">Programming React Native</a> <em>(Leanpub account or valid email requested)</em></li> <li><a href="http://browniefed.com/react-native-animation-book/">React Native Animation Book</a></li> <li><a href="http://www.reactnativeexpress.com">React Native Express</a></li> <li><a href="https://goalkicker.com/ReactNativeBook">React Native Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://www.gitbook.com/book/unbug/react-native-training/details">React Native Training</a></li> </ul> https://tedneward.github.io/Research/presentation/refactoring-to-html/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/refactoring-to-html/index.html <p><em>(by Elliott Rusty Harold (Addison-Wesley, 2008, ISBN-13 978-0-321-50363-3))</em></p> <h1>Well-Formedness</h1> <ul> <li>Change Name to Lowercase (69)</li> <li>Quote Attribute Value (73)</li> <li>Fill In Omitted Attribute Value (76)</li> <li>Replace Empty Tag with Empty-Element Tag (78)</li> <li>Add End-tag (82)</li> <li>Remove Overlap (86)</li> <li>Convert Text to UTF-8 (89)</li> <li>Escape Less-Than Sign (91)</li> <li>Escape Ampersand (93)</li> <li>Escape Quotation Marks in Attribute Values (96)</li> <li>Introduce an XHTML DOCTYPE Declaration (98)</li> <li>Terminate Each Entity Reference (101)</li> <li>Replace Imaginary Entity References (102)</li> <li>Introduce a Root Element (103)</li> <li>Introduce the XHTML Namespace (104)</li> </ul> <h1>Validity</h1> <ul> <li>Introduce a Transitional DOCTYPE Declaration</li> <li>Remove all Nonexistent Tags</li> <li>Add an alt Attribute</li> <li>Replace embed with object</li> <li>Introduce a Strict DOCTYPE Declaration</li> <li>Replace center with CSS</li> <li>Replace font with CSS</li> <li>Replace i with em or CSS</li> <li>Replace b with strong or CSS</li> <li>Replace the color Attribute with CSS</li> <li>Convert img Attributes to CSS</li> <li>Replace applet with object</li> <li>Replace Presentational Elements with CSS</li> <li>Nest Inline Elements inside Block Elements</li> </ul> <h1>Layout</h1> <ul> <li>Replace Table Layouts</li> <li>Replace Frames with CSS Positions</li> <li>Move Content to the Front</li> <li>Mark up Lists as Lists</li> <li>Replace blockquote/ul Indentation with CSS</li> <li>Replace Spacer GIFs</li> <li>Add an ID Attribute</li> <li>Add Width and Height to an Image</li> </ul> <h1>Accessibility</h1> <ul> <li>Convert Images to Text</li> <li>Add Labels to Form Input</li> <li>Introduce Standard Field Names</li> <li>Turn on Autocomplete</li> <li>Add Tab Indexes to Forms</li> <li>Introduce Skip Navigation</li> <li>Add Internal Headings</li> <li>Move Unique Content to the Front of Links and Headlines</li> <li>Make the Input Field Bigger</li> <li>Introduce Table Descriptions</li> <li>Introduce Acronym Elements</li> <li>Introduce lang Attributes</li> </ul> <h1>Web Applications</h1> <ul> <li>Replace Unsafe GET with POST</li> <li>Replace Safe POST with GET</li> <li>Redirect POST to GET</li> <li>Enable Caching</li> <li>Prevent Caching</li> <li>Introduce ETag</li> <li>Replace Flash with HTML</li> <li>Add Web Forms 2.0 Types</li> <li>Replace Contact Forms with mailto Links</li> <li>Block Robots</li> <li>Escape User Input</li> </ul> <h1>Content</h1> <ul> <li>Correct Spelling</li> <li>Repair Broken Links</li> <li>Remove the Entry Page</li> <li>Hide E-mail Addresses</li> </ul> https://tedneward.github.io/Research/presentation/revealjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/revealjs/index.html <p><a href="https://revealjs.com/">Website</a> | <a href="https://github.com/hakimel/reveal.js">Source</a></p> https://tedneward.github.io/Research/presentation/scratch/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/scratch/index.html <p><a href="https://scratch.mit.edu/">Website</a> | <a href="https://github.com/LLK/scratch-blocks">Source</a> | <a href="https://github.com/LLK/scratch-www">Scratch-WWW</a></p> <p><a href="https://scratch.mit.edu/developers">Scratch for developers</a></p> <h2>Learning Principles</h2> <h3>Projects</h3> <p>People learn best when they are actively working on projects — generating new ideas, designing prototypes, making improvements and creating final products.</p> <h3>Passion</h3> <p>When people focus on things they care about, they work longer and harder, persist in the face of challenges, and learn more in the process.</p> <h3>Peers</h3> <p>Learning flourishes as a social activity, with people sharing ideas, collaborating on projects, and building on one another's work.</p> <h3>Play</h3> <p>Learning involves playful experimentation — trying new things, tinkering with materials, testing boundaries, taking risks, iterating again and again.</p> <h2>Design Principles</h2> <h3>Low Floor &amp; Wide Walls</h3> <p>In order to encourage a varied and diverse set of interactions, we explicitly include elements and features that are easy for kids to understand (low floor), but general enough to support diverse uses (wide walls).</p> <h3>Make it as Simple as Possible — And Maybe Even Simpler</h3> <p>Despite the common drive to add more features to software products, we have found that reducing the number of features often improves the user experience. What initially seems like a constraint or limitation can foster new forms of creativity.</p> <h3>Many Paths, Many Styles</h3> <p>Many math and science activities have traditionally been biased towards specific populations. By paying special attention to creating accessible and appealing technologies, we are working to close the gap.</p> <h3>Design for Tinkerability</h3> <p>We believe that the learning process is inherently iterative. Tinkerers start by exploring and experimenting, then revising and refining their goals and creations. To support this style of interaction, we design our interfaces to encourage quick experimentation and rapid cycles of iteration.</p> https://tedneward.github.io/Research/presentation/server-driven-ui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/server-driven-ui/index.html <h2>Implementations</h2> <ul> <li><a href="https://github.com/csmets/Server-Driven-UI">https://github.com/csmets/Server-Driven-UI</a>: A framework example for Server Driven UI (SDUI) that teaches you the best practices to scale. <a href="https://github.com/csmets/Server-Driven-UI/blob/master/docs/README.md">Docs</a></li> </ul> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.apollographql.com/docs/graphos/schema-design/guides/sdui/basics">Server-Driven UI Basics</a>: From a GraphQL vendor.</li> <li><a href="https://engineeringblog.yelp.com/2024/03/chaos-yelps-unified-framework-for-server-driven-ui.html">CHAOS: Yelp's Unified Framework for Server-Driven UI</a>: "In late 2021, we began building a unified SDUI framework called CHAOS or “Content Hosting Architecture with Optimization Strategies”."</li> <li><a href="https://getstream.io/blog/server-driven-compose-firebase/">Design Server-Driven UI with Jetpack Compose and Firebase</a> <a href="https://github.com/skydoves/server-driven-compose">Source</a></li> </ul> https://tedneward.github.io/Research/presentation/skeleton/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/skeleton/index.html <p><a href="http://getskeleton.com/">Website</a> | <a href="https://github.com/dhg/Skeleton">Source</a></p> https://tedneward.github.io/Research/presentation/spice/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/spice/index.html <p><a href="https://github.com/jonathanpeppers/spice">Source</a></p> https://tedneward.github.io/Research/presentation/supercollider/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/supercollider/index.html <p><a href="https://supercollider.github.io/">Website</a> | <a href="https://github.com/supercollider/supercollider">Source</a> | <a href="https://docs.supercollider.online/Tutorials/Getting-Started/01-Introductory-Remarks.html">sclang Tutorial</a></p> <p>SuperCollider is actually three programs:</p> <ul> <li>The text editor you are looking at (also referred to as the IDE or Integrated Development Environment),</li> <li>the language (sclang or the "client" app),</li> <li>and the server, which does the actual synthesis and calculation of audio.</li> </ul> <p>The sclang part is a sophisticated programming language with nice features for building GUIs (Graphical User Interfaces); and the server part is a lean, mean, efficient UNIX command line application (meaning it runs without any GUI representation).</p> <p>They communicate by a protocol called OSC (Open Sound Control), over either UDP (User Datagram Protocol) or TCP (Transmission Control Protocol), which are network protocols also used on the internet. Because the client and server communicate this way, more advanced projects might run them on separate computers for performance reasons. In fact, it's even possible that they could be running in different parts of the world! However, just because these two applications communicate using common internet protocols does not mean they must be connected to the internet or on different computers. Most of the time they will be running on the same computer, and the "networking" aspect of things will be relatively transparent for you. Especially while you're still getting started.</p> <p>You can only communicate with the server using OSC messages over the network, but luckily the language app has lots of powerful objects which represent things on the server and allow you to control them easily and elegantly.</p> <h2>Examples</h2> <p>Hello, world</p> <pre><code>"Hello World!".postln; </code></pre> https://tedneward.github.io/Research/presentation/tauri/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tauri/index.html <p><a href="https://tauri.studio/en/">Website</a> | <a href="https://github.com/tauri-apps/tauri">Source</a></p> <p>Looks to be a competitor to Electron but aiming to be pluggable in a number of places (which I'm not sure is a win).</p> https://tedneward.github.io/Research/presentation/tinywiki/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tinywiki/index.html <p><a href="https://github.com/Kartones/tiny-wiki">Source</a></p> <p>Renders markdown documents provided by items.json, with a list of all documents at the left, and a list of the headings (levels 2+) at the right. No frameworks, just a bit of Javascript, HTML 5 and CSS.</p> https://tedneward.github.io/Research/presentation/twilio/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/twilio/index.html <p><a href="https://www.twilio.com/">Website</a></p> https://tedneward.github.io/Research/presentation/uno/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/uno/index.html <p><a href="https://platform.uno/">Website</a> | <a href="https://github.com/nventive/Uno">Source</a> | <a href="https://gallery.platform.uno/">Gallery</a></p> <p>C# and XAML</p> https://tedneward.github.io/Research/presentation/vanillajs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/vanillajs/index.html <p><a href="http://vanilla-js.com/">Website</a> | <a href="https://vanillajstoolkit.com/">VanillaJS Toolkit</a> (ignore the homepage, look at the menu at the top) | <a href="https://github.com/vanillawc/">VanillaJS Components</a></p> <p><a href="https://youtu.be/VvOsegaN9Wk">"A Framework Author's Case Against Frameworks"</a>: Adrian Holovaty, dotJS 2017</p> <p><a href="https://javarome.medium.com/design-noframework-bbc00a02d9b3">"Design: #noFramework"</a>:</p> <ul> <li>speaks principally to Web frameworks, but I'd argue that this can be true of any framework (thinking Java server-side Web frameworks, C++ GUI frameworks, server-side transaction processing frameworks, ...).* <ul> <li> <p>brings to mind Vlissides' pattern of <a href="/reading/patterns/evolving-frameworks">evolving frameworks</a>*</p> </li> <li> <p>Also brings to mind the Windows++ story: build your own to understand what's going on underneath--not quite the same point as this author, but certainly a viable line of thought/approach.*</p> </li> <li> <p>Drawbacks of frameworks:</p> <ul> <li> <p>The Framework Tax: frameworks require you to:</p> <ul> <li> <p><strong>comply</strong> with their API so that they can provide your their services. This is just the way a framework works: your code will have to adhere to some rules, including more or less boilerplate code. So it’s the framework way, or the highway. Your daily challenges will be less about “how to do this” than “how to make the framework (not) do this”. Dodge those constraints at your own risks: if you bypass a framework by directly calling low-level APIs, don’t expect it to understand what you’re trying to do, don’t expect it to stay consistent. So it’s a false promise frameworks make that you’ll be “focusing on your business”: in reality you have to care on the framework too, and a lot.</p> </li> <li> <p>upgrades are effectively forced if you:<br> 1) want a new feature (even if you didn’t wanted all those of the next release, you need to upgrade the whole thing) or<br> 2) want a bugfix, or<br> 3) want to avoid loosing support (as new versions are shipped, the one on which you have based your app will get deprecated).<br> Upgrades can also be lacking and let you frustrated (and possibly with a project at risk) with an identified bug but no planned date for a fix. Third-party framework-specific libraries (such as widgets) or plugins are no exception to that rule and will be less and less compatible with your app if you keep using old versions. Maintaining backward compatibility has became such a hassle for frameworks maintainers that they now find more profitable to work on tools that automate upgrades of your code as much as possible (Angular’s ng-update, React native Upgrade helper, Facebook’s jscodeshift, etc.).</p> </li> <li> <p><strong>train</strong> to learn how they work (what they can/cannot do, what are their concepts, APIs, ecosystem, tools), including changes that may occur in new versions. Should you pick the most popular framework of the day, this might be easier, but it’s unlikely that you’ll ever know about every aspects of a given framework. Also, hype comes and goes: should you decide to use another framework for a new app (or even worse, to migrate from one to another), the cost of investing in such proprietary knowledge will be lost. This explains a lot of inertia in enterprise projects, even if each project is different than the previous one. “Compatibility means deliberately repeating other people’s mistakes,” said the late David Wheeler.</p> </li> <li> <p><strong>compromise</strong> with the drawbacks implied by delegating control: you may not be able to do whatever you want (or to prevent the framework from doing things you do not want) or you may not achieve the performance you want (because of additional layering, too-generic code, bigger code size or backward compatibility requirements).</p> </li> <li> <p><strong>lose</strong> skills. A number of developers either don’t know much about the lower-level APIs (because they always used the framework layer instead) or live in the past (i.e. are stick on an outdated knowledge of it, not being aware of the latest improvements and new capabilities). The <a href="https://en.wikipedia.org/wiki/Law_of_the_instrument">law of the instrument</a> then leads too often to build overkill solutions to simple problems, and loose (if even once acquired) knowledge to build simpler ones. Being guided by blueprints and recipes, they loose (or not gain) a culture of good software design (principles, patterns) and barely build a significant engineering experience. Just like users of CSS frameworks (Bootstrap, Tailwind, etc.) lack CSS skills, users of web frameworks are doomed to lack experience in both modern web APIs and software design in general.</p> </li> </ul> </li> <li> <p>The Framework Silo: Aside the “tax” that you have to pay to get their benefits, frameworks can also induce an additional major issue when they are not standard. As they enforce rules — but each one of them is different — this implies binding your app with a proprietary ecosystem. That means locking your app code with a proprietary API (and its upgrade process). That’s a risky bet for your project.</p> </li> <li> <p>Are languages (CoffeeScript, TypeScript, Dart, etc) subject to the same problems as frameworks? Yup.</p> </li> </ul> </li> <li> <p>Frameworks are a good thing if they:</p> <ul> <li>are app-specific: any app ends up designing its own “business” framework.</li> <li>are standard or end up with a standard. For instance the web platform is a standard web framework, and Web Components frameworks (lit, stencil, skatejs, etc.) end up building components that comply with the standard.</li> <li>add some unique value that you’re missing in all other alternatives (including other frameworks). In such a case you have almost no choice, as the unique added value justifies the implied cost of locking with it. For instance, an OS-specific framework makes sense since it enforce OS standards and there is no other other way to provide an app or extension for it.</li> <li>are used to build non-critical apps (short lived, with lower quality expectations) where tax and silo effect are acceptable. For instance it makes sense to use Bootstrap to build some prototype, MVP or internal tool.</li> </ul> <p>So, in a nutshell, avoiding a framework to build an app aims to:</p> <ul> <li>maximize flexibility by avoiding “one size fits all” constraints from frameworks. Also, not having blueprints avoids the law of the instrument to increase the creativity for ambitious applications. Most web apps using Bootstrap can be recognized as such, because they’re having a hard time getting out of the predefined components and styles. In the end, they’ll have a hard time thinking another way.</li> <li>minimize dependency to any of the currently hyped frameworks. Not being locked with a framework avoids issues with portability and interoperability.</li> <li>maximize performance by allowing the most fine-grained operations only when required (no framework-dependent refresh cycle for instance) and reducing dependencies to a selection of precise, required-only, set of lightweight libraries.</li> </ul> </li> <li> <p>Building an app without a framework:</p> <ul> <li> <p>Goals and mindset: We must clarify the anti-goal: “building an app without a framework” is NOT to be confused with “replacing the framework”. This is not the challenge at stake: a framework is a general purpose technical solution to host virtually any app, so it is less about your app than all apps. On the opposite, going vanilla is an opportunity to focus on your app’s needs only.</p> <p>This is an important scope narrowing to make to assess the (non-)difficulty of building your app without a framework: it is not as hard as building a framework, because you do NOT aim to build:</p> <ul> <li>a proprietary component model (a container implementing a specific components lifecycle)</li> <li>a proprietary plugins/extension system :</li> <li>a fancy template syntax (JSX, Angular HTML, etc.)</li> <li>optimizations that make sense for general-purpose (change detection, virtual DOM)</li> <li>framework-specific tools (debugging extensions, UI builders, version migration tools)</li> </ul> <p>So building a vanilla app is not an enormous task of “reinventing the wheel” as often caricatured, because the major part of this “wheel” is actually about the APIs/contracts, their implementations, the general-purpose engine and associated optimizations, the debugging capabilities, etc.. Leaving the general-purpose goal and focusing on your app’s goals means that you can rid of most of it. Ironically, this is the real “focus on your app” approach.</p> <p><strong>change your state of mind</strong>: don’t look for the framework-specific services mentioned above. As a vanilla app, you will probably don’t need it. Don’t think change detection, just update the DOM, etc.</p> <p><strong>use technical alternatives</strong> for the common tasks you performed with frameworks (updating the DOM — including reactively — , loading lazily, etc.)</p> </li> <li> <p>Standards: standards APIs are among the “good frameworks” as they:</p> <ul> <li>allow portability: they are expected to be available everywhere. When not yet available, they can be polyfilled.</li> <li>allow interoperability: they can interact with other standards and be used by proprietary code.</li> <li>are long lived: as devised by multiple industry actors rather than only one, they are well designed and here to stay once released. So investing in them is less risky.</li> <li>are immediately available in the browser most of the time, which avoids downloading them. In some case you may have to download polyfills instead but, contrary to proprietary frameworks (which are doomed to be less and less trendy), their fate is to be more and more available (thus reducing download probability).</li> </ul> </li> <li> <p>What about the use of libraries? As for the “rewrite a framework” false assumption, it is often considered that vanilla JS apps are NOT supposed to use libraries. This is utterly false. Once again, “reinventing the wheel”, i.e. rewriting everything from scratch cannot be a sensible goal. The vanilla goal of removing constraints implied by frameworks and not libraries, must not be confused with a “write everything by yourself” dogma.</p> <p><em>(This raises an interesting question not answered by the author: What is the difference between a framework and a library?)</em></p> <p><em>(THis is a fascinating throwaway line at the end of this section:)</em> don’t be fooled by frameworks documentation or articles that would claim that they are not a framework (because they would be “unopinionated”, or not defining a “complete application“, etc.): as soon as they imply a contract, they are. <em>(Can libraries be opinionated?)</em> <em>(NOTE: author also has another post about the <a href="https://javarome.medium.com/framework-or-library-6711f998d978">differences between libraries and frameworks</a> in which he summarizes the two as "frameworks provide you application blueprints with built-in services but enforces predefined contracts to call your code and thus imply a strong dependency; libraries won’t help you design your application, but can be called only when you need them. You can devise a design that limits dependency to them.")</em></p> </li> <li> <p>Patterns. Just use of patterns is not enough, but it helps allow people coming to your app consume it more quickly (lighter cognitive burden).</p> </li> <li> <p>Concerns:</p> <ul> <li> <p>Updating: When interviewing developers about what would be their primary concerns when trying to build a vanilla application, most of them reply that it would be complicated to implement model change detection and subsequent updates in the relevant “views” of the app. This is a typical law of the instrument effect, which makes you think in a framework way, whereas not being a framework actually implies much more simple needs:</p> <ul> <li>The “views” are just DOM elements. You can abstract them of course (and you should) but in the end they are just that.</li> <li>Updating them is just a matter of viewElement.replaceChild(newContent). That’s it. No unnecessary update of a larger DOM scope, no unwanted redraw or scrolling. There are several ways to update the DOM, from inserting text to manipulating real DOM objects. Just pick the one that fits your need.</li> <li>“Detecting” when updating is required is usually not necessary in a vanilla app, since most often you just know what is to be updated following an event as you can just do it imperatively. You grab your DOM target and update it, period. In some cases of course you might want to do a more generic update by reversing the dependency and notifying observers (see below) that will update themselves.</li> </ul> </li> <li> <p>Templates: Another feature that developers fear to miss is the ability to write HTML snippets with dynamics parts, even listeners, etc.</p> <p>First of all the DOM API (document.createElement("button"), etc.) is not that hard, and actually more powerful than any template language since this allows you full access to the API. It can be tedious to build long HTML fragments but, hey, if they are that long, it’s probably that you need to split it in more fine grained components.</p> <p>It is true, however, that viewing those elements as a template improves readability. So how to have them? There are actually multiple ways:</p> <ul> <li> <p><a href="https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_templates_and_slots">HTML Templates</a> are now available in browsers (since 2017 at worse). They provide the ability to build a reusable, off-screen, HTML <template> snippet. This is this actually a part of Web Components and yes, they can support transclusion through <slot> . </slot> </template></p> </li> <li> <p><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">Template literals</a> are available in JavaScript since ES6 (2015). They allow you to easily embed values inside a string. That be enough to embed primitives (numbers, strings, including other HTML code, etc.) but not more sophisticated elements like DOM elements on which you registered listeners for instance.</p> </li> <li> <p>A <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals">tagged template literal</a> function can help embedding complex values like DOM Nodes into such a template that would result in a Node itself. ObservableHQ has devised a pretty handy one that allows you to write things like html<code>&lt;header&gt;${stringOrNode}&lt;/header&gt; or to do more complex templating like html</code></p> <ul> ${items.map(item =&gt; ` <li>${item.title}</li>} </ul> .<p></p> </li> </ul> </li> </ul> <p>What about conditionals or loops in a template? Aside the fact that this might have never been a good idea (UIs should be dumb and not contain logic), you can (and should) just do it JS, then insert the result in your template, using one of the techniques above.</p> <ul> <li> <p>Events: how to bind events to DOM Nodes inside them? There are also several alternatives:</p> <ul> <li>HTML event handlers (&lt;button onclick="myClickHandler(event)“&gt;) can be inserted in any HTML source, but they not very practicable, since they require the specified handlers to be available on the specified scope.</li> <li>Event handlers API (button.addEventListener("click", myClickHandler)) can be used on any node created through the DOM API or an HTML tagged template literal function.</li> </ul> <p>Now what about custom/business events? What if I need to react to some event triggered by a component of my app. There are also multiple options for that:</p> <ul> <li>Custom events: You can create you own events classes extending EventTarget and dispatch or listen to them, just like any “standard” event.</li> <li>EventEmitter is theoretically an option (exists in Node and as libs in the browser), but is rarely used.</li> <li>Observer pattern: You can build your own but RxJs is the de facto-standard for doing such reactive programming: build a Subject then notify all subscribers of a new value so they can react to it.</li> </ul> </li> <li> <p>Components: it is still a good idea design software items as reusable (i.e. context-independent) if they can occur multiple times in your system. Whatever the technology you use, well-grained abstractions remain useful, whether they are business or technical: it’s always a good idea to group data and rules pertaining to the same business concept into a reusable object or to build a widget that can be instantiated in multiple places in your app.</p> <p>Aside <em>[from]</em> standard widget components (which would typically be implemented as standard Web Components), any component should typically be able to <em>(standard component things: methods, state, events)</em>:</p> <ul> <li>split its logic and its view (through a MVC typically). Mixing the two usually makes the code less maintainable, and less flexible (for instance, should you want to display a record in both detailed or tabular form, your RecordComponent would just need to use either a DetailRecordView or a RowRecordView).</li> <li>read inputs to parameterize its behavior or its view.</li> <li>trigger events to notify parties that something occurred in the component (following a user interaction typically).</li> <li>synchronize: your component should be able to redraw if some event occurs. This can be achieved quite easily using reactive libraries such as RxJS.</li> </ul> <p>In any case, whatever the design strategy you choose, you component (or more specifically its associated “view”) must be able to provide some HTML rendering result. A string containing HTML code could be used, but an actual HTMLElement (or just Element) is usually a better choice (easier to read/update rather than parsing, allowing to bind event handlers on it) and a more performant one (no parsing required).</p> <p>Also, you might want to use external components from third-party libraries. For sure, proprietary frameworks, because of their popularity, benefit from a larger number of libraries and components developed by their community. Although most of them are actually not so different from what they would be if they’d had been implemented in pure Javascript (like this was the case at JQuery times) they do suffer from lack of interoperability, and you find yourself looking for either vanilla or Web components.</p> <p>Hopefully such libraries exist, such as the Vanilla JS Toolkit, even if less common. Regarding standards ones, WebComponents.org list 2000+ elements. There’s even vanilla web components, but the vanilla aspect is less relevant here (more about implementation lightweightness than interoperability).</p> </li> <li> <p>Routing: Managing routes in a SPA today requires using the web <a href="https://developer.mozilla.org/en-US/docs/Web/API/History_API">History API</a>. Whereas this is <a href="https://krasimirtsonev.com/blog/article/A-modern-JavaScript-router-in-100-lines-history-api-pushState-hash-url">less complex that you imagine</a>, you might want to delegate this to a simple router library such as <a href="https://github.com/krasimir/navigo">Navigo</a>. All you have to do, then, is to replace an DOM element by another (using replaceChildren() or replaceWith()) when a route is reached.</p> </li> <li> <p>Lazy loading: ES6 (2015) can load modules dynamically. This works in Node, but in browsers too: <code>{WelcomeModule} = await import("./welcome/ModuleImpl"); module = new WelcomeModule()</code> Bundlers like Webpack can insulate modules in dedicated files. Beware that you should only use constants in the import path, though: otherwise, the bundler cannot guess what you will load and will package the whole set of possible files in a single chunk. For instance <code>await import(</code>./welcome/${moduleName}<code>)</code> will bundle everything in the specified directory, given that your bundler doesn’t know what the <code>moduleName</code> variable will hold at runtime.</p> </li> <li> <p>Native apps: <em>(as opposed to using framework derivatives like React Native)</em></p> </li> <li> <p>Server-side rendering (SSR): <em>(I find it hilarious and sad that we talk about using client-side Javascript fameworks to do server-side rendering with a straight face)</em></p> </li> <li> <p>Internationalization: Internationalization have been handled by libraries for many years now (and finally integrated within frameworks). You can easily integrate one of those libs but this could also be a good candidate for an in-house implementation, which would allow more simple and efficient messages types than a general-purpose lib can.</p> <pre><code> ``` interface WelcomeMessages { title: string greetings(user: string, unreadCount: number): string } class WelcomeMessage_en implements WelcomeMessage { title = "Welcome !", greetings = (user, unreadCount) =&gt; `Welcome ${user}, you have ${unreadCount} unread messages.` } class WelcomeMessage_fr implements WelcomeMessage { title = "Bienvenue !", greetings = (user, unreadCount) =&gt; `Bienvenue ${user}, vous avez ${unreadCount} nouveaux messages.` } ``` </code></pre> <p>Note that this provides you:</p> <ul> <li> <p>type checking: every message has a static type (and several implementations/translations), so your IDE can check if you’re using a valid message property or not, and provide you auto-completion.</p> </li> <li> <p>translation exhaustivity check: you can’t compile until all interface keys are implemented in all languages.</p> </li> </ul> <p>All you have is to (load and) instantiate the message class that is relevant to you user’s locale. General purpose libs can’t provide such business-specific types.</p> <p><em>(I think he's oversimplifying the case, but at the same time, I think a lot of frameworks over-complexify the case so let's assume this is the simplest thing, and get more complicated from there as need be.)</em></p> </li> <li> <p>Tools: If you want to free yourself from dependency on a too constraining software stack, it is likely that you’d want to do the same with your tools: you don’t want to depend on them (their limitations, their performance, their bugs, their versions) to be able to move forward. You don’t want tot get stuck with a build problem that you cannot solve (or need hours or days to solve) because you do not own it (especially when using trendy but recent build tools which are not fully battle-tested yet). That said, you will hardy avoid those tools. Most often your production code will have to be bundled in a clever way, involving minification, obfuscation, code splitting, tree shaking, lazy loading, style inclusion, etc. and there is little doubt that existing packagers such as Webpack, Parcel, ESBuild or Vite will do it better that you could. All you can do about it is:</p> <ul> <li>Use less transformations as possible. For instance, using TypeScript might a good thing but implies an additional level of complexity that has to be handled by your tooling chain. Maybe your CSS also, especially with the latest modern versions, is not worth using preprocessors like Sass.</li> <li>Use less tools as possible. The more you add, the more one can fail / not support your needs.</li> <li>When using one, use the most popular tool so what you need is more likely to be supported by such a battle-tested software (and so you won’t be stuck in a position of “change need or change tool”). Moving to the latest hyped bundler too early might save you a few build seconds that might be compensated by the time devoted understanding the new beta documentation, handling bugs or lack of support.</li> </ul> <p><em>(I'm really not sure I agree with his take on this; he has some good points, but this feels like it encourages a very "build it all yourself" approach--which, to be fair, is a known concern that he's addressed repeatedly all over the place. Still, I think the better takeaway here is, "Use tools sparingly but deliberately." Know what the tool does, be able to live one layer deeper than what the tool gives you, and so on.)</em></p> </li> </ul> </li> <li> <p>Challenges: (from others)</p> <ul> <li>“you’re going to write your own framework”: no, you’re going to write an app instead of a framework.</li> <li>“you’re going to write more code”: maybe, maybe not so much (depending on your use of libraries) as this has to be compared with the boilerplate code required by frameworks. In any case, the overall loaded code size will be smaller,</li> <li>“you’re going to reinvent the wheel constantly”: of course not: not using a framework is just choosing to not comply with its predefined rules (configuration, lifecycle, refresh mechanism, etc.) but it’s not forgetting DRY principles and, as stated above, you can still (and should) use battle-tested third-party libraries.</li> <li>“you will write more code for every feature”: no, you’ll just use your own rules instead of the framework boilerplate.</li> <li>“there will be no documentation”: no documentation about the framework for sure (since there’s isn’t any) but, as for any software development, you are still expected to document your app. Notably, the use of patterns will help to auto-document your software design. Your app’s code documentation is the one you care about ; having an additional framework’s documentation is just the consequence of having a framework.</li> <li>“there will be no constraints or patterns to guide the developer”: no, nothing prevents you to enforce constraints if you need it (you just have to define contracts). The difference is that they will be the constraints of your choice, tailored for your app.</li> <li>“you will miss performance improvements” such as the once-hyped Virtual Dom (which is today challenged, including by subsequent frameworks like Svelte or Aurelia): no, as those “performance improvements” are actually more need by the general-purpose nature of frameworks, not by custom apps. On the opposite, general purpose framework will more likely miss a number of performance improvements that a custom code can implement.</li> <li>“You get this problem because you didn’t use a framework” Every difficulty (bug, delay, recruitment, etc) will be blamed on that unorthodox choice. Because most developers’ experience is that everything that has worked was using a framework, not using them will be assumed risky by default. This assumption will be considered as confirmed as soon as a problem will arise, whether it is related to not using a framework or not. They will forget all the similar issues they had when using framework.</li> <li>“We can’t find developers”: You’ll be told that it’s difficult to find developers who can develop vanilla code. That’s both true and false. True because a lot of developers (not speaking about managers) will find themselves more comfortable using known recipes such as frameworks. Candidates might feel a bit scared about building a webapp from scratch if they never did it once, or if they don’t know the basic web APIs well. False because, if you want to build quality apps, you should not look for this kind of developers. For sure it is currently easier to find any React developer, but what you need is not a React developer, but a good developer.</li> <li>“You won’t get the code quality of frameworks”. For sure frameworks or libraries are usually written by major industry players or experienced developers. However, as we saw above, the code of frameworks is mostly related to framework-specific activities (components lifecycle, general-purpose refresh and optimizations, tooling, etc.), not your app. Furthermore, you can still make bad design choices and write poor code using frameworks. The quality of your application always depends more on the quality of your team, than the lack of blueprints.</li> <li>“You won’t get same performance as frameworks”: No, we’ll get better performance. The marketed argument that framework include sophisticated technologies that can “improve performance” such as the “virtual DOM” (which allowed to bufferize DOM changes in order to limit their changes on it) is off-topic, as it relates to solving the performance drawbacks of a general-purpose solution. When the developer updates the DOM by him/herself, (s)he can get the best performance because (s)he will now if it’s worth bufferizing/caching it or if it’s pure overhead.</li> </ul> </li> </ul> </li> </ul> </li> </ul> https://tedneward.github.io/Research/presentation/w3m/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/w3m/index.html <p><a href="http://w3m.rocks/">Website</a> | <a href="http://w3m.sourceforge.net/">Source</a> | <a href="http://w3m.sourceforge.net/MANUAL">Manual</a></p> https://tedneward.github.io/Research/presentation/python/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/index/index.html <p><a href="https://realpython.com/tutorials/gui/">"Python GUI Tutorials"</a></p> <p><a href="https://blog.tooljet.ai/python-gui-framework/#">"Choosing Right Python GUI Framework: A Complete Guide"</a></p> <ul> <li>Tkinter</li> <li>PyQt5</li> <li><a href="/presentation/python/kivy">Kivy</a></li> <li>wxPython</li> <li>Libavg</li> <li>PySimpleGUI</li> <li>PyForms</li> <li>Wax</li> <li>PySide2</li> <li>PyGUI</li> </ul> <p>Related</p> <ul> <li><a href="https://github.com/python-eel/Eel">Eel</a>: A little Python library for making simple Electron-like HTML/JS GUI apps (Discontinued development in June 2025)</li> <li><a href="https://github.com/cztomczak/cefpython">CEFPython</a>: an open source project founded by Czarek Tomczak in 2012 to provide Python bindings for the Chromium Embedded Framework (CEF).</li> </ul> https://tedneward.github.io/Research/presentation/python/streamlit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/streamlit/index.html <p><a href="https://streamlit.io/">Website</a> | <a href="https://github.com/streamlit/streamlit">Source</a></p> <p>Install: <code>pip install streamlit</code></p> <p>Start an app: <code>streamlit hello</code></p> <p>Streamlit builds upon three simple principles:</p> <ul> <li>Embrace scripting: Build an app in a few lines of code with our magically simple API. Then see it automatically update as you iteratively save the source file.</li> <li>Weave in interaction: Adding a widget is the same as declaring a variable. No need to write a backend, define routes, handle HTTP requests, connect a frontend, write HTML, CSS, JavaScript, ...</li> <li>Deploy instantly: Effortlessly share, manage and deploy your apps, directly from Streamlit. All for free!</li> </ul> https://tedneward.github.io/Research/presentation/quasar/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/quasar/index.html <p><a href="https://quasar.dev/">Website</a></p> <p>Quasar is an open-source Vue.js based framework, which allows you as a web developer to quickly create responsive++ websites/apps in many flavours:</p> <ul> <li>SPAs (Single Page App)</li> <li>SSR (Server-side Rendered App) (+ optional PWA client takeover)</li> <li>PWAs (Progressive Web App)</li> <li>BEX (Browser Extension)</li> <li>Mobile Apps (Android, iOS, ...) through <a href="../cordova">Cordova</a> or <a href="../capacitorjs">Capacitor</a></li> <li>Multi-platform Desktop Apps (using <a href="../electron">Electron</a>)</li> </ul> <p>Quasar’s motto is: write code once and simultaneously deploy it as a website, a Mobile App and/or an Electron App. Yes, one codebase for all of them, helping you develop an app in record time by using a state-of-the-art CLI and backed by best-practice, blazing fast Quasar web components.</p> <p>When using Quasar, you won’t need additional heavy libraries like Hammer.js, Moment.js or Bootstrap. It’s got those needs covered internally, and all with a small footprint!</p> https://tedneward.github.io/Research/presentation/raygui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/raygui/index.html <p><a href="https://github.com/raysan5/raygui">Source</a></p> https://tedneward.github.io/Research/presentation/react/redux/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/react/redux/index.html <h3>Reading</h3> <ul> <li><a href="http://teropa.info/blog/2015/09/10/full-stack-redux-tutorial.html">Full-Stack Redux Tutorial</a></li> <li><a href="https://www.robinwieruch.de/the-soundcloud-client-in-react-redux/">SoundCloud Application in React + Redux</a></li> <li><a href="https://leanpub.com/redux-book">The Complete Redux Book</a> - Boris Dinkevich and Ilya Gelman <em>(Leanpub account or valid email requested)</em></li> </ul> https://tedneward.github.io/Research/presentation/remix/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/remix/index.html <p><a href="https://remix.run/">Website</a></p> https://tedneward.github.io/Research/presentation/royale/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/royale/index.html <p><a href="https://royale.apache.org/">Website</a> | <a href="https://royale.apache.org/tourdejewel">Tour De Jewel</a> | <a href="https://github.com/apache/royale-asjs/wiki/Apache-Royale-Source-Code-Repositories">Source</a></p> https://tedneward.github.io/Research/presentation/semanticui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/semanticui/index.html <p><a href="https://semantic-ui.com/">Website</a> | <a href="https://github.com/semantic-org/semantic-ui">Source</a></p> <p>(No updates since 2018)</p> https://tedneward.github.io/Research/presentation/shoelace/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/shoelace/index.html <p><a href="https://shoelace.style/">Website</a> | <a href="https://github.com/shoelace-style/shoelace">Source</a></p> <p><a href="https://css-tricks.com/shoelace-component-frameowrk-introduction/">Introducing Shoelace</a></p> https://tedneward.github.io/Research/presentation/skip/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/skip/index.html <p><a href="https://skip.dev/">Website</a> | <a href="https://github.com/skiptools/skip">Source</a></p> <h2>Getting Started</h2> <p>Skip is installed and updated using the popular Homebrew package management tool, which can be installed from brew.sh ↗. Once Homebrew has been setup, install Skip by running the Terminal command:</p> <pre><code>brew install skiptools/skip/skip </code></pre> <p>This will download and install the skip tool itself, along with the Android SDK and gradle build tool that is necessary for building and running the Android side of your apps.</p> <p>You can ensure that the basic development prerequisites are satisfied by running:</p> <pre><code>skip checkup </code></pre> <p>Create a new Skip app project with:</p> <pre><code>skip create </code></pre> <p>You will be guided through a series of questions about the app you want to create. The first two are especially important:</p> <ul> <li>App or Library? The App option creates a single Swift and SwiftUI app that uses Skip’s Xcode plugin to automatically build for Android. The Library option creates a Swift library that you can use in a unified Skip app, or use as shared business logic in separate iOS and Android apps. Neither option prevents you from using a combination of Swift and Kotlin, SwiftUI and Compose in your final application.</li> <li>Skip Fuse or Skip Lite? Information about the different modes can be found in the Native and Transpiled Modes documentation. In this example, we create a Skip Fuse app, which uses the official Swift SDK for Android ↗.</li> </ul> <p>If you’re new to Skip, we recommend starting your exploration with an App project. An example session for creating a hello-skip/HelloSkip project (which generates a project identical to the minimal Skip Fuse sample app ↗ or Skip Lite sample app ↗) might look like:</p> <pre><code>$ skip create Welcome to Skip! Select type of project to create: 1: App: mobile application for Android and iOS 2: Library: library project with one or more modules Enter selection (default: App) [1..2] 1 Select the mode of the project: 1: Skip Fuse: natively compiled project 2: Skip Lite: transpiled project Enter selection [1..2] 1 Enter the project-name for the App: hello-skip Enter the CamelCase name of the App module: HelloSkip Optionally enter additional module names: Enter the app bundle identifier: com.example.HelloSkip Create a free open-source project? (y/n) [n]: n Initialize git repository for the project? (y/n) [n]: n Initialize a Fastlane configuration for the project? (y/n) [y]: y Pre-build the project? (y/n) [y]: y Install the Swift Android SDK? (y/n) [y]: y Open the Xcode project after initialization? (y/n) [y]: y [✓] Install Swift Android SDK (48.4s) [✓] Resolve dependencies (12.12s) [✓] Build hello-skip (53.35s) [✓] Check project schemes (6.12s) [✓] Archive iOS ipa (17.18s) [✓] Assemble HelloSkip-debug.ipa 1 MB [✓] Verifying HelloSkip-debug.ipa: 1 MB [✓] Assembling Android apk (78.91s) [✓] Verify HelloSkip-debug.apk: 80 MB [✓] Opening Xcode project (0.10s) [✓] Skip create succeeded in 168.63s </code></pre> <p>You can alternatively create a Skip project using the non-interactive skip init command, as described in the command line reference.</p> <p>Once your app pre-builds and opens in Xcode, you are almost ready to go. One more step is needed, which is to create and launch an Android emulator (which is the Android equivalent of the iPhone Simulator and is used for local app development). The skip tool has a command to quickly install and configure an Android emulator for your project:</p> <pre><code>skip android emulator create [✓] Configure Android SDK Manager (1.55s) [✓] Install platform-tools (0.77s) [✓] Install emulator (0.75s) [✓] Install platforms;android-34 (0.76s) [✓] Install system-images;android-34;google_apis;arm64-v8a (0.75s) [✓] Create emulator: emulator-34-medium_phone (0.76s) [✓] Create Android emulator succeeded in 5.35s </code></pre> <p>Once this command completes successfully, you will have an Android emulator installed, which you can launch and observe the logging output with:</p> <pre><code>skip android emulator launch </code></pre> <p>Now you should be ready to launch your new app! In Xcode, ensure the “HelloSkip App” scheme is selected along with a simulator destination (e.g., “iPhone 17”), and then Run the scheme.</p> <p>You may be prompted to “Trust and Enable” the Skip plugin, and then the project will build and launch simultaneously on both the iPhone Simulator and Android Emulator.</p> https://tedneward.github.io/Research/presentation/stackedit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/stackedit/index.html <p><a href="https://stackedit.io/">Website</a> | <a href="https://benweet.github.io/stackedit.js/">Source Website</a> | <a href="https://github.com/benweet/stackedit.js">Source</a></p> <pre><code>&lt;blockquote&gt; &lt;textarea&gt;Hello **Markdown** writer!&lt;/textarea&gt; &lt;/blockquote&gt; &lt;script src="https://unpkg.com/stackedit-js@1.0.7/docs/lib/stackedit.min.js"&gt;&lt;/script&gt; &lt;script&gt; // Import the lib. const stackedit = new Stackedit(); function makeEditButton(el) { const div = document.createElement('div'); div.className = 'stackedit-button-wrapper'; div.innerHTML = '&lt;a href="javascript:void(0)"&gt;&lt;img src="icon.svg"&gt;Edit with StackEdit&lt;/a&gt;'; el.parentNode.insertBefore(div, el.nextSibling); return div.getElementsByTagName('a')[0]; } const textareaEl = document.querySelector('textarea'); console.log(textareaEl) makeEditButton(textareaEl) .addEventListener('click', function onClick() { const stackedit = new Stackedit(); stackedit.on('fileChange', function onFileChange(file) { textareaEl.value = file.content.text; }); stackedit.openFile({ name: 'Markdown with StackEdit', content: { text: textareaEl.value } }); }); const htmlEl = document.querySelector('.html'); let markdown = 'Hello **Markdown** writer!'; function doOpen(background) { const stackedit = new Stackedit(); stackedit.on('fileChange', function onFileChange(file) { markdown = file.content.text; htmlEl.innerHTML = file.content.html; }); stackedit.openFile({ name: 'HTML with StackEdit', content: { text: markdown } }, background); } doOpen(true); makeEditButton(htmlEl) .addEventListener('click', function onClick() { doOpen(false); }); &lt;/script&gt; </code></pre> https://tedneward.github.io/Research/presentation/svelte/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/svelte/index.html <p><a href="https://svelte.dev/tutorial">Tutorial</a> | <a href="https://svelte.dev/docs">Docs/API</a> | <a href="https://discordapp.com/channels/457912077277855764/473466028106579978">Discord</a></p> <h3>Getting Started</h3> <p>NodeJS-based tool chain. Svelte starts from either a .zip of the <code>sveltejs/template</code> project or by using <code>npx degit sveltejs/template &lt;project-directory&gt;</code>. From there it's a <code>cd</code> and <code>npm install</code>. Once updated, <code>npm run dev</code> puts a local web server on <a href="http://localhost:5000">http://localhost:5000</a> for dev examination.</p> <p>Project directory layout:</p> <p><code>/</code> - contains Rollup configuration, npm details, READMe, and so on. Probably nothing to do here.<br> <code>/public</code> - looks like this has the browser-accessible assets, including <code>index.html</code>.<br> <code>/src</code> - looks like this has the component files (<code>App.svelte</code>) as well as a global tie-together (<code>main.js</code>) that instantiates the top-level component. Betting that <code>main</code> gets referenced directly or indirectly from <code>index.html</code> at some point--nope, looks like it gets bundled into a build directory and that's what's referenced from the index.</p> <h3>Tutorial</h3> <p>Component-centric; HTML/CSS/JS all go into a <code>.svelte</code> file for use.</p> <h3>Reading</h3> <ul> <li><a href="https://svelte.dev/tutorial/basics">Svelte Tutorial</a> - Svelte.dev</li> <li><a href="https://flaviocopes.com/page/svelte-handbook/">The Svelte Handbook</a> - Flavio Copes (PDF, EPUB, Kindle) <em>(email address requested)</em></li> </ul> <h3>Sapper</h3> <p>"Sapper is a framework for building extremely high-performance web apps. You're looking at one right now! There are two basic concepts:<br> * Each page of your app is a component<br> * You create pages by adding files to the src/routes directory of your project. These will be server-rendered so that a user's first visit to your app is as fast as possible, then a client-side app takes over</p> <p>Building an app with all the modern best practices — code-splitting, offline support, server-rendered views with client-side hydration — is fiendishly complicated. Sapper does all the boring stuff for you so that you can get on with the creative part.</p> <p>You don't need to know Svelte to understand the rest of this guide, but it will help. In short, it's a UI framework that compiles your components to highly optimized vanilla JavaScript.</p> <p><a href="https://sapper.svelte.dev/docs/">Docs</a></p> https://tedneward.github.io/Research/presentation/tgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tgui/index.html <p><a href="https://tgui.eu/">Website</a> | <a href="https://github.com/texus/TGUI/">Source</a></p> https://tedneward.github.io/Research/presentation/tiptap/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tiptap/index.html <p><a href="https://tiptap.dev">Website</a> | <a href="https://github.com/ueberdosis/tiptap">Source</a></p> <p>Based on the highly-available ProseMirror library. Complemented by collaboration open-source backend Hocuspocus.</p> https://tedneward.github.io/Research/presentation/twitch/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/twitch/index.html <p><a href="https://www.twitch.tv/">Website</a></p> https://tedneward.github.io/Research/presentation/vaadin/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/vaadin/index.html <p><a href="https://vaadin.com/">Website</a> | <a href="https://vaadin.com/learn/tutorials">Tutorials</a> | <a href="https://github.com/vaadin/">Source</a></p> <h2><a href="https://vaadin.com/flow">Vaadin Flow</a> (<a href="https://github.com/vaadin/flow">Source</a>)</h2> <p><em>The Java framework binding Vaadin web components to Java. This is part of Vaadin 10+ Platform for building modern web sites that look great, perform well and make you and your users happy.</em> "Vaadin Flow is an open-source framework for building web apps in Java. You build your app from UI components without ever having to touch HTML or JavaScript."</p> <pre><code>@Route("hello-world") public class HelloWorld extends VerticalLayout { public HelloWorld() { TextField name = new TextField("Name"); Paragraph greeting = new Paragraph(""); Button button = new Button("Greet", event -&gt; { greeting.setText("Hello " + name.getValue()); }); add(name, button, greeting); } } </code></pre> <h2><a href="https://vaadin.com/fusion">Vaadin Fusion</a></h2> <p><em>A TypeScript and Java web framework for building modern web applications. You can create UIs in TypeScript and connect to any backend through endpoints written in Java.</em> "Vaadin Fusion is a framework that helps you rapidly deliver beautiful reactive client-side web apps using TypeScript with a Java backend."</p> <pre><code class="language-ts">@SpringBootApplication @PWA(name = "Fusion App", shortName = "Fusion", offlineResources = {"images/logo.png"}) public class Application extends SpringBootServletInitializer implements AppShellConfigurator { public static void main(String[] args) { SpringApplication.run(Application.class, args); } } // --- Server --- @Endpoint public class TodoEndpoint { private TodoRepository repo; // Repository autowired by Spring public TodoEndpoint(TodoRepository repo) { this.repo = repo; } // Publish a typed endpoint public List&gt;Todo&lt; getTodos() { return repo.findAll(); } } // --- Client --- // Call the backend in TypeScript const todos: Todo[] = await TodoEndpoint.getTodos(); </code></pre> <p><strong>Reactive views with declarative templates</strong>; Vaadin Fusion has a reactive programming model based on <a href="https://lit.dev/">Lit</a>.</p> <ul> <li>The declarative template is automatically updated when the application state changes.</li> <li>Lit uses a simple template syntax based on standard JavaScript.</li> <li>Fusion comes with built-in support for <a href="https://mobx.js.org/">MobX</a> for application state management.</li> </ul> <pre><code>@customElement('hello-world-view') export class HelloWorldView extends View { @state() name = ''; @state() allNames: string[] = []; render() { return html` &lt;h1&gt;Greetings!&lt;/h1&gt; &lt;ul&gt; ${this.allNames.map((name) =&gt; html`&lt;li&gt;Hello, ${name}&lt;/li&gt;`)} &lt;li&gt; &lt;vaadin-text-field placeholder="Your name" .value=${this.name} @value-changed=${this.nameChanged} &gt;&lt;/vaadin-text-field&gt; &lt;vaadin-button @click=${this.addGreeting}&gt;Add name&lt;/vaadin-button&gt; &lt;/li&gt; &lt;/ul&gt; `; } nameChanged(e: CustomEvent) { this.name = e.detail.value; } addGreeting() { this.allNames = [...this.allNames, this.name]; this.name = ''; } } </code></pre> <h2>Vaadin Framework (<a href="https://github.com/vaadin/framework">Source</a>)</h2> <p><em>Vaadin 8 and Vaadin 7, both of which use GWT as the base of client-side implementations.</em></p> https://tedneward.github.io/Research/presentation/visjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/visjs/index.html <p><a href="https://visjs.org/">Website</a> | <a href="https://github.com/visjs">Source</a></p> <p>The library consists of the components DataSet, Timeline, Network, Graph2d and Graph3d.</p> https://tedneward.github.io/Research/presentation/webcomponents/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/webcomponents/index/index.html <p><a href="https://www.webcomponents.org/">Website</a></p> <ul> <li><a href="https://www.webcomponents.org/introduction">Introduction</a></li> <li><a href="https://www.webcomponents.org/specs">Specifications</a>: Four main specifications <ul> <li><a href="https://w3c.github.io/webcomponents/spec/custom/">Custom Elements</a>: lays the foundation for designing and using new types of DOM elements.</li> <li><a href="https://w3c.github.io/webcomponents/spec/shadow/">Shadow DOM</a>: defines how to use encapsulated style and markup in web components.</li> <li><a href="https://html.spec.whatwg.org/multipage/webappapis.html#integration-with-the-javascript-module-system">ES Modules</a>: defines the inclusion and reuse of JS documents in a standards based, modular, performant way.</li> <li><a href="https://html.spec.whatwg.org/multipage/scripting.html#the-template-element/">HTML Template</a>: defines how to declare fragments of markup that go unused at page load, but can be instantiated later on at runtime.</li> </ul> </li> <li> <p><a href="https://www.webcomponents.org/libraries">Libraries</a></p> <ul> <li><a href="https://github.com/hybridsjs/hybrids">Hybrids</a> is a UI library for creating Web Components with simple and functional API.</li> <li><a href="https://github.com/Polymer/lit-element">LitElement</a> uses lit-html to render into the element's Shadow DOM and adds API to help manage element properties and attributes.</li> <li><a href="https://www.polymer-project.org/">Polymer</a> provides a set of features for creating custom elements.</li> <li><a href="http://slimjs.com/">Slim.js</a> is an opensource lightweight web component library that provides data-binding and extended capabilities for components, using es6 native class inheritance.</li> <li><a href="https://stenciljs.com/">Stencil</a> is an opensource compiler that generates standards-compliant web components.</li> </ul> </li> <li> <p><a href="https://github.com/mdn/web-components-examples">Examples</a>: The following examples are available:</p> <ul> <li><a href="https://github.com/mdn/web-components-examples/blob/main/composed-composed-path">composed-composed-path</a>. A very simple example that shows the behavior of the <code>Event</code> object <code>composed</code> and <code>composedPath</code> properties. <a href="https://mdn.github.io/web-components-examples/composed-composed-path/">See composed-composed-path live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/defined-pseudo-class">defined-pseudo-class</a>. A very simple example that shows how the <code>[:defined pseudo-class](https://developer.mozilla.org/en-US/docs/Web/CSS/:defined)</code> works. <a href="https://mdn.github.io/web-components-examples/defined-pseudo-class/">See defined-pseudo-class live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/editable-list">editable-list</a> –&nbsp;<code>&lt;editable-list&gt;</code>. A simple example showing how elements can be consolidated to create a list with addable/removable items. Items are added by using a <code>list-item</code> attribute or by entering text and clicking the plus sign. <a href="https://mdn.github.io/web-components-examples/editable-list/">See editable-list live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/edit-word">edit-word</a> — <code>&lt;edit-word&gt;</code>. Wrapping one or more words in this element means that you can then click/focus the element to reveal a text input that can then be used to edit the word(s). <a href="https://mdn.github.io/web-components-examples/edit-word/">See edit-word live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/element-details">element-details</a> — <code>&lt;element-details&gt;</code>. Displays a box containing an HTML element name and description. Provides an example of an autonomous custom element that gets its structure from a <code>&lt;template&gt;</code> element (that also has its own styling defined), and also contains <code>&lt;slot&gt;</code> elements populated at runtime. <a href="https://mdn.github.io/web-components-examples/element-details/">See element-details live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/expanding-list-web-component">expanding-list-web-component</a> — <code>&lt;ul is="expanding-list"&gt;</code>. Creates an unordered list with expandable/collapsible children. Provides an example of a customized built-in element (the class inherits from <code>HTMLUListElement</code> rather than <code>HTMLElement</code>). <a href="https://mdn.github.io/web-components-examples/expanding-list-web-component/">See expanding-list live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/life-cycle-callbacks">life-cycle-callbacks</a> — <code>&lt;custom-square l="" c=""&gt;</code>. A trivial example web component that creates a square colored box on the page. The demo also includes buttons to create, destroy, and change attributes on the element, to demonstrate how the <a href="https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements#Using_the_lifecycle_callbacks">web components life cycle callbacks</a> work <a href="https://mdn.github.io/web-components-examples/life-cycle-callbacks/">See life-cycle-callbacks live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/popup-info-box-web-component">popup-info-box-web-component</a> — <code>&lt;popup-info img="" text=""&gt;</code>. Creates an info icon that when focused displays a popup info box. Provides an example of an autonomous custom element that takes information from its attributes, and defines structure and basic style in an attached shadow DOM. <a href="https://mdn.github.io/web-components-examples/popup-info-box-web-component/">See popup-info-box live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/simple-template">simple-template</a> — A very simple trivial example that quickly demonstrates usage of the <code>&lt;template&gt;</code> and <code>&lt;slot&gt;</code> elements. <a href="https://mdn.github.io/web-components-examples/simple-template/">See simple-template live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/slotchange">slotchange example</a> — <code>&lt;summary-display&gt;</code>. An example that takes as its two slot values a list of possible choices, and a description for the selected choice. Multiple paragraphs are included inside the element containing all the possible descriptions; when a choice is clicked, its corresponding description paragraph is given an appropriate slot attribute so that it appears in the second slot. This example is written to demonstrate usage of the slotchange attribute, and features of the HTMLSlotElement interface <a href="https://mdn.github.io/web-components-examples/slotchange">See the slotchange example live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/slotted-pseudo-element">slotted-pseudo-element</a>. A very simple example that shows how the <code>::slotted</code> pseudo-element works. <a href="https://mdn.github.io/web-components-examples/slotted-pseudo-element/">See slotted-pseudo-element live</a>.</li> <li><a href="https://github.com/mdn/web-components-examples/blob/main/word-count-web-component">word-count-web-component</a> — <code>&lt;word-count&gt;</code>. When added to an element, counts all the words inside that element and displays them inside an attached shadow DOM. It also contains an interval that periodically updates the word count as it changes. Provides an example of a customized built-in element (the class inherits from <code>HTMLParagraphElement</code> rather than <code>HTMLElement</code>). <a href="https://mdn.github.io/web-components-examples/word-count-web-component/">See word-count live</a>.</li> </ul> </li> <li> <p><a href="https://gomakethings.com/my-html-web-component-boilerplate-for-2026/">My HTML Web Component boilerplate</a>: the <a href="https://gomakethings.com/snippets/boilerplates/web-component/">boilerplate</a></p> </li> <li> <p>Components ("Standalones")</p> <ul> <li> <p><a href="https://genericcomponents.netlify.app/">"Generic"</a>:</p> <ul> <li><a href="https://genericcomponents.netlify.app/demo/demo-app">showcase app</a></li> <li><a href="https://genericcomponents.netlify.app/generic-accordion/demo/">generic-accordion</a></li> <li><a href="https://genericcomponents.netlify.app/generic-alert/demo/">generic-alert</a></li> <li><a href="https://genericcomponents.netlify.app/generic-dialog/demo/">generic-dialog</a></li> <li><a href="https://genericcomponents.netlify.app/generic-disclosure/demo/">generic-disclosure</a></li> <li><a href="https://genericcomponents.netlify.app/generic-listbox/demo/">generic-listbox</a></li> <li><a href="https://genericcomponents.netlify.app/generic-radio/demo/">generic-radio</a></li> <li><a href="https://genericcomponents.netlify.app/generic-skiplink/demo/">generic-skiplink</a></li> <li><a href="https://genericcomponents.netlify.app/generic-spinner/demo/">generic-spinner</a></li> <li><a href="https://genericcomponents.netlify.app/generic-switch/demo/">generic-switch</a></li> <li><a href="https://genericcomponents.netlify.app/generic-tabs/demo/">generic-tabs</a></li> <li><a href="https://genericcomponents.netlify.app/generic-visually-hidden/demo/">generic-visually-hidden</a></li> </ul> </li> <li> <p>Extensions: Web Components that extend or add to an existing semantic element.</p> <ul> <li><a href="https://codepen.io/levimcg/pen/ZEYapRY"><code>&lt;accordion-container&gt;</code></a></li> <li><a href="https://github.com/andrico1234/autosize-textarea"><code>&lt;autosize-textarea&gt;</code></a></li> <li><a href="https://github.com/github/details-dialog-element"><code>&lt;details-dialog&gt;</code></a></li> <li><a href="https://github.com/github/details-menu-element"><code>&lt;details-menu&gt;</code></a></li> <li><a href="https://github.com/zachleat/details-utils"><code>&lt;details-utils&gt;</code></a></li> <li><a href="https://darn.es/heading-anchors-web-component"><code>&lt;heading-anchors&gt;</code></a></li> <li><a href="https://image-compare-component.netlify.app"><code>&lt;image-compare&gt;</code></a></li> <li><a href="https://darn.es/link-peek-web-component"><code>&lt;link-peek&gt;</code></a></li> <li><a href="https://github.com/rg-wood/link-preview"><code>&lt;link-preview&gt;</code></a></li> <li><a href="https://github.com/github/time-elements"><code>&lt;local-time&gt;</code></a></li> <li><a href="https://darn.es/play-button-web-component"><code>&lt;play-button&gt;</code></a></li> <li><a href="https://darn.es/random-source-web-component"><code>&lt;random-source&gt;</code></a></li> <li><a href="https://github.com/github/time-elements"><code>&lt;relative-time&gt;</code></a></li> <li><a href="https://darn.es/share-button-web-component"><code>&lt;share-button&gt;</code></a></li> <li><a href="https://stephen.band/slide-show/"><code>&lt;slide-show&gt;</code></a></li> <li><a href="https://github.com/github/tab-container-element"><code>&lt;tab-container&gt;</code></a></li> <li><a href="https://github.com/github/task-lists-element"><code>&lt;task-lists&gt;</code></a></li> <li><a href="https://github.com/github/time-elements"><code>&lt;time-ago&gt;</code></a></li> </ul> </li> <li> <p>Form: Web Components that are, or can be used with, form submissions.</p> <ul> <li><a href="https://github.com/lamplightdev/aeon"><code>&lt;aeon-datepicker&gt;</code></a></li> <li><a href="https://github.com/RamseyInHouse/feedback-component"><code>&lt;feedback-component&gt;</code></a></li> <li><a href="https://github.com/GoogleChromeLabs/file-drop"><code>&lt;file-drop&gt;</code></a></li> <li><a href="https://github.com/GoogleChromeLabs/input-knob"><code>&lt;input-knob&gt;</code></a></li> <li><a href="https://github.com/andreruffert/range-slider-element"><code>&lt;range-slider&gt;</code></a></li> <li><a href="https://github.com/sunnywalker/show-password-toggle"><code>&lt;show-password-toggle&gt;</code></a></li> <li><a href="https://codepen.io/nonsalant/pen/OPJLJoL"><code>&lt;toggle-categories&gt;</code></a></li> <li><a href="https://jquery.github.io/typesense-minibar/demo/"><code>&lt;typesense-minibar&gt;</code></a></li> </ul> </li> <li> <p>Utility: Useful little Web Components.</p> <ul> <li><a href="https://oddbird.github.io/browser-support/"><code>&lt;browser-support&gt;</code></a></li> <li><a href="https://github.com/github/clipboard-copy-element"><code>&lt;clipboard-copy&gt;</code></a></li> <li><a href="https://darn.es/duration-property-web-component"><code>&lt;duration-property&gt;</code></a></li> <li><a href="https://github.com/sakamies/filter-element"><code>&lt;filter-&gt;</code></a></li> <li><a href="https://darn.es/is-playing-web-component"><code>&lt;is-playing&gt;</code></a></li> <li><a href="https://darn.es/live-filter-web-component"><code>&lt;live-filter&gt;</code></a></li> <li><a href="https://github.com/knowler/log-form-element"><code>&lt;log-form&gt;</code></a></li> <li><a href="https://codepen.io/nonsalant/pen/yyBmeBp"><code>&lt;print-button&gt;</code></a></li> <li><a href="https://darn.es/sample-input-web-component"><code>&lt;sample-input&gt;</code></a></li> <li><a href="https://slide-deck.netlify.app/"><code>&lt;slide-deck&gt;</code></a></li> <li><a href="https://darn.es/storage-form-web-component"><code>&lt;storage-form&gt;</code></a></li> <li><a href="https://jamesbasoo.com/visual-viewport/"><code>&lt;visual-viewport&gt;</code></a></li> </ul> </li> <li> <p>Media: Web Components that are, or can be used with, media of any kind.</p> <ul> <li><a href="https://darn.es/bluesky-post-web-component"><code>&lt;bluesky-post&gt;</code></a></li> <li><a href="https://darn.es/bluesky-replies-web-component"><code>&lt;bluesky-replies&gt;</code></a></li> <li><a href="https://darn.es/code-pen-web-component"><code>&lt;code-pen&gt;</code></a></li> <li><a href="https://github.com/paulirish/lite-youtube-embed"><code>&lt;lite-youtube&gt;</code></a></li> <li><a href="https://darn.es/mastodon-post-web-component"><code>&lt;mastodon-post&gt;</code></a></li> <li><a href="https://github.com/GoogleWebComponents/model-viewer"><code>&lt;model-viewer&gt;</code></a></li> <li><a href="https://github.com/GoogleChromeLabs/pinch-zoom"><code>&lt;pinch-zoom&gt;</code></a></li> <li><a href="https://mirisuzanne.github.io/track-list/demo.html"><code>&lt;track-list&gt;</code></a></li> <li><a href="https://github.com/GoogleChromeLabs/two-up"><code>&lt;two-up&gt;</code></a></li> <li><a href="https://youtube-player.alanwsmith.com/"><code>&lt;youtube-player&gt;</code></a></li> </ul> </li> <li> <p>Design: Web Components that help with visual presentation.</p> <ul> <li><a href="https://github.com/GoogleChromeLabs/dark-mode-toggle"><code>&lt;dark-mode-toggle&gt;</code></a></li> <li><a href="https://github.com/compmeist/easy-row"><code>&lt;easy-row&gt;</code></a></li> <li><a href="https://github.com/lekoala/flex-grid"><code>&lt;flex-grid&gt;</code></a></li> <li><a href="https://github.com/sakamies/conditional-elements"><code>&lt;if-/or-/else-&gt;</code></a></li> <li><a href="https://github.com/lekoala/last-icon"><code>&lt;l-i&gt;</code></a></li> <li><a href="https://github.com/ingmarh/scroll-shadow-element"><code>&lt;scroll-shadow&gt;</code></a></li> </ul> </li> <li> <p>Custom: Frequently-solved problems in web component form.</p> <ul> <li><a href="https://github.com/mattdsteele/bt-device"><code>&lt;bt-device&gt;</code></a></li> <li><a href="https://www.npmjs.com/package//html-include-element"><code>&lt;html-include&gt;</code></a></li> <li><a href="https://github.com/justinfagnani/katex-elements"><code>&lt;katex-display&gt;</code></a></li> <li><a href="https://github.com/andrico1234/no-spoilers"><code>&lt;no-spoilers&gt;</code></a></li> <li><a href="https://github.com/Noleli/place-holder"><code>&lt;place-holder&gt;</code></a></li> <li><a href="https://github.com/pwa-builder/pwa-install"><code>&lt;pwa-install&gt;</code></a></li> </ul> </li> <li> <p>Novelty</p> <ul> <li><a href="https://wbrowar.com/web-components/admin-bar-component"><code>&lt;admin-bar&gt;</code></a></li> <li><a href="https://justinfagnani.github.io/chessboard-element/"><code>&lt;chess-board&gt;</code></a></li> <li><a href="https://darn.es/contrast-details-web-component"><code>&lt;contrast-details&gt;</code></a></li> <li><a href="https://noahliebman.net/projects/plucky-underline/"><code>&lt;plucky-underline&gt;</code></a></li> <li><a href="https://github.com/erikkroes/spacer-gif"><code>&lt;spacer-gif&gt;</code></a></li> <li><a href="https://github.com/grislyeye/vellum-dice"><code>&lt;vellum-dice&gt;</code></a></li> </ul> </li> <li> <p>Accessible: (from <a href="https://scottaohara.github.io/accessible_components/">https://scottaohara.github.io/accessible_components/</a>) I've <em>[sic]</em> built a good handful of accessible markup patterns and widgets at this point. Most of which were bespoke and incorporated into websites/products. But, there are a baseline patterns I was able to use over and over again, and the following represent the most common widgets that I was able to distill into basic examples (i.e., vanicall html/css/js). Each was based on testing with users, UX and design needs from those past projects, and from following normative specifications &amp; informative notes.</p> <ul> <li><a href="https://github.com/scottaohara/accessible_accordions">Accordions (v3)</a></li> <li><a href="https://github.com/scottaohara/aria-links">ARIA Links (v2)</a></li> <li><a href="https://github.com/scottaohara/a11y_button">ARIA Buttons (v2)</a></li> <li><a href="https://github.com/scottaohara/a11y_breadcrumbs">Breadcrumb Navigation (v1.1)</a></li> <li><a href="https://github.com/scottaohara/clear-text-field-button">Clear text field buttons (v1)</a></li> <li><a href="https://github.com/scottaohara/aria_disclosure_widget">Disclosure Widgets (v2.0.1)</a></li> <li><a href="https://github.com/scottaohara/landmarks_demo">Landmarks Demonstration</a></li> <li><a href="https://github.com/scottaohara/accessible_modal_window">Modal Dialogs (v3 - archived)</a></li> <li><a href="https://github.com/scottaohara/a11y_switch_web_component">Switch Toggle Web Component (v0)</a></li> <li><a href="https://github.com/scottaohara/aria-switch-button">Switch Toggles (v2.0.1)</a></li> <li><a href="https://github.com/scottaohara/a11y_tab_widget">Tab Widgets (v2)</a></li> <li><a href="https://github.com/scottaohara/a11y_tooltips">Tooltips (v1)</a></li> <li><a href="https://github.com/scottaohara/a11y_styled_form_controls">Various Styled Form Controls</a></li> </ul> </li> </ul> </li> </ul> <h2>Reading</h2> <h3>Articles/Blog Posts</h3> <ul> <li><a href="https://dev.to/ryansolid/web-components-are-not-the-future-48bh">"WebComponents are not the future"</a></li> <li><a href="https://dev.to/besworks/the-hidden-power-of-web-components-1jei">"The Hidden Power of WebComponents"</a></li> <li><a href="https://dev.to/besworks/the-problem-with-web-components-4f8">"The Problem with WebComponents"</a></li> <li><a href="https://kinsta.com/blog/web-components/">"A Complete Introduction to WebComponents"</a></li> <li><a href="https://thenewstack.io/web-components-are-the-comeback-nobody-saw-coming/">"Web Components are the Comback Nobody Saw Coming"</a></li> <li><a href="https://frontendmasters.com/blog/the-missing-link-for-web-components/">The Missing Link for Web Components</a>:</li> </ul> https://tedneward.github.io/Research/presentation/python/justpy/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/justpy/index.html <p><a href="https://justpy.io/">Website</a> | <a href="https://github.com/justpy-org/justpy">Source</a></p> <p>With a few lines of only Python code, you can create interactive websites without any JavaScript programming. JustPy can also be used to create graphic user interfaces for Python programs.</p> <p><strong>NOTE:</strong> "justpy has been a great project and our goal to reach 1000 stars soon was reached now.<br> Given that nicegui is somewhat of a fork of justpy and is much more successful with &gt;4000 stars there is not much use to try to build on justpy and move it further. Therefore I intend to close most issues now that ask for enhancements pointing to nicegui as an alternative that might implement these features and concentrate on helping people migrate to nicegui or keep their existing justpy code running for a while. justpy 0.13 seems to be sufficiently stable to not cause too many worries."</p> https://tedneward.github.io/Research/presentation/q5xjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/q5xjs/index.html <p><a href="https://q5xjs.netlify.app/">Website</a> | <a href="https://github.com/LingDong-/q5xjs">Source</a></p> https://tedneward.github.io/Research/presentation/qwik/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/qwik/index.html <p><a href="https://github.com/builderio/qwik">Source</a></p> <p>From the blog:<br> - <a href="https://dev.to/mhevery/a-first-look-at-qwik-the-html-first-framework-af">First look</a><br> - <a href="https://dev.to/mhevery/death-by-closure-and-how-qwik-solves-it-44jj">Death by closure</a><br> - <a href="https://dev.to/mhevery/html-first-javascript-last-the-secret-to-web-speed-4ic9">HTML first, JS last</a><br> - <a href="https://dev.to/mhevery/qwik-the-answer-to-optimal-fine-grained-lazy-loading-2hdp">Answer to optimal fine-grained loading</a></p> https://tedneward.github.io/Research/presentation/react/elemental-ui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/react/elemental-ui/index.html <p><a href="http://elemental-ui.com/">Website</a></p> https://tedneward.github.io/Research/presentation/react/tanstack/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/react/tanstack/index.html <p><a href="https://tanstack.com">Website</a> | <a href="https://github.com/tanstack">Source</a></p> <h2>Features/Contents</h2> <p>Data and State Management</p> <ul> <li>TanStack Start: Full-stack Framework powered by TanStack Router for React and Solid</li> <li>TanStack Router: A powerful React router for client-side and full-stack react applications. Fully type-safe APIs, first-class search-params for managing state in the URL and seamless integration with the existing React ecosystem.</li> <li>TanStack Query: Powerful asynchronous state management, server-state utilities and data fetching. Fetch, cache, update, and wrangle all forms of async data in your TS/JS, React, Vue, Solid, Svelte &amp; Angular applications all without touching any "global state"</li> <li>TanStack DB: TanStack DB extends TanStack Query with collections, live queries and optimistic mutations that keep your UI reactive, consistent and blazing fast</li> <li>TanStack Store: The immutable-reactive data store that powers the core of TanStack libraries and their framework adapters.</li> <li>TanStack Pacer: Optimize your application's performance with TanStack Pacer's core primitives: Debouncing, Throttling, Rate Limiting, Queuing, and Batching.</li> </ul> <p>Headless UI</p> <ul> <li>TanStack Table: Supercharge your tables or build a datagrid from scratch for TS/JS, React, Vue, Solid, Svelte, Qwik, Angular, and Lit while retaining 100% control over markup and styles.</li> <li>TanStack Form: Headless, performant, and type-safe form state management for TS/JS, React, Vue, Angular, Solid, Lit and Svelte.</li> <li>TanStack Virtual: Virtualize only the visible content for massive scrollable DOM nodes at 60FPS in TS/JS, React, Vue, Solid, Svelte, Lit &amp; Angular while retaining 100% control over markup and styles.</li> <li>TanStack Ranger: Headless, lightweight, and extensible primitives for building range and multi-range sliders.</li> </ul> <p>Other<br> * TanStack Devtools: A unified devtools panel that houses all TanStack devtools and allows you to create and integrate your own custom devtools.<br> * TanStack Config: Configuration and tools for publishing and maintaining high-quality JavaScript packages. Opinionated tooling to lint, build, test, version, and publish JS/TS packages — minimal config, consistent results.</p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://www.freecodecamp.org/news/how-to-build-a-crud-app-with-tanstack-start-and-tanstackdb-with-rxdb-integration/">How to Build a CRUD App with TanStack Start and TanStackDB (with RxDB Integration)</a></li> </ul> https://tedneward.github.io/Research/presentation/responsivegridsystem/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/responsivegridsystem/index.html <p><a href="http://www.responsivegridsystem.com/">Website</a> | <a href="https://github.com/grahammiller/ResponsiveGridSystem">Source</a></p> https://tedneward.github.io/Research/presentation/ruffle/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ruffle/index.html <p><a href="https://ruffle.rs/">Website</a> | <a href="https://github.com/ruffle-rs/ruffle">Github</a></p> https://tedneward.github.io/Research/presentation/sencha/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/sencha/index.html <p><a href="https://www.sencha.com/products/extjs/">Website</a> | Commercial--no source</p> <h2>Sencha Tools</h2> <p>Sencha Architect: Build Ext JS applications using drag-and-drop features and spend less time on manual coding.</p> <p>Sencha Themer: Design customized application themes using graphical tools — without writing code.</p> <p>IDE and Code Editor Plugins: integrate Sencha frameworks into your enterprise workflow, enabling code completion, inspection, generation, navigation, refactoring and more.</p> <p>ExtGen: a node based cross-platform command line tool that provides multiple modes for application generation.</p> <p>ExtBuild: a new node based tool to build Ext JS applications that currently uses Sencha Cmd and Google Closure compiler for building and transpiling Ext JS applications.</p> <p>Sencha Fiddle: a free tool that allows you to try Ext JS code in your browser without downloading or installing anything. You can easily share your Ext JS code by saving and sharing fiddle URLs.</p> <p>Sencha Cmd: A full set of lifecycle management features including scaffolding, code minification, transpilation from ES6, dynamic package loading, build generation for PWAs and more.</p> <p>Inspector: Debugging tool that provides direct access to components, classes, objects, and more for apps built using Sencha frameworks.</p> <p>Sencha Stencils: a complete UI asset kit for Adobe Illustrator, Sketch, Balsamiq and is also available as SVG/PNG for use with other programs.</p> https://tedneward.github.io/Research/presentation/simplemde/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/simplemde/index.html <p><a href="https://simplemde.com/">Website</a> | <a href="https://github.com/sparksuite/simplemde-markdown-editor">Source</a></p> https://tedneward.github.io/Research/presentation/smenu/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/smenu/index.html <p><a href="https://github.com/p-gen/smenu">Source</a></p> <p>smenu is a selection filter just like sed is an editing filter.</p> <p>This tool reads words from standard input or from a file, and presents them to the terminal screen in different layouts in a scrolling window. A cursor, easily moved using the keyboard and/or the mouse, makes it possible to select one or more words.</p> <p>Note that the screen is not cleared at the start and end of smenu execution. The selection window is displayed at the cursor position, and the previous contents of the terminal are neither modified nor lost.</p> <p>I've tried to make it as easy to use as possible. It should work on all terminals managed in the terminfo database.</p> <p>UTF-8 encoding is supported. This support includes double-width characters and extended grapheme clusters. The latter is still experimental, however, and works much better if appropriate terminals such as WezTerm or iTerm are used.</p> <p>The encoding of UTF-8 glyphs must also be in canonical form, as no effort will be made to put them in this form.</p> <p>Please refer to the included man page to find out more about this little program.</p> https://tedneward.github.io/Research/presentation/static-site-generators/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/static-site-generators/index.html <p><a href="https://staticsitegenerators.net/">Website</a></p> <h3>JVM</h3> <ul> <li><a href="https://github.com/jbake-org/jbake">jbake</a></li> <li><a href="https://github.com/agrison/jssg">jssg</a></li> </ul> <h3>Blogs</h3> <ul> <li><a href="https://www.11ty.io/">11ty</a> - Eleventy is a simpler static site generator. - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="http://ablog.readthedocs.org/">ABlog</a> - A Sphinx extension that converts any documentation or personal website project into a full-fledged blog. - <code>#Python</code></li> <li><a href="https://adduce.vale.rocks">Adduce</a> - A versatile static site generator that handles both blogs and standard pages. - <code>#Rust</code></li> <li><a href="http://github.com/atas/ssg">Ata's SSG</a> - A PHP-based blog-aware vanilla-HTML static site generator for GitHub Pages with markdown support.</li> <li><a href="https://github.com/capjamesg/aurora">Aurora</a> - An extensible, Python-based static site generator with support for jinja2 and markdown templates. - <code>#Python</code></li> <li><a href="https://github.com/cfenollosa/bashblog">BashBlog</a> - A single Bash script to create blogs. - <code>#Bash</code></li> <li><a href="https://github.com/venthur/blag">Blag</a> - blag is a blog-aware, static site generator -- it uses Markdown and is written in Python. - <code>#Python</code></li> <li><a href="https://github.com/tesar-tech/BlazorStatic">BlazorStatic</a> - Use ASP.NET Blazor to generate static pages. - <code>#.NET</code> <code>#C#</code></li> <li><a href="https://code.rosaelefanten.org/blogcpp/">BlogC++</a> - A static blog generator, written in C++17. - <code>#C++</code></li> <li><a href="https://bloggrify.com/">Bloggrify</a> - A static blog generator using Markdown, built on top of Nuxt-Content. - <code>#Vue.js</code> <code>#Markdown</code></li> <li><a href="https://cobalt-org.github.io">Cobalt</a> - A Rust static site generator. - <code>#Rust</code></li> <li><a href="https://github.com/coleslaw-org/coleslaw">Coleslaw</a> - Flexible blog and site generator with a lot of plugins. - <code>#CommonLisp</code></li> <li><a href="https://github.com/spekulatius/laravel-commonmark-blog">CommonMark Blog for Laravel</a> - Static generator to use with Laravel projects. Uses CommonMark and FrontMatter to generate and publish files in the <code>public</code>-folder. Supports articles as well as listing pages <code>#PHP</code></li> <li><a href="https://franklinjl.org/">Franklin</a> - A static site generator written in Julia that is primarily intended for technical blogging. - <code>#Julia</code></li> <li><a href="https://github.com/imfunniee/gitfolio">Gitfolio</a> - A portfolio website where you could showcase your work + a blog that will help you spread your ideas into real world. - <code>#Node.js</code></li> <li><a href="https://github.com/dannyvankooten/gozer">Gozer</a> - Simple &amp; fast static site generator in a single static binary. - <code>#Go</code> <code>#Golang</code></li> <li><a href="https://github.com/hexojs/hexo">Hexo</a> - A fast, simple &amp; powerful blog framework. - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://github.com/hyde/hyde">Hyde</a> - <code>#Python</code></li> <li><a href="https://github.com/hydephp/hyde">HydePHP</a> - Elegant and powerful static site generator with rich support for blogs, documentation sites, and more using Markdown and/or Blade - <code>#PHP</code> <code>#Laravel</code></li> <li><a href="https://www.jamify.org/">Jamify Gatsby</a> - Publish flaring fast blogs with Gatsby and Ghost. - <code>#JavaScript</code> <code>#React</code> <code>#Gastby</code></li> <li><a href="https://github.com/styxlab/next-cms-ghost">Jamify NextJS</a> - Publish flaring fast blogs with Next.js and Ghost. - <code>#JavaScript</code> <code>#React</code> <code>#Next.js</code></li> <li><a href="https://github.com/jekyll/jekyll">Jekyll</a> - Jekyll is a simple, blog-aware, static site generator perfect for personal, project, or organization sites. - <code>#Ruby</code></li> <li><a href="https://github.com/kabukky/journey">Journey</a> - <code>#Go</code></li> <li><a href="https://github.com/sunainapai/makesite">Makesite</a> - Simple, lightweight, and magic-free static site/blog generator (&lt; 130 lines) for Python coders <code>#Python</code></li> <li><a href="https://mkws.sh">mkws</a> - simple static site generator <code>#sh</code></li> <li><a href="https://getnikola.com/">Nikola</a> - <code>#Python</code></li> <li><a href="https://github.com/imathis/octopress">Octopress</a> - Similar to Jekyll but where everything you need is already setup. - <code>#Ruby</code> <code>#Jekyll</code></li> <li><a href="https://orchid.netlify.app">Orchid Posts</a> - Add Jekyll-like blogging functionality to your Orchid site. - <code>#Orchid</code> <code>#Java</code> <code>#Kotlin</code></li> <li><a href="https://github.com/getpelican/pelican">Pelican</a> - Uses Markdown or ReST for content and Jinja2 for themes. - <code>#Python</code></li> <li><a href="https://prpl.dev">PRPL</a> - A modular static site generator built for longevity - <code>#TypeScript</code> <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="http://www.qgoda.net/">Qgoda</a> - Arbitrary taxonomies and cross-links, multi-lingua. - <code>#Perl</code></li> <li><a href="https://quartz.jzhao.xyz/">Quartz</a> - Batteries-included extensible Markdown renderer for your digital garden, complete with GH Pages integration and RSS. - <code>#TypeScript</code> <code>#Node.js</code></li> <li><a href="https://github.com/williamd1k0/sake">Sake</a> - A simple static site generator built with make. - <code>#make</code> <code>#Jinja2</code></li> <li><a href="https://dalgona.github.io/Serum/">Serum</a> - Serum is a static website generator written in Elixir Programming Language. - <code>#Elixir</code></li> <li><a href="https://sphido.cz">Sphido</a> - A rocket 🚀 fast, lightweight, static site generator <code>#Node.js</code></li> <li><a href="https://github.com/spress/Spress/">Spress</a> - <code>#PHP</code></li> <li><a href="https://www.stapy.net">Stapy</a> - Works without any additional package and without command line on Windows. - <code>#Python</code></li> <li><a href="https://github.com/nakkaya/static">Static</a> - Supports org-mode and markdown. - <code>#Clojure</code></li> <li><a href="http://preaction.me/statocles/">Statocles</a> - Markdown, code highlighting etc. ' <code>#Perl</code></li> <li><a href="https://github.com/vladris/tinkerer">Tinkerer</a> - <code>#Python</code></li> <li><a href="https://github.com/leonstafford">WP2Static</a> - WordPress static site generator for security, performance and cost benefits</li> <li><a href="https://xlog.emadelsaid.com/">Xlog</a> - Personal knowledge management application with focus on enriching markdown files and surfacing implicit links between pages.</li> </ul> <h3>CMS</h3> <ul> <li><a href="https://ari.se.net">Arise</a> - A 90s-web inspired Bash static site generator designed around slow-moving stable dependencies, modular page hierarchy, and cloud-native CI deployment. - <code>#Bash</code></li> <li><a href="https://datenstrom.se/yellow/">Datenstrom Yellow</a> - For people who make small websites. - <code>#PHP</code></li> <li><a href="https://www.getlektor.com/">Lektor</a> - An easy to use static CMS and blog engine. - <code>#Python</code></li> <li><a href="https://primo.so">Primo</a> - An all-in-one static site builder. - <code>#Svelte</code> <code>#Electron</code></li> <li><a href="http://getpublii.com/">Publii</a> - Easy-to-use desktop app to generate static websites. - <code>#JavaScript</code> <code>#Electron</code></li> <li><a href="https://tipe.io/">Tipe</a> - An easy to use API-first CMS engine to generate static sites. - <code>#JavaScript</code></li> <li><a href="https://wordmogul.com">Wordmogul</a> - Minimalistic blogging platform with ZIP export of .md files (GUI for Hugo/Jekyll). - <code>#Go</code> <code>#Golang</code></li> </ul> <h3>Courseware</h3> <ul> <li><a href="https://sjzeil.github.io/CoWeM/">CoWeM</a> <a href="https://github.com/sjzeil/CoWeM">Source</a>: Static site generator for course websites, based on markdown</li> </ul> <h3>Documentation</h3> <ul> <li><a href="https://heinventions.github.io/docnado-site">Docnado</a> - A rapid documentation tool that will blow you away; batteries and style included. - <code>#Python</code> <code>#Jinja2</code></li> <li><a href="https://github.com/docpad/docpad">DocPad</a> - <code>#JavaScript</code> <code>#CoffeeScript</code> <code>#Node.js</code></li> <li><a href="https://docsify.js.org/">Docsify</a> - A magical documentation site generator. <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://github.com/Doctave/doctave">Doctave</a> - A batteries-included developer documentation site generator. - <code>#Rust</code></li> <li><a href="https://docusaurus.io/">Docusaurus</a> - Easy to maintain open source documentation websites. - <code>#JavaScript</code> <code>#React</code></li> <li><a href="https://www.mkdocs.org/">MkDocs</a> - Write your docs in Markdown and configure the generator with a single YAML configuration file. - <code>#Python</code></li> <li><a href="https://orchid.run/plugins/orchidjavadoc">Orchid Javadoc</a> - Create beautiful Javadocs for your project within your Orchid site. - <code>#Orchid</code> <code>#Java</code> <code>#Kotlin</code></li> <li><a href="https://github.com/lord/slate">Slate</a> - <code>#Ruby</code></li> <li><a href="http://sphinx-doc.org/">Sphinx</a> - <code>#Python</code></li> </ul> <h3>Science</h3> <ul> <li><a href="https://curvenote.com/docs/web">Curvenote</a> - Build scientific and technical websites for books, preprints, papers and reports using MyST Markdown and Jupyter Notebooks - <code>#Node.js</code></li> </ul> <h3>Marketing</h3> <ul> <li><a href="https://github.com/Tsukumogami-Software/milou">Milou</a> - Create beautiful and informative press kits to showcase your company and products to the medias - <code>#Node.js</code></li> </ul> <h3>Code Playground</h3> <ul> <li><a href="https://github.com/shi-yan/codestage">CodeStage</a> - A static site generator to create javascript playgrounds / interactive demos.</li> </ul> <h3>Frameworks</h3> <ul> <li><a href="http://assemble.io/">Assemble</a> - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://docs.astro.build/">Astro</a> - Framework agnostic static site generator with component-level support for partial hydration on page load, page idle, component visible or none. - <code>#JavaScript</code> <code>#Node.Js</code> <code>#React.js</code> <code>#Vue.js</code> <code>#Svelte.js</code></li> <li><a href="https://github.com/blurry-dev/blurry">Blurry</a> - Blurry is a schema-first, plugin-enabled static site generator. Markdown front matter directly to Schema.org types, so your content is SEO-friendly and rich results-ready out of the box. - <code>#Python</code></li> <li><a href="https://www.bridgetownrb.com/">Bridgetown</a> - A next-generation, progressive site generator &amp; fullstack framework, powered by Ruby. - <code>#Ruby</code></li> <li><a href="https://github.com/koenbok/Cactus">Cactus</a> - <code>#Python</code></li> <li><a href="https://capri.build/">Capri</a> - Static site generator based on Vite with partial hydration for a variety of frameworks. - <code>#TypeScript</code> <code>#React.js</code> <code>#Preact</code> <code>#Solid</code> <code>#Vue</code> <code>#Svelte</code></li> <li><a href="https://github.com/xy2z/capro">Capro</a> - Flexible PHP8 static site generator, using Blade template engine. - <code>#PHP</code> <code>#Blade</code></li> <li><a href="https://github.com/badosu/coil">coil</a> - <code>#elixir</code></li> <li><a href="http://complexity.readthedocs.org/en/latest/">Complexity</a> - For those who like to work in HTML. - <code>#Python</code></li> <li><a href="https://amokfa.github.io/posts/constexprjs.html">ConstexprJS</a> - Evaluate and struo JS in your website ahead of time. - <code>#JavaScript</code></li> <li><a href="https://cuttlebelle.com/">Cuttlebelle</a> - The react static site generator that separates editing and code concerns. - <code>#JavaScript</code> <code>#Node.js</code> <code>#React.js</code></li> <li><a href="https://github.com/tfpk/dewar">Dewar</a> - A flask-like static site generator. - <code>#Python</code></li> <li><a href="https://elderguide.com/tech/elderjs/">Elder.js</a> - An opinionated, SEO focused, static site generator for Svelte.js. - <code>#JavaScript</code> <code>#Node.js</code> <code>#Svelte.js</code></li> <li><a href="https://github.com/SimonSapin/Frozen-Flask">Flask-Frozen</a> - Freezes a Flask application into a set of static files. - <code>#Python</code> <code>#Flask</code></li> <li><a href="https://github.com/kobo/gaiden">Gaiden</a> - A tool that makes it easy to create documentation with Markdown. - <code>#Groovy</code></li> <li><a href="https://github.com/gatsbyjs/gatsby">Gatsby</a> - <code>#JavaScript</code> <code>#Node.js</code> <code>#React.js</code></li> <li><a href="https://github.com/FooSoft/goldsmith">Goldsmith</a> - Fast and simple pipeline-based static site generator, extensible via plugins. <code>#Go</code></li> <li><a href="https://github.com/sysgears/grain">Grain</a> - <code>#Groovy</code></li> <li><a href="https://gridsome.org">Gridsome</a> - Vue.js-powered, modern site generator for building the fastest possible websites for any Headless CMS, APIs or Markdown-files. - <code>#Vue.js</code> <code>#JavaScript</code></li> <li><a href="https://github.com/jaspervdj/hakyll">Hakyll</a> - Hakyll is a Haskell library for generating static sites, mostly aimed at small-to-medium sites and personal blogs. It is written in a very configurable way and uses an xmonad-like DSL for configuration. - <code>#Haskell</code></li> <li><a href="http://harpjs.com/">Harp</a> - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://hexo.io/">Hexo</a> - <code>#JavaScript</code> <code>#Node.js</code> <a href="https://github.com/hexojs/hexo">Source</a></li> <li><a href="https://github.com/spf13/hugo">Hugo</a> - It is optimized for speed, easy use and configurability. - <code>#Go</code></li> <li><a href="https://iles.pages.dev">îles</a> - Generates static sites with islands of interactivity - <code>#Vue</code>, <code>#Preact</code>, <code>#Svelte</code>, <code>#Solid</code></li> <li><a href="https://github.com/jbake-org/jbake">JBake</a> <code>#Java</code></li> <li><a href="https://github.com/gar1t/lambdapad">LambdaPad</a> - <code>#Erlang</code></li> <li><a href="https://github.com/datagrok/makebakery">m4-bakery</a> - <code>#make</code> <code>#m4</code></li> <li><a href="https://github.com/segmentio/metalsmith">Metalsmith</a> - An extremely simple, pluggable static site generator. - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://github.com/middleman/middleman">Middleman</a> - Middleman is a static site generator using all the shortcuts and tools in modern web development. - <code>#Ruby</code></li> <li><a href="https://github.com/nanoc/nanoc">nanoc</a> - Flexible support for small personal to large corporate web sites. - <code>#Ruby</code></li> <li><a href="https://nuxtjs.org/">Nuxt.js</a> - Versatile vue-based framework for server-side-rendered, static and traditional single page applications.</li> <li><a href="https://github.com/BennyHallett/obelisk">obelisk</a> - <code>#elixir</code></li> <li><a href="https://orchid.netlify.com/">Orchid</a> - A beautiful and truly unique documentation engine and static site generator. - <code>#Java</code> <code>#Kotlin</code></li> <li><a href="http://pagegen.phnd.net/">Pagegen</a> - <code>#Python</code></li> <li><a href="https://phenomic.io/">Phenomic</a> - Modern static website generator based on the React and Webpack ecosystem. - <code>#Javascript</code> <code>#Node.js</code> <code>#React.js</code></li> <li><a href="https://github.com/JohnSundell/Publish">Publish</a> - enables entire websites to be built using Swift, and supports themes, plugins and tons of other powerful customization options - <code>#Swift</code></li> <li><a href="https://quarto.org/">Quarto</a> - An open-source scientific and technical publishing system with literate programming in multiple languages.</li> <li><a href="https://github.com/nozzle/react-static">React Static</a> - A progressive static-site framework for React. - <code>#React.js</code></li> <li><a href="https://github.com/scullyio/scully">Scully</a> - The Static Site Generator for Angular apps. - <code>#TypeScript</code> <code>#Angular</code></li> <li><a href="https://sculpin.io/">Sculpin</a> - <code>#PHP</code></li> <li><a href="https://gitlab.com/olfek/silssig">Silssig</a> - Dart powered static site generator for GitHub/GitLab pages and other similar services with support for markdown. - <code>#Dart</code></li> <li><a href="https://github.com/leafo/sitegen">Sitegen</a> - <code>#Lua</code> <code>#MoonScript</code></li> <li><a href="https://designmodo.com/slides/">Slides</a> - <code>#JavaScript</code></li> <li><a href="https://soupault.app/">Soupault</a> - A static site framework and post-processor that works with HTML element trees and can automatically manipulate them. - <code>#Lua</code> <code>#OCaml</code></li> <li><a href="https://styx-static.github.io/styx-site/">Styx</a> - The Purely Functional Static Site Generator. - <code>#Nix</code></li> <li><a href="https://kit.svelte.dev/">SvelteKit</a> - SvelteKit is a framework for building web applications of all sizes, with a beautiful development experience and flexible filesystem-based routing. - <code>#JavaScript</code> <code>#TypeScript</code> <code>#Svelte</code></li> <li><a href="https://github.com/braceio/tags">Tags</a> - <code>#Python</code></li> <li><a href="http://urubu.jandecaluwe.com/">Urubu</a> - <code>#Python</code></li> <li><a href="https://vuepress.vuejs.org/">VuePress</a> - Vue.js powered static site generator. - <code>#JavaScript</code></li> <li><a href="http://webgen.gettalong.org/">webgen</a> - <code>#Ruby</code></li> <li><a href="https://github.com/jnordberg/wintersmith">Wintersmith</a> - <code>#JavaScript</code> <code>#Node.js</code></li> <li><a href="https://github.com/bk/wmk">wmk</a> - Flexible and versatile, uses Mako templates - <code>#Python</code></li> <li><a href="https://github.com/jgm/yst">yst</a> - Powered by YAML. - <code>#Haskell</code></li> <li><a href="https://www.getzola.org">Zola</a> - Rust powered static site generator - <code>#Rust</code></li> </ul> <h3>Photography</h3> <ul> <li><a href="https://github.com/Jack000/Expose">Expose</a> - For photoessays. - <code>#Bash</code></li> <li><a href="https://github.com/waynezhang/foto">foto</a> - Yet another another publishing tool for minimalist photographers. - <code>#Go</code> <code>#Golang</code></li> <li><a href="https://github.com/Psycojoker/prosopopee/">Prosopopee</a> A static website generator to make beautiful customizable pictures galleries that tell a story - <code>#Python</code></li> <li><a href="https://sigal.readthedocs.org/en/latest/">Sigal</a> - <code>#Python</code></li> </ul> <h3>Single Page</h3> <ul> <li><a href="https://github.com/chrisdiana/cms.js">CMS.js</a> - A fully client-side, JavaScript Markdown site generator. - <code>#JavaScript</code></li> <li><a href="http://ricostacruz.com/flatdoc/">Flatdoc</a> - A small JavaScript file that fetches Markdown files and renders them as full pages. - <code>#JavaScript</code></li> </ul> <h3>Tools</h3> <ul> <li><a href="https://git.mills.io/prologic/zs">zs</a> - an extremely minimal static site generator. - <code>#Go</code></li> </ul> <h3>Wikis</h3> <ul> <li><a href="https://github.com/GitbookIO/gitbook">GitBook</a> - Modern documentation format and toolchain using Git and Markdown. - <code>#GitBook</code></li> <li><a href="https://github.com/gollum/gollum">Gollum</a> - Gollum is a simple wiki system built on top of Git. - <code>#Ruby</code></li> <li><a href="https://ikiwiki.info/">ikiwiki</a> - <code>#Perl</code></li> <li><a href="https://github.com/rust-lang/mdBook">mdBook</a> - Create book from markdown files. Like Gitbook but implemented in Rust. - <code>#Rust</code></li> <li><a href="https://orchid.run/wiki/learn/tutorials/how-to-document-kotlin#wiki">Orchid Wiki</a> - A Gitbook-like wiki for your Orchid site. - <code>#Orchid</code> <code>#Java</code> <code>#Kotlin</code></li> </ul> <h2>Helpful Tools and Services</h2> <ul> <li><a href="https://www.cloud66.com/">Cloud 66</a> - Builds and deploys static websites to your own cloud account.</li> <li><a href="http://www.formspree.io/">Formspree</a> - Adds functional forms to your static web sites.</li> <li><a href="http://git-annex.branchable.com/tips/setup_a_public_repository_on_a_web_site/">git-annex</a> - Configure git-annex for a public repository for a static web site.</li> <li><a href="https://jamstackthemes.dev/">JAMStack Themes</a> - A collection of themes filterable by static site generator and CMS support.</li> <li><a href="https://statichunt.com/">Statichunt</a> - An open sources directory of 700+ free themes and resources for static site generators submitted by the community.</li> </ul> <h3>Web Hosts</h3> <ul> <li><a href="http://aws.amazon.com/s3/">AWS S3</a></li> <li><a href="https://docs.microsoft.com/en-us/azure/static-web-apps/">Azure Static Web Apps</a></li> <li><a href="https://about.gitlab.com/product/pages/">GitLab Pages</a></li> <li><a href="https://kinsta.com/static-site-hosting/">Kinsta Static Site Hosting</a></li> <li><a href="https://www.netlify.com/">Netlify</a></li> <li><a href="https://surge.sh/">Surge</a></li> <li><a href="https://vercel.com">Vercel</a></li> </ul> <h2>Other Lists of Static Web Site Generators</h2> <ul> <li><a href="https://bejamas.io/blog/static-site-generators/">Beginner’s Guide to Static Site Generators</a></li> <li><a href="https://snipcart.com/blog/choose-best-static-site-generator">Best Static Site Generators in 2019</a></li> <li><a href="https://headlesscms.org/">HeadlessCMS</a></li> <li><a href="http://staticsitegenerators.net/">Static Site Generators</a></li> <li><a href="https://www.staticgen.com/">StaticGen</a></li> </ul> <h2>Related Awesome</h2> <p>Awesome lists that help you with your new static web site.</p> <ul> <li><a href="https://github.com/therebelrobot/awesome-bootstrap">Awesome Bootstrap</a> - A curated list of free Bootstrap themes.</li> <li><a href="https://github.com/diegocard/awesome-html5">Awesome HTML5</a></li> <li><a href="https://github.com/heyalexej/awesome-images">Awesome Images</a> - A curated list of amazingly awesome free (stock) photo resources.</li> <li><a href="https://github.com/b-long/awesome-static-hosting">Awesome Static Hosting Providers</a> - A collection of awesome static hosting providers and related resources.</li> <li><a href="https://github.com/aharris88/awesome-static-website-services">Awesome Static Website Services</a> - A curated list of awesome static websites services</li> <li><a href="https://github.com/shime/creative-commons-media">Creative Commons Media</a> - A curated list of resources that provide media licensed under Creative Commons licenses.</li> <li><a href="https://github.com/dypsilon/frontend-dev-bookmarks">Frontend Development</a></li> </ul> <h2>Static Web Site Generators in Other Awesome Lists</h2> <ul> <li><a href="https://github.com/forhappy/awesome-lua#static-site-generator">forhappy/awesome-lua - Static Site Generator</a></li> <li><a href="https://github.com/h4cc/awesome-elixir#static-page-generation">h4cc/awesome-elixir - Static Page Generation</a></li> <li><a href="https://github.com/kdabir/awesome-groovy#static-web">kdabir/awesome-groovy - Static Web</a></li> <li><a href="https://github.com/markets/awesome-ruby#static-site-generation">markets/awesome-ruby - Static Site Generators</a></li> <li><a href="https://github.com/Sdogruyol/awesome-ruby#static-page-generation">Sdogruyol/awesome-ruby - Static Page Generation</a></li> <li><a href="https://github.com/sindresorhus/awesome-nodejs#static-site-generators">sindresorhus/awesome-nodejs - Static Site Generators</a></li> <li><a href="https://github.com/vinta/awesome-python#static-site-generator">vinta/awesome-python - Static Site Generators</a></li> <li><a href="https://github.com/ziadoz/awesome-php#static-site-generators">ziadoz/awesome-php - Static Site Generators</a></li> </ul> https://tedneward.github.io/Research/presentation/tabrisjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/tabrisjs/index.html <p><a href="https://tabris.com/">Website</a> | <a href="https://github.com/eclipsesource/tabris-js/">Source</a></p> <p>Cross-platform, true native. Tabris.js does not use WebViews for rendering the UI. Instead, it creates native widgets on the mobile platform via a JavaScript to native bridge. ... use existing JavaScript libraries, node modules, and Cordova plugins to build your apps. Tabris.js implements many APIs core to web applications, e.g. XMLHttpRequest or the 2D canvas. Create gorgeous graphs with chart.js, easily integrate underscore.js or access device features with Cordova device.</p> https://tedneward.github.io/Research/presentation/threejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/threejs/index.html <p><a href="https://threejs.org/">Website</a> | <a href="https://github.com/mrdoob/three.js/">Source</a></p> https://tedneward.github.io/Research/presentation/titanium/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/titanium/index.html <p><a href="https://titaniumsdk.com/">Website</a> | <a href="https://github.com/tidev/titanium_mobile">Source</a></p> <blockquote> <p>"All private source code of the Titanium SDK will be made public in the open source Titanium SDK github repository by March 1, 2022"</p> </blockquote> <p>Uses extensions for VSCode or Atom as its IDE. There is also a CLI (<code>npm install titanium -g</code>) available.</p> <p>The Titanium SDK supports two project types: Alloy and Classic. Alloy is an MVC framework that lets you develop applications in less time and with less code than is possible in a Classic project. In general, it's recommended you use Alloy for your projects.</p> <p>Example code from <a href="https://titaniumsdk.com/guide/Titanium_SDK/Titanium_SDK_Getting_Started/Hello_Titanium_App_Tutorial.html">tutorial</a>:</p> <pre><code class="language-xml">&lt;Alloy&gt; &lt;Window class="container"&gt; &lt;Label id="label" onClick="doClick"&gt;Hello, Titanium&lt;/Label&gt; &lt;/Window&gt; &lt;/Alloy&gt; </code></pre> <p>Uses XML as its visual layout (replacement for HTML, similar in spirit to XAML, sort of), and Javascript as its event-handling/programmatic support language, and "TSS" (a variant of CSS) for styling and visual organization. Renders on both iOS and Android, so it's a single-source/multi-deliverable tool.</p> <p>Also does game development?</p> https://tedneward.github.io/Research/presentation/uhtml/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/uhtml/index.html <p><a href="https://www.npmjs.com/package/uhtml">NPM</a> | <a href="https://github.com/WebReflection/uhtml">Source</a></p> https://tedneward.github.io/Research/presentation/valdi/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/valdi/index.html <p><a href="https://github.com/Snapchat/Valdi">Source</a></p> https://tedneward.github.io/Research/presentation/vivaldi/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/vivaldi/index.html <p><a href="https://vivaldi.com/">Website</a> | Source unavailable, free download</p> https://tedneward.github.io/Research/presentation/openvidu/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/openvidu/index.html <p><a href="https://openvidu.io/">Website</a> | <a href="https://github.com/OpenVidu/openvidu">Source</a></p> https://tedneward.github.io/Research/presentation/p5/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/p5/index.html <p><a href="https://p5js.org/">Website</a> | <a href="https://github.com/processing/p5.js/">Source</a></p> <p>Part of the Processing project.</p> https://tedneward.github.io/Research/presentation/photino/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/photino/index.html <p><a href="https://www.tryphotino.io/">Website</a> | <a href="https://github.com/tryphotino">GitHub</a> | <a href="https://docs.tryphotino.io/">Docs</a></p> <h2>Getting started</h2> <pre><code>dotnet new -i TryPhotino.VSCode.Project.Templates\ &amp;&amp; dotnet new photinoapp -o MyPhotinoApp\ &amp;&amp; cd MyPhotinoApp\ &amp;&amp; dotnet run </code></pre> https://tedneward.github.io/Research/presentation/pretext/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/pretext/index.html <p><a href="https://chenglou.me/pretext/">Website</a> | <a href="https://github.com/chenglou/pretext">Source</a></p> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://venturebeat.com/technology/midjourney-engineer-debuts-new-vibe-coded-open-source-standard-pretext-to">Midjourney engineer debuts new vibe coded, open source standard Pretext to revolutionize web design</a></li> </ul> https://tedneward.github.io/Research/presentation/pwa/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/pwa/index.html <p><a href="https://web.dev/learn/pwa/?utm_source=tldrnewsletter">Learn PWA</a></p> https://tedneward.github.io/Research/presentation/oqtane/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/oqtane/index.html <p><a href="https://www.oqtane.org/">Website</a> | <a href="https://github.com/oqtane/framework">Source</a></p> <p>"Oqtane was inspired by DotNetNuke, the pionering open source .NET CMS. Specifically it supports multi-tenancy, a fully dynamic page compositing model, designer friendly skins, extensibility via third party modules, and a familiar data model."</p> https://tedneward.github.io/Research/presentation/panel/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/panel/index.html <p><a href="https://panel.holoviz.org/index.html">Website</a> | <a href="https://github.com/holoviz/panel">Source</a></p> <h2><a href="https://panel.holoviz.org/how_to/wasm/standalone.html">Using Pyodide and Panel</a></h2> <p>Create a file called script.html with the following content:</p> <pre><code>&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;meta charset="UTF-8"&gt; &lt;meta name="viewport" content="width=device-width,initial-scale=1.0"&gt; &lt;script type="text/javascript" src="https://cdn.jsdelivr.net/pyodide/v0.28.2/full/pyodide.js"&gt;&lt;/script&gt; &lt;script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-3.8.1.js"&gt;&lt;/script&gt; &lt;script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.8.1.min.js"&gt;&lt;/script&gt; &lt;script type="text/javascript" src="https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.8.1.min.js"&gt;&lt;/script&gt; &lt;script type="text/javascript" src="https://cdn.jsdelivr.net/npm/@holoviz/panel@1.8.4/dist/panel.min.js"&gt;&lt;/script&gt; &lt;/head&gt; &lt;body&gt; &lt;div id="simple_app"&gt;&lt;/div&gt; &lt;script type="text/javascript"&gt; async function main() { let pyodide = await loadPyodide(); await pyodide.loadPackage("micropip"); const micropip = pyodide.pyimport("micropip"); await micropip.install([ "https://cdn.holoviz.org/panel/1.8.4/dist/wheels/bokeh-3.8.1-py3-none-any.whl", "https://cdn.holoviz.org/panel/1.8.4/dist/wheels/panel-1.8.4-py3-none-any.whl" ]); pyodide.runPython(` import panel as pn pn.extension(sizing_mode="stretch_width") slider = pn.widgets.FloatSlider(start=0, end=10, name='Amplitude') def callback(new): return f'Amplitude is: {new}' pn.Row(slider, pn.bind(callback, slider)).servable(target='simple_app'); `); } main(); &lt;/script&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>Serve the app with: <code>python -m http.server</code></p> <p>Open the app in your browser at <a href="http://localhost:8000/script.html">http://localhost:8000/script.html</a>.</p> https://tedneward.github.io/Research/presentation/pixijs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/pixijs/index.html <p><a href="http://www.pixijs.com/">Website</a> | <a href="https://pixijs.download/release/docs/index.html">Docs</a> | <a href="https://pixijs.io/examples/#/demos-basic/container.js">Examples</a> | <a href="https://github.com/pixijs/pixijs">Source</a></p> <p><a href="https://github.com/kittykatattack/learningPixi">Tutorial</a></p> <p>Features</p> <ul> <li>WebGL renderer (with automatic smart batching allowing for REALLY fast performance)</li> <li>Canvas renderer (Fastest in town!)</li> <li>Full scene graph</li> <li>Super easy to use API (similar to the flash display list API)</li> <li>Support for texture atlases</li> <li>Asset loader / sprite sheet loader</li> <li>Auto-detect which renderer should be used</li> <li>Full Mouse and Multi-touch Interaction</li> <li>Text</li> <li>BitmapFont text</li> <li>Multiline Text</li> <li>Render Texture</li> <li>Primitive Drawing</li> <li>Masking</li> <li>Filters</li> <li>User Plugins</li> </ul> <h3>Basic Usage Example</h3> <pre><code class="language-js">import * as PIXI from 'pixi.js'; // The application will create a renderer using WebGL, if possible, // with a fallback to a canvas render. It will also setup the ticker // and the root stage PIXI.Container const app = new PIXI.Application(); // The application will create a canvas element for you that you // can then insert into the DOM document.body.appendChild(app.view); // load the texture we need app.loader.add('bunny', 'bunny.png').load((loader, resources) =&gt; { // This creates a texture from a 'bunny.png' image const bunny = new PIXI.Sprite(resources.bunny.texture); // Setup the position of the bunny bunny.x = app.renderer.width / 2; bunny.y = app.renderer.height / 2; // Rotate around the center bunny.anchor.x = 0.5; bunny.anchor.y = 0.5; // Add the bunny to the scene we are building app.stage.addChild(bunny); // Listen for frame updates app.ticker.add(() =&gt; { // each frame we spin the bunny around a bit bunny.rotation += 0.01; }); }); </code></pre> https://tedneward.github.io/Research/presentation/processing/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/processing/index.html <p><a href="https://processing.org/">Processing Website</a> | <a href="https://github.com/processing">Source</a></p> <p><a href="https://android.processing.org/">Processing for Android Website</a> | <a href="https://github.com/processing/processing-android">Source</a></p> <p><a href="https://p5js.org/">p5js Website</a> | <a href="https://github.com/processing/p5.js">p5js Source</a></p> <p><a href="https://py.processing.org/">Processing.py</a> | <a href="https://github.com/jdf/processing.py">Source</a></p> https://tedneward.github.io/Research/presentation/python/flet/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/python/flet/index.html <p><a href="https://flet.dev/">Website</a> | <a href="https://github.com/flet-dev/flet">Source</a></p> <h3>What is Flet?</h3> <p>Flet is a framework that allows building web, desktop and mobile applications in Python without prior experience in frontend development.</p> <p>You can build a UI for your program with Flet <a href="https://flet.dev/docs/controls">controls</a> which are based on <a href="https://flutter.dev/">Flutter</a> by Google. Flet goes beyond merely wrapping Flutter widgets. It adds its own touch by combining smaller widgets, simplifying complexities, implementing UI best practices, and applying sensible defaults. This ensures that your applications look stylish and polished without requiring additional design efforts on your part.</p> <h3>Flet app example</h3> <p>Create a sample "Counter" app:</p> <p>counter.py</p> <pre><code>import flet as ft def main(page: ft.Page): page.title = "Flet counter example" page.vertical_alignment = ft.MainAxisAlignment.CENTER txt_number = ft.TextField(value="0", text_align=ft.TextAlign.RIGHT, width=100) def minus_click(e): txt_number.value = str(int(txt_number.value) - 1) page.update() def plus_click(e): txt_number.value = str(int(txt_number.value) + 1) page.update() page.add( ft.Row( [ ft.IconButton(ft.Icons.REMOVE, on_click=minus_click), txt_number, ft.IconButton(ft.Icons.ADD, on_click=plus_click), ], alignment=ft.MainAxisAlignment.CENTER, ) ) ft.app(main) </code></pre> <p>To run the app install <code>flet</code> module (<a href="https://flet.dev/docs/getting-started">create a new Flet environment</a>):</p> <pre><code>pip install flet </code></pre> <p>and <a href="https://flet.dev/docs/getting-started/running-app">run the program</a>:</p> <pre><code>flet run counter.py </code></pre> <p>The app will be started in a native OS window - what a nice alternative to Electron!</p> <h3>macOS</h3> <p><img src="https://flet.dev/img/docs/getting-started/flet-counter-macos.png" alt=""></p> <h3>Windows</h3> <p><img src="https://flet.dev/img/docs/getting-started/flet-counter-windows.png" alt=""></p> <p>Now, run your app as a web app:</p> <pre><code>flet run --web counter.py </code></pre> <p>A new browser window or tab will be opened:</p> <p><img src="https://flet.dev/img/docs/getting-started/flet-counter-safari.png" alt=""></p> https://tedneward.github.io/Research/presentation/orbitiny/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/orbitiny/index.html <p><a href="https://gitea.com/sasko.usinov/orbitiny-desktop">Source</a></p> <h2>Reading</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://www.zdnet.com/article/orbitiny-portable-modular-unique-linux-desktop/">This new Linux desktop runs like an app on your existing desktop</a></li> </ul> https://tedneward.github.io/Research/presentation/paper-websites/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/paper-websites/index.html <p><a href="https://daily.tinyprojects.dev/paper_website">Blog post</a> | <a href="https://paperwebsite.com/">Website</a></p> https://tedneward.github.io/Research/presentation/polymer/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/polymer/index.html <p>The Polymer library is in maintenance mode. For new development, we recommend <a href="./lit">Lit</a>.</p> <p><a href="https://polymer-library.polymer-project.org/">Website</a></p> https://tedneward.github.io/Research/presentation/prosemirror/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/prosemirror/index.html <p><a href="https://prosemirror.net">Website</a> | <a href="https://github.com/ProseMirror/prosemirror">Source</a></p> https://tedneward.github.io/Research/presentation/ooui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ooui/index.html <p><a href="https://github.com/praeclarum/Ooui">Source</a> | <a href="http://ooui.mecha.parts/">Interactive examples</a> and <a href="https://github.com/praeclarum/Ooui/tree/master/Samples">source</a></p> <h3>Example App</h3> <pre><code>using System; using Ooui; class Program { static void Main(string[] args) { // Create the UI var button = new Button("Click me!"); // Add some logic to it var count = 0; button.Click += (s, e) =&gt; { count++; button.Text = $"Clicked {count} times"; }; // Publishing makes an object available at a given URL // The user should be directed to http://localhost:8080/shared-button UI.Publish ("/shared-button", button); // Don't exit the app until someone hits return Console.ReadLine (); } } </code></pre> <p>Make sure to add a reference to Ooui before you start running!</p> <pre><code class="language-shell">dotnet add package Ooui dotnet run </code></pre> <h3><a href="">Button-XAML Example</a></h3> <p>An example of doing an interactive Xamarin Forms app.</p> <p>XAML:</p> <pre><code>&lt;?xml version="1.0" encoding="utf-8" ?&gt; &lt;ContentPage xmlns="http://xamarin.com/schemas/2014/forms" xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml" x:Class="ButtonXaml.ButtonXamlPage"&gt; &lt;StackLayout Padding="20"&gt; &lt;Label Text="Welcome to Xamarin.Forms!" FontSize="32" FontAttributes="Bold" /&gt; &lt;Entry x:Name="LabelCount" Text="Click Count: 0"/&gt; &lt;Button Text="Tap for click count!" Clicked="OnButtonClicked" /&gt; &lt;/StackLayout&gt; &lt;/ContentPage&gt; </code></pre> <p>XAML.CS:</p> <pre><code>using System; using Xamarin.Forms; namespace ButtonXaml { public partial class ButtonXamlPage { int count = 0; public ButtonXamlPage() { InitializeComponent(); } public void OnButtonClicked(object sender, EventArgs args) { count++; LabelCount.Text = $"Click Count: {count}"; } } } </code></pre> <p>C#:</p> <pre><code>using Ooui; using Xamarin.Forms; namespace Samples { public class ButtonXamlSample : ISample { public string Title =&gt; "Xamarin.Forms Button XAML"; public string Path =&gt; "buttons"; public Ooui.Element CreateElement () { var page = new ButtonXaml.ButtonXamlPage (); return page.GetOouiElement (); } public void Publish() { UI.Publish(Path, CreateElement); } } } </code></pre> https://tedneward.github.io/Research/presentation/otterwiki/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/otterwiki/index.html <p><a href="https://otterwiki.com/">Website</a> | <a href="https://github.com/redimp/otterwiki">Source</a></p> https://tedneward.github.io/Research/presentation/patterns/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/patterns/index.html <h2>Articles</h2> <p><a href="https://www.smashingmagazine.com/taking-pattern-libraries-next-level/">Taking Pattern Libraries to the Next Level</a></p> <p><a href="https://www.smashingmagazine.com/2009/06/10-ui-design-patterns-you-should-be-paying-attention-to/">10 UI Design Patterns You Should Be Paying Attention To</a></p> <p><a href="https://www.smashingmagazine.com/2010/02/designing-user-interfaces-for-business-web-applications/">Designing UI for Business Web Apps</a></p> <p>7 Rules for Creating Gorgeous UI: <a href="https://www.learnui.design/blog/7-rules-for-creating-gorgeous-ui-part-1.html">Part 1</a> <a href="https://www.learnui.design/blog/7-rules-for-creating-gorgeous-ui-part-2.html">Part 2</a>:</p> <ol> <li>Light comes from the sky <ol> <li>Black and white first</li> <li>Double your whitespace</li> <li>Learn the methods of overlaying text on images</li> <li>Make text pop — and un-pop</li> <li>Use only good fonts</li> <li>Steal like an artist</li> </ol> </li> </ol> <h2>Websites</h2> <p><a href="http://designingwebinterfaces.com/">Designing Web Interfaces</a>: Supporting website for the book of the same name. <em>(Posts until 2012)</em></p> <ul> <li><a href="http://designingwebinterfaces.com/designing-web-interfaces-12-screen-patterns">Standard Screen Patterns</a> <ul> <li><a href="http://designingwebinterfaces.com/essential_controls">Essential Controls</a></li> <li><a href="http://designingwebinterfaces.com/15-common-components">Components for Commonly Requested Features</a></li> </ul> </li> </ul> <p><a href="https://centercentre.com/">https://centercentre.com/</a> - For UX leaders.</p> <p><a href="https://ui-patterns.com/">https://ui-patterns.com/</a> - a large collection of design patterns for UI designers to gain inspiration from. The site allows users to keep sets of their own (publicly accessible to site visitors) so that you can see other UI design pattern collections.</p> <p><a href="https://uipatternfactory.com/">https://uipatternfactory.com/</a> - The UI Pattern Factory is a UI design library and gallery. What’s great at UIPF is that they sometimes share videos in each entry to improve the description of design problems and solutions. Entries are further enhanced by user-submitted examples of the pattern, which they archive in their Flickr group: UIPatternFactory.com.</p> <p><a href="https://www.lukew.com/ff/">https://www.lukew.com/ff/</a> - the weblog of UI designer and author Luke Wroblewski. You can learn about designing for humans by reading through his insightful articles on the site and find useful lists such as a collection of resources on website buttons.</p> <p><a href="https://www.hcipatterns.org/">https://www.hcipatterns.org/</a> - This is the homepage of people interested in pattern languages in human-computer interaction (HCI) and user interface design and the IFIP TC13 HCI Patterns Task Group.</p> <p><a href="https://speckyboy.com/topic/web-design-inspiration/">https://speckyboy.com/topic/web-design-inspiration/</a> - Web design inspiration, a collection of ideas</p> <h3>Flickr Groups:</h3> <p><a href="https://www.flickr.com/groups/namics_uipatterns/">https://www.flickr.com/groups/namics_uipatterns/</a></p> <p><a href="https://www.flickr.com/groups/designpatterns/">https://www.flickr.com/groups/designpatterns/</a></p> https://tedneward.github.io/Research/presentation/preact/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/preact/index.html <p><a href="https://preactjs.com/">Website</a> | <a href="https://github.com/preactjs/preact">Source</a></p> https://tedneward.github.io/Research/presentation/pug/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/pug/index.html <p><a href="https://pugjs.org/">Website</a> | <a href="https://pugjs.org/api/getting-started.html">Docs</a> | <a href="https://github.com/pugjs/pug">Source</a></p> <h3><a href="https://learnxinyminutes.com/docs/pug/">Learn X in Y Minutes Quick Intro</a></h3> <pre><code class="language-pug"> //- Single Line Comment //- Multi Line Comment //- ---TAGS--- //- Basic div //- &lt;div&gt;&lt;/div&gt; h1 //- &lt;h1&gt;&lt;/h1&gt; my-customTag //- &lt;my-customTag&gt;&lt;/my-customTag&gt; //- Sibling div div //- &lt;div&gt;&lt;/div&gt; &lt;div&gt;&lt;/div&gt; //- Child div div //- &lt;div&gt; &lt;div&gt;&lt;/div&gt; &lt;/div&gt; //- Text h1 Hello there //- &lt;h1&gt;Hello there&lt;/h1&gt; //- Multi Line Text div. Hello There //- &lt;div&gt; Hello There &lt;/div&gt; //- ---ATTRIBUTES--- div(class="my-class" id="my-id" my-custom-attrs="data" enabled) //- &lt;div class="my-class" id="my-id" my-custom-attrs="data" enabled&gt;&lt;/div&gt; //- Short Hand span.my-class //- &lt;span class="my-class"&gt;&lt;/span&gt; .my-class //- &lt;div class="my-class"&gt;&lt;/div&gt; div#my-id //- &lt;div id="my-id"&gt;&lt;/div&gt; div#my-id.my-class //- &lt;div class="my-class" id="my-id"&gt;&lt;/div&gt; //- ---JS--- - const lang = "pug"; //- Multi Line JS - const lang = "pug"; const awesome = true; //- JS Classes - const myClass = ['class1', 'class2', 'class3'] div(class=myClass) //- &lt;div class="class1 class2 class3"&gt;&lt;/div&gt; //- JS Styles - const myStyles = {'color':'white', 'background-color':'blue'} div(styles=myStyles) //- &lt;div styles="{&amp;quot;color&amp;quot;:&amp;quot;white&amp;quot;,&amp;quot;background-color&amp;quot;:&amp;quot;blue&amp;quot;}"&gt;&lt;/div&gt; //- JS Attributes - const myAttributes = {"src": "photo.png", "alt": "My Photo"} img&amp;attributes(myAttributes) //- &lt;img src="photo.png" alt="My Photo"&gt; - let disabled = false input(type="text" disabled=disabled) //- &lt;input type="text"&gt; - disabled = true input(type="text" disabled=disabled) //- &lt;input type="text" disabled&gt; //- JS Templating - const name = "Bob"; h1 Hi #{name} h1= name //- &lt;h1&gt;Hi Bob&lt;/h1&gt; //- &lt;h1&gt;Bob&lt;/h1&gt; //- ---LOOPS--- //- 'each' and 'for' do the same thing we will use 'each' only. each value, i in [1,2,3] p=value //- &lt;p&gt;1&lt;/p&gt; &lt;p&gt;2&lt;/p&gt; &lt;p&gt;3&lt;/p&gt; each value, index in [1,2,3] p=value + '-' + index //- &lt;p&gt;1-0&lt;/p&gt; &lt;p&gt;2-1&lt;/p&gt; &lt;p&gt;3-2&lt;/p&gt; each value in [] p=value //- each value in [] p=value else p No Values are here //- &lt;p&gt;No Values are here&lt;/p&gt; //- ---CONDITIONALS--- - const number = 5 if number &lt; 5 p number is less then 5 else if number &gt; 5 p number is greater then 5 else p number is 5 //- &lt;p&gt;number is 5&lt;/p&gt; - const orderStatus = "Pending"; case orderStatus when "Pending" p.warn Your order is pending when "Completed" p.success Order is Completed. when -1 p.error Error Occurred default p No Order Record Found //- &lt;p class="warn"&gt;Your order is pending&lt;/p&gt; //- --INCLUDE-- //- File path -&gt; "includes/nav.png" h1 Company Name nav a(href="index.html") Home a(href="about.html") About Us //- File path -&gt; "index.png" html body include includes/nav.pug //- &lt;html&gt; &lt;body&gt; &lt;h1&gt;Company Name&lt;/h1&gt; &lt;nav&gt;&lt;a href="index.html"&gt;Home&lt;/a&gt;&lt;a href="about.html"&gt;About Us&lt;/a&gt;&lt;/nav&gt; &lt;/body&gt; &lt;/html&gt; //- Importing JS and CSS script include scripts/index.js style include styles/theme.css //- ---MIXIN--- mixin basic() div Hello +basic("Bob") //- &lt;div&gt;Hello&lt;/div&gt; mixin comment(name, comment) div span.comment-name= name div.comment-text= comment +comment("Bob", "This is Awesome") //- &lt;div&gt;Hello&lt;/div&gt; </code></pre> https://tedneward.github.io/Research/presentation/kraken/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/kraken/index.html <p><a href="https://openkraken.com/en-US">Website</a> | <a href="https://github.com/openkraken/kraken">Source</a></p> <p>"Kraken is a high-performance rendering engine based on W3C standards. Kraken's bottom layer is rendered based on Flutter, and its self-drawn rendering feature ensures multi-end consistency. The upper layer is implemented based on W3C standards and has a very large front -end developer ecosystem."</p> https://tedneward.github.io/Research/presentation/libui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/libui/index.html <p><a href="https://github.com/andlabs/libui">Source</a></p> https://tedneward.github.io/Research/presentation/litho/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/litho/index.html <p><a href="https://fblitho.com/">Website</a> | <a href="https://github.com/facebook/litho">Source</a></p> <ul> <li>Declarative: Litho uses a declarative API to define UI components. You simply describe the layout for your UI based on a set of immutable inputs and the framework takes care of the rest.</li> <li>Asynchronous layout: Litho can measure and layout your UI ahead of time without blocking the UI thread.</li> <li>View flattening: Litho uses Yoga for layout and automatically reduces the number of ViewGroups that your UI contains.</li> <li>Fine-grained recycling: Any component such as a text or image can be recycled and reused anywhere in the UI.</li> </ul> https://tedneward.github.io/Research/presentation/lucee/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lucee/index.html <p><a href="https://www.lucee.org/">Website</a> | <a href="https://github.com/lucee">Source</a></p> <p>Looks like another Java Web framework, supporting <a href="../../languages/cfml">CFML</a>, albeit with some interesting elements to it.</p> <p>Forked from <a href="https://github.com/getrailo/railo">Railo</a>, another CFML engine.</p> https://tedneward.github.io/Research/presentation/marp/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/marp/index.html <p><a href="https://marp.app">Website</a> | <a href="https://github.com/marp-team/marp">Source</a></p> <p>Build presentations in Markdown. (Broken out into components, could be useful in other ways as well.)</p> https://tedneward.github.io/Research/presentation/metawidget/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/metawidget/index.html <p>"Metawidget does [UI generation] without introducing new technologies. It inspects your existing back-end architecture (such as JSON, REST, existing annotations, existing configuration files) and creates widgets native to your existing front-end framework (such as JavaScript, Java Server Faces, Android). Metawidget does not replace or restrict your existing UI framework. It does not try to 'own' your UI. Its Open Source license allows the use of Metawidget in open source and commercial projects."</p> <p><a href="http://metawidget.org/">Website</a> | <a href="http://metawidget.org/download.php">Download</a> | <a href="https://github.com/metawidget/metawidget">Source</a></p> <p>Last update appears to be 2015; according to the author, this is due to maturity and personal distraction, but that "others have taken up the mantle" and points to ports to major SPA frameworks: <a href="https://github.com/SDS-React-Metawidget/react-metawidget">React</a> and <a href="https://github.com/AmitsBizruntime/MetawidetA2">Angular</a>.</p> https://tedneward.github.io/Research/presentation/mobileangularui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/mobileangularui/index.html <p><a href="http://mobileangularui.com/">Website</a> | <a href="https://github.com/mcasimir/mobile-angular-ui">Source</a></p> https://tedneward.github.io/Research/presentation/nanogui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nanogui/index.html <p><a href="https://github.com/wjakob/nanogui">Source</a></p> https://tedneward.github.io/Research/presentation/noesisgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/noesisgui/index.html <p><a href="https://www.noesisengine.com/">Website</a> | Commercial/closed-source</p> <p>Pros: battle tested / production ready, impressive list of platforms and backends, markup language, best-in-class visual editor (Microsoft Blend), great documentation with many examples<br> Cons: closed-source, limited free license, developers and UI designers must use Windows, heavy setup and integration</p> https://tedneward.github.io/Research/presentation/ohmyform/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ohmyform/index.html <p><a href="https://ohmyform.com/">Website</a> | <a href="https://github.com/ohmyform">Source</a></p> <p>Looks to be in Python.</p> <p>Features</p> <ul> <li>Multi-Language Support</li> <li>11 possible question types</li> <li>Editable start and end pages</li> <li>Export Submissions to XLS, JSON or CSV</li> <li>Native Analytics and Google Analytics Support</li> <li>Embeddable Forms</li> <li>Forms as a Service API</li> <li>Customizable Notifications on Form Submission</li> <li>Web Hooks on Form Submission</li> <li>Deployable with Heroku and DockerHub</li> <li>PostgreSQL and sqlite</li> </ul> https://tedneward.github.io/Research/presentation/lcarsde/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lcarsde/index.html <p><a href="https://lcarsde.github.io/index.html">Website</a> | <a href="https://github.com/lcarsde">Source</a></p> <p>Interesting, but doesn't look really ready yet. (Aug 2025)</p> https://tedneward.github.io/Research/presentation/lightning/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lightning/index.html <p><a href="https://developer.salesforce.com/docs/component-library/overview/components">Components</a> | <a href="https://www.lightningdesignsystem.com/">Design System</a> | <a href="https://developer.salesforce.com/docs/atlas.en-us.lightning.meta/lightning/intro_framework.htm">Lightning Aura Components</a></p> https://tedneward.github.io/Research/presentation/livekit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/livekit/index.html <p><a href="https://livekit.io/">Website</a> | <a href="https://github.com/livekit/livekit-server">Source</a></p> https://tedneward.github.io/Research/presentation/lynxjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lynxjs/index.html <p><a href="https://lynxjs.org/">Website</a> | <a href="https://github.com/lynx-family/lynx">Source</a></p> https://tedneward.github.io/Research/presentation/maui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/maui/index.html <p><a href="https://learn.microsoft.com/en-us/dotnet/maui">MAUI Docs</a> (<a href="https://learn.microsoft.com/en-us/dotnet/maui/opbuildpdf/fb09dc9e6631000255afc54e0ec4c50a/toc.pdf?branch=live&amp;view=net-maui-7.0">PDF</a>) | <a href="https://github.com/dotnet/maui">Github</a></p> <h2>Lists and resources</h2> <ul> <li><a href="https://github.com/jsuarezruiz/awesome-dotnet-maui">Awesome MAUI</a></li> </ul> https://tedneward.github.io/Research/presentation/microui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/microui/index.html <p><a href="https://github.com/rxi/microui">Source</a></p> https://tedneward.github.io/Research/presentation/morphic/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/morphic/index.html <p><a href="https://en.wikipedia.org/wiki/Morphic_(software)">Wikipedia</a></p> <p><a href="http://wiki.squeak.org/squeak/morphic">Squeak Morphic</a> | <a href="http://rmod-files.lille.inria.fr/FreeBooks/CollectiveNBlueBook/morphic.final.pdf">Intro to Squeak Morphic</a></p> <p><a href="https://handbook.selflanguage.org/2017.1/morphic.html">Self Morphic</a></p> <p><a href="https://github.com/jmoenig/morphic.js">morphic.js</a> | <a href="https://thiscontext.com/2017/06/28/a-faster-morphic-with-morphic-js/">"A faster Morphic with morphic.js"</a></p> <p><a href="https://github.com/jmoenig/morphic.py">Morphic for Python</a></p> <p>See also <a href="../lively">Lively Kernel</a>.</p> https://tedneward.github.io/Research/presentation/nativeblocks/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nativeblocks/index.html <p><a href="https://nativeblocks.io/">Website</a> | <a href="https://github.com/nativeblocks">Source</a> | <a href="https://nativeblocks.io/docs/get-started/introduction/">Docs</a> | <a href="https://github.com/nativeblocks/nbx">NBX UI DSL</a></p> https://tedneward.github.io/Research/presentation/nuklear/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nuklear/index.html <p><a href="https://immediate-mode-ui.github.io/Nuklear/doc/nuklear.html">Website</a> | <a href="https://github.com/Immediate-Mode-UI/Nuklear">Source</a> (<a href="https://github.com/vurtun/nuklear">Archived repo</a>)</p> <p>Pros: single header :o, no dependencies, pretty</p> <p>Bindings (some out of date) for Java, D, Go, Rust, Chicken, Nim, Lua, Python, C#, V</p> https://tedneward.github.io/Research/presentation/one/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/one/index.html <p><a href="https://onestack.dev/">Website</a> | <a href="https://github.com/onejs/one">Source</a></p> <p>One takes Vite, makes it serve React web and React Native. Then adds FS routes, render modes, loaders, middleware, a CLI, Hono, etc.</p> <p>One is a simpler framework because it's being designed alongside a sync engine, Zero.</p> https://tedneward.github.io/Research/presentation/leanqt/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/leanqt/index.html <p><a href="https://github.com/rochus-keller/LeanQt">Source</a></p> https://tedneward.github.io/Research/presentation/lisagui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lisagui/index.html <p><a href="https://lisagui.com/">Website</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://hackaday.com/2025/07/14/explore-the-granddaddy-of-all-macs-with-lisagui/">https://hackaday.com/2025/07/14/explore-the-granddaddy-of-all-macs-with-lisagui/</a></li> </ul> https://tedneward.github.io/Research/presentation/lively/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lively/index.html <p><a href="https://lively-kernel.org/">Website</a> | <a href="https://github.com/LivelyKernel/LivelyKernel">Source</a></p> <p>Modern <a href="../morphic">Morphic</a> interface for browsers.</p> https://tedneward.github.io/Research/presentation/markedjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/markedjs/index.html <p><a href="https://marked.js.org/">Website</a> | <a href="https://github.com/markedjs/marked">Source</a></p> <p>Written in TypeScript, usable either server-side/static-generation or client-side/render-on-demand.</p> https://tedneward.github.io/Research/presentation/mdp/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/mdp/index.html <p><a href="https://github.com/visit1985/mdp">Source</a></p> https://tedneward.github.io/Research/presentation/miso/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/miso/index.html <p><a href="https://github.com/dmjio/miso">Source</a></p> <blockquote> <p>It features a virtual-dom, recursive diffing / patching algorithm, attribute and property normalization, event delegation, event batching, SVG, Server-sent events, Websockets, type-safe servant-style routing and an extensible Subscription-based subsystem. Inspired by Elm, Redux and Bobril. Miso is pure by default, but side effects (like XHR) can be introduced into the system via the Effect data type. Miso makes heavy use of the GHCJS FFI and therefore has minimal dependencies. Miso can be considered a shallow embedded domain-specific language for modern web programming.</p> </blockquote> <p><a href="https://github.com/smelc/miso-darkcraw">Card game using Miso</a></p> https://tedneward.github.io/Research/presentation/mux/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/mux/index.html <p><a href="https://mux.com/">Website</a></p> <p>SDK support for Ruby Go Python PHP Elixir Node</p> https://tedneward.github.io/Research/presentation/nextjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nextjs/index.html <p><a href="https://nextjs.org">Website</a> | <a href="https://github.com/vercel/next.js">Source</a></p> https://tedneward.github.io/Research/presentation/nwjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nwjs/index.html <p><a href="https://nwjs.io/">Website</a> | <a href="https://github.com/nwjs/nw.js">Source</a> | <a href="https://docs.nwjs.io/">Docs</a></p> https://tedneward.github.io/Research/presentation/onsenui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/onsenui/index.html <p><a href="https://onsen.io/">Website</a> | <a href="https://github.com/OnsenUI/OnsenUI">Source</a></p> https://tedneward.github.io/Research/presentation/jujutsuui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/jujutsuui/index.html <p><a href="https://idursun.github.io/jjui/">Website</a> | <a href="https://github.com/idursun/jjui">Source</a></p> https://tedneward.github.io/Research/presentation/less/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/less/index.html <p><a href="https://lesscss.org/">Website</a> | <a href="https://github.com/less/">Source</a></p> <p>Less (which stands for Leaner Style Sheets) is a backwards-compatible language extension for CSS. This is the official documentation for Less, the language and Less.js, the JavaScript tool that converts your Less styles to CSS styles.</p> <p>Because Less looks just like CSS, learning it is a breeze. Less only makes a few convenient additions to the CSS language, which is one of the reasons it can be learned so quickly.</p> https://tedneward.github.io/Research/presentation/lit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/lit/index.html <p><a href="https://lit.dev/">Website</a> | <a href="https://github.com/lit/lit/">Source</a></p> https://tedneward.github.io/Research/presentation/ltk/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ltk/index.html <p><a href="https://pyscript.github.io/ltk/?runtime=py">Website</a> (running Pyodide) | <a href="pyscript.github.io/ltk/?runtime=mpy">Website</a> (running MicroPython) | <a href="https://github.com/pyscript/ltk">Source</a> | <a href="https://pyscript.github.io/ltk/">LTK kitchensink</a></p> <h2>Installing LTK</h2> <p>Install LTK from pypi:</p> <pre><code>python3 -m pip install pyscript-ltk </code></pre> <h2>Hello World</h2> <pre><code class="language-python">import ltk ltk.Text("Hello World").appendTo(ltk.body) </code></pre> <h2>Getting Started</h2> <p>To get started with LTK, we recommend you try it out on pyscript.com:<br> - <a href="https://pyscript.com/@laffra/ltk-on-micropython/latest">Minimal LTK with MicroPython</a><br> - <a href="https://pyscript.com/@laffra/ltk-on-pyodide/latest">Minimal LTK with PyOdide</a></p> <h2>Widget Specification</h2> <p>New widget types are created by subclassing <code>ltk.Widget</code>:</p> <pre><code class="language-python">class HBox(Widget): classes = [ "ltk-hbox" ] </code></pre> <p>By default, widgets are created as <code>div</code> DOM elements. You can choose a different tag:</p> <pre><code class="language-python">class Preformatted(Widget): classes = [ "ltk-pre" ] tag = "pre" </code></pre> <h2>Creating a UI</h2> <p>To create a UI, elements are constructed declaratively:</p> <pre><code class="language-python">ltk.Table( ltk.TableRow( ltk.TableHeader("header1") ltk.TableHeader("header2") ), [ ltk.TableRow( ltk.TableData(value1), ltk.TableData(value2), ) for value1, value2 in data ], ) </code></pre> <p>Widgets are added to others by using <code>jQuery</code>'s <code>append</code> and <code>appendTo</code> calls:</p> <pre><code class="language-python">ltk.body.append( ltk.Table(...).element ) container = ltk.VBox(...) ltk.H1("This is a header").appendTo(container) </code></pre> <p>When an LTK widget is created, a corresponding jQuery element is attached to it in<br> the <code>ltk.Widget.__init__</code> constructor. It uses the <code>tag</code> value defined by the<br> declaring class and the constructed element is referred to as <code>element</code>.<br> As the <code>append</code> call is a JavaScript function, implemented by jQuery, we do not<br> pass the LTK widget directly, but pass its <code>element</code> to append to the DOM.</p> <h2>Styling</h2> <p>Widgets can be styled using using three different approaches:</p> <ol> <li>Styling with element styles using <code>jQuery</code>'s <code>css</code> function:</li> </ol> <pre><code class="language-python">ltk.Text("Error: Flux capacitor low!") .css("background-color", "red") .css("color", "white") .css("padding", 8) </code></pre> <ol> <li>Styling using a <code>dict</code> to make it easier to share styles:</li> </ol> <pre><code class="language-python">error = { "background-color": "red", "color": "white", "padding": 8, } ltk.Text("Error: Flux capacitor low!", error) </code></pre> <ol> <li>Styling using CSS classes and an external stylesheet, using <code>jQuery</code>'s <code>addClass</code> function:</li> </ol> <pre><code class="language-python">ltk.Text("Some text").addClass("my-special-text) </code></pre> <p>The external style sheet should have these rules:</p> <pre><code class="language-css">.ltk-text { font-family: Arial; } .my-special-text { font-family: Courier; background-color: red; color: white; padding: 8px; } </code></pre> <p>External stylesheets can be included in the original <code>index.html</code> or injected at runtime from Python using:</p> <pre><code class="language-python">ltk.inject_style("https://example.org/awesome_styles.css") </code></pre> <h2>Events</h2> <p>Event handlers are attached using <code>jQuery</code> mechanisms.</p> <pre><code class="language-python">def buy(event): purchase(...) ltk.Card("Buy Now").on("click", ltk.proxy(buy)) </code></pre> <p>You can also use the more declarative decorator:</p> <pre><code class="language-python">@ltk.callback def buy(event): purchase(...) ltk.Card("Buy Now").on("click", buy) </code></pre> https://tedneward.github.io/Research/presentation/markojs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/markojs/index.html <p><a href="https://markojs.com/">Website</a> | <a href="https://github.com/marko-js/marko">Source</a></p> https://tedneward.github.io/Research/presentation/mdwiki/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/mdwiki/index.html <p><a href="https://dynalon.github.io/mdwiki/#!index.md">Website</a> | <a href="https://github.com/Dynalon/mdwiki">Source</a></p> <p>Not really a "wiki", per se (doesn't automatically create links to other pages, doesn't create a new page on request, etc). More like a JS browser-hosted Markdown renderer.</p> <p>Does support client-side "gimmicks" to do various Bootstrappy things, a la alerts, specially-rendered links, and so on.</p> <h2>Configuration</h2> <p>You can create a <code>config.json</code> file in the same folder as the <code>mdwiki.html</code> file which is then used for configuration. The file has to be valid JSON. Currently these options are available:</p> <ul> <li><code>"useSideMenu": false</code> - disable the side navigation</li> <li><code>"lineBreaks": "original"</code> - Instead of using the <a href="https://help.github.com/articles/github-flavored-markdown">GFM</a> line breaks, use the original line breaking as <a href="http://daringfireball.net/projects/markdown/">introduced by John Gruber</a>: <ul> <li>line breaks in the markdown files are ignored, except if a line ends with two spaces</li> <li>Default is <code>lineBreaks: "gfm"</code> (line breaks in markdown will create a new paragraph)</li> </ul> </li> <li><code>"additionalFooterText": ""</code> <ul> <li>Can be used to add text to the copyright footer at the bottom, like custom copyright notices.</li> <li>Example: <code>additionalFooterText: "All content and images &amp;copy; by John Doe"</code></li> <li>Default: <code>""</code> (empty string)</li> </ul> </li> <li><code>"anchorCharacter": "&amp;para;"</code> <ul> <li>The character/text displayed and used as a hyperlink when hovering over headings.</li> <li>Unicode characters can be used in HTML notation. Example: <code>&amp;#x2693;</code> will render as ⚓</li> <li>Default: The pilcrow (paragraph) sign: ¶</li> </ul> </li> <li><code>"title": "ACME Industries Wiki"</code> <ul> <li>Changes the title of the webpage.</li> <li>Default: <code>"MDWiki"</code></li> <li>Note - this parameter does not change the title of your wiki in Google search results. Edit <code>mdwiki.html</code> directly to correct this. (<a href="https://github.com/Dynalon/mdwiki/issues/58">Issue #58</a>)</li> </ul> </li> </ul> <p>A sample <code>config.json</code> might thus look like this:</p> <pre><code class="language-javascript">{ "useSideMenu": true, "lineBreaks": "gfm", "additionalFooterText": "All content and images &amp;copy; by John Doe", "anchorCharacter": "#" } </code></pre> <p>Note: More configuration options will be available in future versions of MDwiki.</p> <p>Hint: It is adviced that you create an empty config.json in each cases, to avoid 404 errors which will not get cached by your browser. Having an <code>config.json</code> file present thus will speed up page loading (even if its empty).</p> <h2>Example</h2> <p>Website-as-Markdown-source: <a href="https://github.com/Dynalon/mdwiki/tree/gh-pages">https://github.com/Dynalon/mdwiki/tree/gh-pages</a></p> <h4>Gimmicks: <a href="https://dynalon.github.io/mdwiki/#!./gimmicks.md">https://dynalon.github.io/mdwiki/#!./gimmicks.md</a></h4> <p>Gimmicks are little helper that bring plenty of dynamic features into your page. For example, you can use them to inline youtube videos, image slideshows or facebook like buttons.</p> <p>To use Gimmicks, all you have to do is include some specially crafted link into your markdown file. For example, if you want to embed a Youtube video (instead of linking to it), you just have to insert a link to the video:</p> <pre><code>[](http://www.youtube.com/watch?v=RMINSD7MmT4) </code></pre> <p>Gimmicks are realized via Javascript and work out of the box. Some gimmicks can be called with parameters on them, to customize their behaviour:</p> <pre><code>[gimmick:ForkMeOnGitHub ({ color: 'red', position: 'left' })](http://www.github.com/Dynalon/mdwiki) </code></pre> <p>The arguments are passed as a Javascript object. But for convienience reasons, you can omit the curly brackets <code>{ }</code>. The gimmick name after <code>gimmick:</code> is also case-insensitive:</p> <pre><code>[gimmick:forkmeongithub(color: 'red', position: 'left')](http://www.github.com/Dynalon/mdwiki) </code></pre> <p>Gimmicks are designed to always chose sane default values when no parameters are given, therefore <em>most</em> gimmicks do not require any parameters to work.</p> <p>Note: Gimmicks will usually load code or stylesheets from the internet, therefore they won't work in offline mode</p> <hr> <h5>Alerts</h5> <p>Alerts are automatically placed whenever you start a paragraph with a special <em>trigger</em> word, that <em>has</em> to be followed by a colon <code>:</code> or exclamation mark <code>!</code>.</p> <p>Trigger words are case insensitive, and must be one of the following:</p> <table> <thead> <tr> <th> Type </th> <th> Trigger </th> </tr> </thead> <tbody> <tr> <td> Warning </td> <td> warning, achtung, attention, warnung, atención, guarda, advertimiento, attenzione </td> </tr> <tr> <td> Note </td> <td> note, beachte, nota </td> </tr> <tr> <td> Hint </td> <td> hint, tip, tipp, hinweis, suggerimento </td> </tr> </tbody> </table> <p>Preview:</p> <p>Attention: This text is important.</p> <p>Note! This is a note.</p> <p>Hint: This is a hint.</p> <hr> <h5>GitHub Gists</h5> <p>Gists on github can be embedded by passing their numeric id:</p> <pre><code>[gimmick:gist](5641564) </code></pre> <p>Preview:</p> <p><a href="5641564">gimmick:gist</a></p> <hr> <h5>UML Diagrams via yUML.me</h5> <p>Embeds diagrams from the excellent <a href="http://yuml.me">yUML.me</a> service (see their website for documentation).</p> <p>Examples:</p> <p>[gimmick:yuml]( [HttpContext]uses -.-&gt;[Response] )</p> <pre><code>[gimmick:yuml]( [HttpContext]uses -.-&gt;[Response] ) </code></pre> <hr> <p><a href="[User|+Forename;+Surname;+HashedPassword;-Salt]">gimmick:yuml (type: 'class')</a></p> <pre><code>[gimmick:yuml]([User|+Forename+;Surname;+HashedPassword;-Salt|+Login();+Logout()]) </code></pre> <hr> <p>[gimmick:yuml (type: 'activity', style: 'plain') ]( <code>Make Coffee´-&gt;</code>want more coffee´ )</p> <pre><code>[gimmick:yuml (type: 'activity', style: 'plain') ]( `Make Coffee´-&gt;`want more coffee´ ) </code></pre> <hr> <p>[gimmick:yuml (type: 'usecase', scale: 150) ]( [Customer]-<code>Sign In´, [Customer]-</code>Buy Products´ )</p> <pre><code>[gimmick:yuml (diag: 'usecase', scale: 150) ]( [Customer]-`Sign In´, [Customer]-`Buy Products´ ) </code></pre> <p>Arguments</p> <ul> <li><strong>direction</strong></li> <li>is one of [ 'TB', 'LR' ]</li> <li>direction of the diagram: top-to-bottom or left-to-right</li> <li><strong>scale</strong></li> <li>is an integer percentage value, i.e. 150 or 200</li> <li>defines the scaling applied to the diagram in percent, 100% = no scaling</li> <li><strong>type</strong></li> <li>is one of [ 'class', 'activity', 'usecase' ]</li> <li>type of the UML diagram</li> <li><strong>style</strong></li> <li>is one of [ 'plain', 'scruffy' ]</li> <li>defines the applied theme, <em>plain</em> for clean and <em>scruffy</em> for comic-style look</li> </ul> <h5>Math</h5> <p><a href="">gimmick: math</a><br> Math formulas are realized through the <a href="http://www.mathjax.org">MathJax</a> library. To enable math formulas on a page, the <code>math</code> gimmick must be loaded by adding this link anywhere in the file:</p> <pre><code>[gimmick: math]() </code></pre> <p>To enable math for all sites, put the above link into the <code>navigation.md</code> file. Putting this link onto the site will load MathJax dynamically from a CDN provider.</p> <p>Note: The MathJax script is very large and loads some more dependencies like fonts. Using the math gimmick might result in slow page loads.</p> <h6>Math inserted to new paragraph</h6> <p>You can add math formulas by putting them between to <code>$$</code> signs and use LateX syntax:</p> <pre><code>$$ x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} $$ </code></pre> <p>$$ x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a} $$</p> <h6># Inline equations</h6> <p>Inline equations can be inserted by surrounding them with the delimiters <code>\\(</code> and <code>\\)</code>:</p> <pre><code>Diameter \\( d \\) of a circle given area \\( A \\) can by obtained via \\(d=\sqrt{\frac{4A}{\pi}}\\) </code></pre> <p>Will show as: Diameter \( d \) of a circle given area \( A \) can by obtained via \(d=\sqrt{\frac{4A}{\pi}}\)</p> <h6># Some examples</h6> <p>$$ \frac{\partial \phi}{\partial x} \vert_b = \frac{1}{\Delta x/2}(\phi_0-\phi_b) $$</p> <p>$$<br> \int u \frac{dv}{dx},dx=uv-\int<br> \frac{du}{dx}v,dx\lim_{n\rightarrow \infty }<br> \left ( 1 +\frac{1}{n} \right )^n<br> $$</p> <h5>Youtube</h5> <p>Whenever you insert a regular link with an empty caption that points to a video on <code>youtube.com</code> or <code>youtu.be</code>, the link is automatically turned into an embedded iframe, which will display a preview thumbnail of the video on your website.</p> <p>Example:</p> <pre><code>This will show the video preview on your website: [](https://www.youtube.com/watch?v=aqz-KE-bpKQ) </code></pre> <p><a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ"></a></p> <p>To omit the preview and just get a regular link, add a caption:</p> <pre><code>[Click to see an awesome video](https://www.youtube.com/watch?v=aqz-KE-bpKQ) </code></pre> <p><a href="https://www.youtube.com/watch?v=aqz-KE-bpKQ">Click to see an awesome video</a></p> <h5>Chart</h5> <p>Adds a chart to the screen using data from your Markdown table.</p> <h6>Options</h6> <ul> <li>labelColumn: This is a string that indicates which column will be used to label the data points. This String must be a direct match to your table header fro the column.</li> <li>dataColumns: This is an array of strings that indicated the column to be plotted. This String must be a direct match to your table header fro the column.</li> <li>canvasId: This is an ID for the generated chart. Defaults to a random number between 1-1000.</li> <li>chartOptions: This is an object that is passed to chartjs to configure its options. Refer to chartjs for documentation on available options.</li> <li>chartType: This string is the type of chart render. Bar, Line, or Radar. Defaults to Line.</li> </ul> <p>Note: Currently only support a single table on a page. You CAN have multiple charts from the same table.</p> <pre><code>| # | Sprint | Points | Sum | Avg | Note | | - | -------- |------- | --- | ---- | ---- | | 1 | Sprint 1 | 6 | 6 | 6.0 | | | 2 | Sprint 2 | 6 | 12 | 6.0 | | | 3 | Sprint 3 | 15 | 27 | 9.0 | | | 4 | Sprint 4 | 9 | 36 | 9.0 | | | 5 | Sprint 5 | 6 | 42 | 8.4 | | | 6 | Sprint 6 | 9 | 51 | 8.5 | | [gimmick:chart ({dataColumns: ['Avg'], labelColumn: "Sprint", chartType: 'Line', width: '660px', height: '300px'})]() [gimmick:chart ({dataColumns: ['Avg'], labelColumn: "Sprint", chartType: 'Bar', width: '660px', height: '300px'})]() </code></pre> <p>Example:</p> <table> <thead> <tr> <th> # </th> <th> Sprint </th> <th> Points </th> <th> Sum </th> <th> Avg </th> <th> Note </th> </tr> </thead> <tbody> <tr> <td> 1 </td> <td> Sprint 1 </td> <td> 6 </td> <td> 6 </td> <td> 6.0 </td> <td> </td> </tr> <tr> <td> 2 </td> <td> Sprint 2 </td> <td> 6 </td> <td> 12 </td> <td> 6.0 </td> <td> </td> </tr> <tr> <td> 3 </td> <td> Sprint 3 </td> <td> 15 </td> <td> 27 </td> <td> 9.0 </td> <td> </td> </tr> <tr> <td> 4 </td> <td> Sprint 4 </td> <td> 9 </td> <td> 36 </td> <td> 9.0 </td> <td> </td> </tr> <tr> <td> 5 </td> <td> Sprint 5 </td> <td> 6 </td> <td> 42 </td> <td> 8.4 </td> <td> </td> </tr> <tr> <td> 6 </td> <td> Sprint 6 </td> <td> 9 </td> <td> 51 </td> <td> 8.5 </td> <td> </td> </tr> </tbody> </table> <p><a href="">gimmick:chart ({dataColumns: ['Avg'], labelColumn: "Sprint", chartType: 'Line', width: '660px', height: '300px'})</a></p> <p><a href="">gimmick:chart ({dataColumns: ['Avg'], labelColumn: "Sprint", chartType: 'Bar', width: '660px', height: '300px'})</a></p> https://tedneward.github.io/Research/presentation/mithril/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/mithril/index.html <p><a href="https://mithril.js.org/index.html">Website</a> | <a href="https://github.com/MithrilJS/mithril.js">Source</a></p> https://tedneward.github.io/Research/presentation/nakedobjects/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nakedobjects/index.html <p><a href="https://github.com/NakedObjectsGroup/NakedObjectsFramework">Source</a> | <a href="https://github.com/NakedObjectsGroup/NakedObjectsFramework/blob/master/Template_NOF10/Template_NOF10.zip?raw=true">NOF 10 Starter Template</a> | <a href="https://github.com/NakedObjectsGroup/NakedObjectsFramework/blob/master/Documentation/NOF10DeveloperManual.docx?raw=true">Developer's Manual</a></p> <p>See also: <a href="/distribution/restfulobjects.html">Restful Objects</a></p> https://tedneward.github.io/Research/presentation/nodify/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nodify/index.html <p><a href="https://miroiu.github.io/nodify/">Website</a> | <a href="https://github.com/miroiu/nodify">Source</a> | <a href="https://github.com/miroiu/nodify/wiki">Docs</a></p> <p>Avalonia port: <a href="https://github.com/BAndysc/nodify-avalonia">https://github.com/BAndysc/nodify-avalonia</a></p> https://tedneward.github.io/Research/presentation/nxui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/nxui/index.html <p><a href="http://www.nxui.net/">Website</a> | <a href="https://github.com/wieslawsoltes/NXUI">Source</a></p> <p>Usage:</p> <pre><code>Window Build() =&gt; Window().Content(Label().Content("NXUI")); AppBuilder.Configure&lt;Application&gt;() .UsePlatformDetect() .UseFluentTheme() .StartWithClassicDesktopLifetime(Build, args); var count = 0; </code></pre> <pre><code>Window Build() =&gt; Window(out var window) .Title("NXUI").Width(400).Height(300) .Content( StackPanel() .Children( Button(out var button) .Content("Welcome to Avalonia, please click me!"), TextBox(out var tb1) .Text("NXUI"), TextBox() .Text(window.BindTitle()), Label() .Content(button.ObserveOnClick().Select(_ =&gt; ++count).Select(x =&gt; $"You clicked {x} times.")))) .Title(tb1.ObserveText().Select(x =&gt; x?.ToUpper())); AppBuilder.Configure&lt;Application&gt;() .UsePlatformDetect() .UseFluentTheme() .WithApplicationName("NXUI") .StartWithClassicDesktopLifetime(Build, args); </code></pre> https://tedneward.github.io/Research/presentation/console/lanterna/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/lanterna/index.html <p><a href="https://github.com/mabe02/lanterna">Source</a> | <a href="https://github.com/mabe02/lanterna/blob/master/docs/contents.md">Documentation</a></p> <p>Lanterna is supporting xterm compatible terminals and terminal emulators such as konsole, gnome-terminal, putty, xterm and many more. One of the main benefits of lanterna is that it's not dependent on any native library but runs 100% in pure Java.</p> <p>Also, when running Lanterna on computers with a graphical environment (such as Windows or Xorg), a bundled terminal emulator written in Swing will be used rather than standard output. This way, you can develop as usual from your IDE (most of them doesn't support ANSI control characters in their output window) and then deploy to your headless server without changing any code.</p> <p>Lanterna is structured into three layers, each built on top of the other and you can easily choose which one fits your needs best.</p> <ol> <li> <p>The first is a low level terminal interface which gives you the most basic control of the terminal text area. You can move around the cursor and enable special modifiers for characters put to the screen. You will find these classes in package com.googlecode.lanterna.terminal.</p> </li> <li> <p>The second level is a full screen buffer, the whole text screen in memory and allowing you to write to this before flushing the changes to the actual terminal. This makes writing to the terminal screen similar to modifying a bitmap. You will find these classes in package com.googlecode.lanterna.screen.</p> </li> <li> <p>The third level is a full GUI toolkit with windows, buttons, labels and some other components. It's using a very simple window management system (basically all windows are modal) that is quick and easy to use. You will find these classes in package com.googlecode.lanterna.gui2.</p> </li> </ol> https://tedneward.github.io/Research/presentation/console/textual/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/textual/index.html <h2>Articles</h2> <ul> <li><a href="https://realpython.com/python-textual/">"Python Textual: Build Beautiful UIs in the Terminal"</a></li> </ul> https://tedneward.github.io/Research/presentation/corecalc/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/corecalc/index.html <p><a href="https://www.itu.dk/people/sestoft/corecalc/">Website</a></p> <p>Corecalc is an implementation of core spreadsheet functionality in C#, intended as a platform for experiments with technology and novel functionality. Corecalc is a research prototype, not a usable replacement for Microsoft Excel, Gnumeric or LibreOffice Calc.</p> <p><a href="https://www.itu.dk/people/sestoft/corecalc/corecalc-0.7.1.zip">Corecalc (old) Source</a> | <a href="https://www.itu.dk/people/sestoft/corecalc/ITU-TR-2006-91.pdf">Report (PDF)</a></p> <p>Funcalc is an extension of Corecalc in which users can define their own functions via sheet-defined functions, without resorting to external languages such as VBA. These are compiled to .NET bytecode at run-time, thus offering high performance while preserving the usual mode of interaction in which all edits take effect immediately. Funcalc is a research prototype.</p> <p><a href="https://www.itu.dk/people/sestoft/corecalc/funcalc-20140927.zip">Funcalc Source</a> <a href="https://www.itu.dk/people/sestoft/corecalc/funcalc-src-20140927.pdf">A4 PDF pretty-printed</a></p> https://tedneward.github.io/Research/presentation/css/picocss/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/css/picocss/index.html <p><a href="https://picocss.com/">Website</a></p> <h2>Articles</h2> <ul> <li><a href="https://blog.logrocket.com/getting-started-pico-css/">"Getting Started with Pico CSS"</a></li> </ul> https://tedneward.github.io/Research/presentation/dearimgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/dearimgui/index.html <p><a href="https://github.com/ocornut/imgui">Source</a></p> <p>Designed for game platforms, but I don't see any reason why it couldn't work for other things too.</p> <ul> <li>.NET: <a href="https://github.com/ImGuiNET/ImGui.NET">ImGuiNET</a>: An ImGui wrapper for .NET. Included is a basic sample program that shows how to use the library, and renders the UI using Veldrid, a portable graphics library for .NET. By itself, Dear ImGui does not care what technology you use for rendering; it simply outputs textured triangles. Example renderers also exist for MonoGame and OpenTK (OpenGL).</li> </ul> <p>Pros: battle tested / production ready, available on conan, minimal dependencies</p> https://tedneward.github.io/Research/presentation/discourse/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/discourse/index.html <p><a href="https://www.discourse.org/">Website</a> <a href="https://github.com/discourse/discourse">Source</a></p> https://tedneward.github.io/Research/presentation/dotvvm/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/dotvvm/index.html <p><a href="https://www.dotvvm.com/">Website</a></p> <p>Commercial options (more components available).</p> https://tedneward.github.io/Research/presentation/eleventy/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/eleventy/index.html <p><a href="https://www.11ty.dev/">Website</a> | <a href="https://github.com/11ty/eleventy/">Source</a></p> https://tedneward.github.io/Research/presentation/epoxy/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/epoxy/index.html <p><a href="https://github.com/airbnb/epoxy">Source</a> | <a href="https://medium.com/airbnb-engineering/epoxy-airbnbs-view-architecture-on-android-c3e1af150394#.9hno4vdc0">Blog post</a></p> https://tedneward.github.io/Research/presentation/flakes/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/flakes/index.html <p><a href="https://getflakes.com/">Website</a> | <a href="https://github.com/kumailht/flakes/">Source</a></p> <h3>From the Website</h3> <p><strong>Quick Start Guide</strong></p> <ol> <li>Download the Flakes framework on Github.</li> <li>Install Dependencies using the "bower install" command. If you don't have the Bower Package Manager, install it using the instructions on their official site.</li> <li>Example.html has some boilerplate code that you can use to get started.</li> </ol> <p><strong>When should I use Flakes?</strong></p> <p>Flakes is intended to be used for business facing web applications. If you're building an internal tool to help your sales team manage leads or are building an inventory tracking solution for your watch business, you'll need Flakes.</p> <p>Flakes is a starting point, It helps you get a quick start. It is not a collection of everything you'll ever need and it doesn't aim to become one.</p> <p><strong>What sort of layouts can I build with Flakes?</strong></p> <p>Flakes is a collection of components. These components follow the Flakes aesthetic and go well together. Since these components are low level abstractions like forms and grids, you really can build almost anything you'll need.</p> <p>I've built a few example layouts for you play around with. Check out the example layouts using the navigation to your left to get a feel for what is possible.</p> <p><strong>How do I contribute to Flakes?</strong></p> <p>Until now, Flakes has been a one man show with me designing and developing for it. When you work on something for long enough, you'll run out of ideas on how to improve it. If you are keen on contributing I'd love to have you make a pull request with your change. If it's a sufficiently big change, I'd recommend filing an issue on Github to discuss the change to avoid duplicate or unnecessary work.</p> https://tedneward.github.io/Research/presentation/formio/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/formio/index.html <p><a href="https://form.io/">Website</a> | Commercial?</p> https://tedneward.github.io/Research/presentation/frappe/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/frappe/index.html <p><a href="https://frappeframework.com/">Website</a> | <a href="https://github.com/frappe/frappe">Source</a></p> <p>From the website:</p> <h2>Batteries Included</h2> <p>Frappe ships with essential modules that are required to build any serious business application</p> <h3>Metadata First</h3> <p>Everything in Frappe is a DocType. DocTypes can be defined easily without code used everywhere.</p> <h3>Admin User Interface</h3> <p>Frappe comes with a rich single page application (SPA) with built in forms, list, search and navigation.</p> <h3>Roles and Permissions</h3> <p>You define user roles and permissions that are applied out of the box on all interactions.</p> <h3>Extensible</h3> <p>With a modular architecture, you can create your own apps that can be extended by other apps.</p> <h3>REST API + Webhooks</h3> <p>Frappe is integration friendly and comes with REST API and Webhooks on all models based on authentication.</p> <h3>Job Scheduler</h3> <p>You can configure background workers and run periodic tasks powered by Python RQ.</p> <h3>Socket.io</h3> <p>Frappe comes with first class socket.io support based on NodeJS and using Redis pub-sub.</p> <h3>Email Setup</h3> <p>Send, receive, view and manage emails easily using SMTP and IMAP based email accounts.</p> <h3>Multi-tenant</h3> <p>Database driven multi-tenant architecture easily lets you host multiple sites on a single server.</p> https://tedneward.github.io/Research/presentation/graphics/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/graphics/index.html <h3>Graphics Programming Books</h3> <ul> <li><a href="https://github.com/lettier/3d-game-shaders-for-beginners">3D Game Shaders For Beginners</a> - David Lettier (Git) <a href="https://lettier.github.io/3d-game-shaders-for-beginners">(HTML)</a></li> <li><a href="https://en.wikibooks.org/wiki/Blender_3D%3A_Noob_to_Pro">Blender 3D: Noob to Pro</a> - Wikibooks</li> <li><a href="https://docs.blender.org/manual/en/latest">Blender Manual</a></li> <li><a href="http://gabrielgambetta.com/computer-graphics-from-scratch">Computer Graphics from scratch</a> - Gabriel Gambetta (:construction: <em>in process</em>)</li> <li><a href="http://user.xmission.com/~legalize/book/download/index.html">DirectX manual</a> (draft)</li> <li><a href="https://developer.nvidia.com/gpugems/GPUGems/gpugems_pref01.html">GPU Gems</a></li> <li><a href="http://www.gamedev.net/page/resources/_/technical/graphics-programming-and-theory/graphics-programming-black-book-r1698">Graphics Programming Black Book</a> - Michael Abrash</li> <li><a href="https://open.gl">Introduction to Modern OpenGL</a> - Alexander Overvoorde (HTML, EPUB, PDF) (C++)</li> <li><a href="https://leanpub.com/introductiontotouchdesigner/">Introduction to TouchDesigner 099</a> <em>(Leanpub account or valid email requested)</em></li> <li><a href="https://www.scratchapixel.com">Learn Computer Graphics From Scratch!</a> - Scratchapixel (:construction: <em>in process</em>)</li> <li><a href="http://learnopengl.com">Learn OpenGL</a> - Joey de Vries</li> <li><a href="https://github.com/bwasty/learn-opengl-rs">Learn OpenGL RS</a> - Benjamin Wasty et al. (:construction: <em>in process</em>) (Rust)</li> <li><a href="https://web.archive.org/web/20150225192611/http://www.arcsynthesis.org/gltut/index.html">Learning Modern 3D Graphics Programming</a> - Jason L. McKesson (draft)</li> <li><a href="https://www.cs.csustan.edu/~rsc/CS3600F00/Notes.pdf">Notes for a Computer Graphics Programming Course</a> - Dr. Steve Cunningham (PDF)</li> <li><a href="http://www.songho.ca/opengl/index.html">OpenGL</a> - Concepts and illustrations.</li> <li><a href="https://raytracing.github.io">Ray Tracing in One Weekend</a> - Peter Shirley (HTML)</li> <li><a href="http://www.realtimerendering.com/resources/shaderx/">ShaderX series</a> - Wolfgang Engel</li> <li><a href="http://www.opengl-tutorial.org">Tutorials for modern OpenGL</a></li> <li><a href="http://webglinsights.com">WebGL Insights</a> - Patrick Cozzi and Contributors</li> </ul> https://tedneward.github.io/Research/presentation/hbuilder/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/hbuilder/index.html <p><a href="https://www.dcloud.io/index.html">Website</a> and <a href="https://www.dcloud.io/hbuilderx.html">here</a></p> <p>Not much English docs, but seems reasonable to use. <a href="../vue">Vue</a>-based</p> https://tedneward.github.io/Research/presentation/html/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/html/index/index.html <p><a href="https://html.spec.whatwg.org/multipage/">Standard</a> (<a href="https://html.spec.whatwg.org/print.pdf">PDF</a>)</p> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="http://learn.shayhowe.com/html-css/">A beginner's guide to HTML&amp;CSS</a></li> <li><a href="http://marksheet.io">A free guide to learn HTML and CSS</a></li> <li><a href="http://adaptivewebdesign.info/1st-edition/">Adaptive Web Design</a> - Aaron Gustafson</li> <li><a href="http://learn.shayhowe.com/advanced-html-css/">An advanced guide to HTML&amp;CSS</a></li> <li><a href="https://atomicdesign.bradfrost.com">Atomic Design</a> - Brad Frost</li> <li><a href="https://web.archive.org/web/20160505010319/http://learnjs.io/canvassing/read/">Canvassing</a></li> <li><a href="http://mdo.github.io/code-guide/">Code Guide: Standards for developing flexible, durable, and sustainable HTML and CSS</a> - Mark Otto</li> <li><a href="http://diveinto.html5doctor.com">Dive Into HTML5</a> - Mark Pilgrim (<a href="http://mislav.net/2011/10/dive-into-html5/">PDF</a>)</li> <li><a href="http://domenlightenment.com">DOM Enlightenment</a> - Cody Lindley (HTML)</li> <li><a href="https://dash.generalassemb.ly">GA Dash</a></li> <li><a href="https://google.github.io/styleguide/htmlcssguide.html">Google's HTML/CSS Style Guide</a></li> <li><a href="https://www.digitalocean.com/community/books/how-to-build-a-website-with-html-ebook">How To Build a Website with HTML</a> - Erin Glass (PDF, EPUB)</li> <li><a href="https://web.archive.org/web/20180816174417/http://howtocodeinhtml.com/HowToCodeInHTML5AndCSS3.pdf">How to Code in HTML5 and CSS3</a> - Damian Wielgosik (PDF)</li> <li><a href="http://joshondesign.com/p/books/canvasdeepdive/toc.html">HTML Canvas Deep Dive</a> - Josh Marinacci</li> <li><a href="http://www.htmldog.com">HTML Dog Tutorials</a></li> <li><a href="https://www.oreilly.com/library/view/html5-canvas/9781449308032/ch01.html">HTML5 Canvas</a> - Steve Fulton &amp; Jeff Fulton</li> <li><a href="https://goalkicker.com/HTML5CanvasBook/">HTML5 Canvas Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://www.oreilly.com/library/view/html5-for-publishers/9781449320065/pr02.html">HTML5 for Publishers</a> - Sanders Kleinfeld</li> <li><a href="http://html5forwebdesigners.com">HTML5 For Web Designers</a> - Jeremy Keith</li> <li><a href="https://goalkicker.com/HTML5Book/">HTML5 Notes for Professionals</a> - Compiled from StackOverflow documentation (PDF)</li> <li><a href="https://www.ossblog.org/wp-content/uploads/2017/06/html5-quick-learning-quide.pdf">HTML5 Quick Learning Guide</a> - HTML5Templates (PDF)</li> <li><a href="https://leanpub.com/html5shootemupinanafternoon/read">HTML5 Shoot 'em Up in an Afternoon</a> - Bryan Bibat (HTML)</li> <li><a href="https://www.internetingishard.com">Interneting is Hard (But it Doesn't Have to Be)</a> - Oliver James</li> <li><a href="https://learn.shayhowe.com">Learn to Code HTML &amp; CSS</a> - Shay Howe</li> <li><a href="https://svgpocketguide.com">Pocket Guide to Writing SVG</a> - Joni Trythall</li> <li><a href="https://web.archive.org/web/20181215200026/http://apress.jensimmons.com/v5/pro-html5-programming/ch0.html">Pro HTML5 Programming</a> - Jen Simmons, Chris O'Connor, Dylan Wooters, Peter Lubbers</li> <li><a href="https://resilientwebdesign.com/#Resilientweb%20design">Resilient Web Design</a> - Jeremy Keith</li> <li><a href="https://rtlstyling.com">RTL Styling 101</a> - Ahmad Shadeed</li> <li><a href="https://flaviocopes.com/page/html-handbook/">The HTML Handbook</a> - Flavio Copes (PDF, EPUB, Kindle) <em>(email address requested)</em></li> <li><a href="http://chimera.labs.oreilly.com/books/1234000001552">Web Audio API</a> - Boris Smus</li> <li><a href="https://leanpub.com/web-visual-effects-with-css3/read">Web Visual Effects with CSS3</a> - Thomas Mak</li> </ul> <h3>Articles</h3> <ul> <li><a href="https://dev.to/maxprilutskiy/html5-elements-you-didnt-know-you-need-gan">"HTML5 Elements You Didn't Know You Need"</a></li> <li><a href="https://www.htmhell.dev/adventcalendar/2025/27/">Replacing JS with just HTML</a> (Accordions / Expanding Content Panels; Input with Autofilter Suggestions Dropdown; Modals / Popovers; Offscreen Nav / Content)</li> <li><a href="https://www.smashingmagazine.com/2025/12/masonry-things-you-wont-need-library-anymore/">Masonry: Things You Won’t Need A Library For Anymore</a>: <ul> <li><strong>Popovers And Dialogs</strong>: The <a href="https://developer.mozilla.org/docs/Web/API/Popover_API">Popover API</a>, the <a href="https://developer.mozilla.org/docs/Web/HTML/Reference/Elements/dialog"><code>&lt;dialog&gt;</code> HTML element</a>, and the <a href="https://developer.mozilla.org/docs/Web/CSS/Reference/Selectors/::backdrop"><code>::backdrop</code> pseudo-element</a> can help you get rid of dependencies on popup, tooltip, and dialog libraries. They handle accessibility and focus management for you, out of the box, are highly customizable by using CSS, and can easily be animated.</li> <li><strong>Accordions</strong>: The <a href="https://developer.mozilla.org/docs/Web/HTML/Reference/Elements/details"><code>&lt;details&gt;</code> element</a>, its <a href="https://developer.mozilla.org/docs/Web/HTML/Reference/Elements/details#name"><code>name</code> attribute</a> for mutually exclusive elements, and the <a href="https://developer.mozilla.org/docs/Web/CSS/Reference/Selectors/::details-content"><code>::details-content</code> pseudo-element</a> remove the need for accordion components.</li> <li><strong>CSS Syntax</strong>: Cascade layers, for a more organized CSS codebase, CSS nesting, for more compact CSS, new color functions, relative colors, and color-mix, new Maths functions like abs(), sign(), pow() and others help reduce dependencies on CSS pre-processors, utility libraries like Bootstrap and Tailwind, or even runtime CSS-in-JS libraries. The game changer :has(), one of the most requested features for a long time, removes the need for more complicated JS-based solutions.</li> <li><strong>JS Utilities</strong>: Modern Array methods like <code>findLast()</code>, or <code>at()</code>, as well as Set methods like <code>difference()</code>, <code>intersection()</code>, <code>union()</code> and others can reduce dependencies on libraries like Lodash.</li> <li><strong>Container Queries</strong>: <a href="https://developer.mozilla.org/docs/Web/CSS/CSS_containment/Container_queries">Container queries</a> make UI components respond to things other than the viewport size, and therefore make them more reusable across different contexts. No need to use a JS-heavy UI library for this anymore, and no need to use a polyfill either.</li> <li><strong>Layout</strong>: <a href="https://developer.mozilla.org/docs/Web/CSS/CSS_grid_layout">Grid</a>, <a href="https://developer.mozilla.org/docs/Web/CSS/CSS_grid_layout/Subgrid">subgrid</a>, <a href="https://developer.mozilla.org/docs/Learn_web_development/Core/CSS_layout/Flexbox">flexbox</a>, or <a href="https://developer.mozilla.org/docs/Web/CSS/Reference/Properties/columns">multi-column</a> have been around for a long time now, but looking at the results of the State of CSS surveys, it’s clear that developers tend to be very cautious with adopting new things, and wait for a very long time before they do.</li> </ul> <p>Things you might not need anymore in the near future:</p> <ul> <li><strong>Anchor Positioning</strong>: CSS anchor positioning handles the positioning of popovers and tooltips relative to other elements, and takes care of keeping them in view, even when moving, scrolling, or resizing the page. This is a great complement to the Popover API mentioned before, which will make it even easier to migrate away from more performance-intensive JS solutions.</li> <li><strong>Navigation API</strong>: The Navigation API can be used to handle navigation in single-page apps and might be a great complement, or even a replacement, to React Router, Next.js routing, or Angular routing tasks.</li> <li><strong>View Transitions API</strong>: The View Transitions API can animate between the different states of a page. On a single-page application, this makes smooth transitions between states very easy, and can help you get rid of animation libraries such as Anime.js, GSAP, or Motion.dev. Even better, the API can also be used with multiple-page applications.</li> <li><strong>Scroll-driven Animations</strong>: Scroll-driven animations run on the user’s scroll position, rather than over time, making them a great solution for storytelling and product tours.</li> <li><strong>Customizable Selects</strong>: A customizable select is a normal <code>&lt;select&gt;</code> element that lets you fully customize its appearance and content, while ensuring accessibility and performance benefits.</li> <li><strong>CSS Masonry</strong>: <a href="https://developer.chrome.com/blog/masonry-update">CSS Masonry</a> is another upcoming web platform feature that I want to spend more time on. With CSS Masonry, you can achieve layouts that are very hard, or even impossible, with flex, grid, or other built-in CSS layout primitives. Developers often resort to using third-party libraries to achieve Masonry layouts, such as the Masonry JS library. "Masonry is a type of layout that was made popular by Pinterest years ago. It creates independent tracks of content within which items pack themselves up as close to the start of the track as they can."</li> </ul> </li> </ul> https://tedneward.github.io/Research/presentation/ionic/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ionic/index.html <p><a href="https://ionicframework.com/">Website</a></p> <p>Used to build on top of <a href="../cordova">Cordova</a> but now uses its own runtime which is backwards-compatible.</p> https://tedneward.github.io/Research/presentation/jquery/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/jquery/index.html <h1>jQuery</h1> <p><a href="https://jquery.com/">Website</a> | <a href="https://github.com/jquery/jquery">Source</a></p> <p><a href="https://terminal.jcubic.pl/">jQuery Terminal</a>: creating command line interpreters in your Web applications.</p> <h1>jQuery UI</h1> <p>A curated set of user interface interactions, effects, widgets, and themes built on top of the jQuery JavaScript Library.</p> <p><a href="https://jqueryui.com/">Website</a> | <a href="https://github.com/jquery/jquery-ui">Source</a></p> <h1>jQuery Mobile -- No longer supported</h1> <p><a href="https://jquerymobile.com/">Website</a> | <a href="https://github.com/jquery/jquery-mobile">Source</a></p> <ul> <li><a href="http://jqfundamentals.com">jQuery Fundamentals</a> - Bocoup (HTML)</li> </ul> https://tedneward.github.io/Research/presentation/console/ftxui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/ftxui/index.html <p><a href="https://github.com/ArthurSonzogni/FTXUI">Source</a></p> <h4>Example:</h4> <p><img src="https://private-user-images.githubusercontent.com/4759106/243190749-569bf043-4e85-4245-aad5-2324572135c4.png" alt=""></p> <pre><code> vbox({ hbox({ text("one") | border, text("two") | border | flex, text("three") | border | flex, }), gauge(0.25) | color(Color::Red), gauge(0.50) | color(Color::White), gauge(0.75) | color(Color::Blue), }); </code></pre> https://tedneward.github.io/Research/presentation/console/patat/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/patat/index.html <p><a href="https://github.com/jaspervdj/patat">Source</a></p> https://tedneward.github.io/Research/presentation/console/turbovision/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/turbovision/index.html <p><a href="https://github.com/magiblot/tvision">tvision</a>: A modern port of Turbo Vision 2.0, the classical framework for text-based user interfaces. Now cross-platform and with Unicode support.</p> <p><a href="https://github.com/tomer/tvision/">TVision-themed CSS</a></p> https://tedneward.github.io/Research/presentation/createjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/createjs/index.html <p><a href="https://createjs.com/">Website</a> | <a href="https://github.com/CreateJS">Source</a></p> <p>Consists of:</p> <ul> <li><a href="https://createjs.com/easeljs">EaselJS</a>: A JavaScript library that makes working with the HTML5 Canvas element easy.</li> <li><a href="https://createjs.com/tweenjs">TweenJS</a>: A JavaScript library for tweening and animating HTML5 and JavaScript properties.</li> <li><a href="https://createjs.com/soundjs">SoundJS</a>: A JavaScript library that lets you easily and efficiently work with audio on the web.</li> <li><a href="https://createjs.com/preloadjs">PreloadJS</a>: A JavaScript library that lets you manage and co-ordinate the loading of assets and data.</li> </ul> <h2>Reading</h2> <p><a href="https://www.luxanimals.com/blog/article/combining_easel_box2d">"Combining Easel.js and Box2d in Canvas"</a></p> https://tedneward.github.io/Research/presentation/css/snippets/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/css/snippets/index.html <p><a href="https://codepen.io/jenanemone/pen/MWQbPeR">CSS for an "index card" effect</a>:</p> <pre><code class="language-html">&lt;div class="card-title"&gt; Milkshake &lt;/div&gt; &lt;div class="card"&gt; &lt;span class="card-text"&gt; &lt;ul&gt; &lt;li&gt;1c 2% milk&lt;/li&gt; &lt;li&gt;1/8c cream&lt;/li&gt; &lt;li&gt;1c ice cream&lt;/li&gt; &lt;/ul&gt; &lt;p&gt;Blend well, top with whipped cream or crushed candy, and serve cold.&lt;/p&gt; &lt;/span&gt; &lt;/div&gt; </code></pre> <pre><code class="language-css">/* Styling specific to an index card */ .card { background-color: white; background: repeating-linear-gradient(rgb(240,250,250), rgb(240,250,250) 25px, #9198e5 26px, #9198e5 27px); height: 300px; width: 500px; padding: 0; } .card-title { background-image: linear-gradient(rgb(240,250,250), rgb(240,250,250) 42px, pink 43px, pink 44px, rgb(240,250,250) 46px); height: 46px; width: 500px; padding-left: 25px; padding-top: 3px; color: black; line-height: 46px; font-size: 30px; font-weight: bold; } .card-text { display: flex; flex-direction: column; justify-content: flex-start; font-size: 18px; line-height: 27px; padding-top: 4px; } p { padding-left: 25px; } </code></pre> <hr> <p><a href="https://markodenic.com/css-tips/">CSS Tips</a>:</p> <ul> <li> <p>Typing Effect:<br> HTML:</p> <pre><code>&lt;div class="wrapper"&gt; &lt;div class="typing-demo"&gt; This is a typing demo. &lt;/div&gt; &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>.wrapper { height: 100vh; /*This part is important for centering*/ display: grid; place-items: center; } .typing-demo { width: 22ch; animation: typing 2s steps(22), blink .5s step-end infinite alternate; white-space: nowrap; overflow: hidden; border-right: 3px solid; font-family: monospace; font-size: 2em; } @keyframes typing { from { width: 0 } } @keyframes blink { 50% { border-color: transparent } } </code></pre> </li> <li> <p>Drop shadow: When you work with transparent images you can use <code>drop-shadow()</code> filter function to create a shadow on the image’s content, instead of <code>box-shadow</code> property which creates a rectangular shadow behind an element’s entire box: filter: <code>drop-shadow(2px 4px 8px #585858)</code><br> HTML</p> <pre><code>&lt;div class="wrapper"&gt; &lt;div class="mr-2"&gt; &lt;div class="mb-1 text-center"&gt; box-shadow &lt;/div&gt; &lt;img class="box-shadow" src="https://markodenic.com/man_working.png" alt="Image with box-shadow"&gt; &lt;/div&gt; &lt;div&gt; &lt;div class="mb-1 text-center"&gt; drop-shadow &lt;/div&gt; &lt;img class="drop-shadow" src="https://markodenic.com/man_working.png" alt="Image with drop-shadow"&gt; &lt;/div&gt; &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>.wrapper { height: 100vh; display: flex; align-items: center; justify-content: center; } .mr-2 { margin-right: 2em; } .mb-1 { margin-bottom: 1em; } .text-center { text-align: center; } .box-shadow { box-shadow: 2px 4px 8px #585858; } .drop-shadow { filter: drop-shadow(2px 4px 8px #585858); } </code></pre> </li> <li> <p>Smooth scrolling:<br> HTML:</p> <pre><code>&lt;nav&gt; Scroll to: &lt;a href="#sectionA" class="link bg-red"&gt;A&lt;/a&gt; &lt;a href="#sectionB" class="link bg-blue"&gt;B&lt;/a&gt; &lt;a href="#sectionC" class="link bg-green"&gt;C&lt;/a&gt; &lt;/nav&gt; &lt;div class="wrapper"&gt; &lt;div id="sectionA" class="section bg-red"&gt;A&lt;/div&gt; &lt;div id="sectionB" class="section bg-blue"&gt;B&lt;/div&gt; &lt;div id="sectionC" class="section bg-green"&gt;C&lt;/div&gt; &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>html { scroll-behavior: smooth; } nav { position: fixed; left: calc(50vw - 115px); top: 0; width: 200px; text-align: center; padding: 15px; background: #fff; box-shadow: 0 2px 5px 1px rgba(0, 0, 0, 0.2); } nav .link { padding: 5px; color: white; } .section { height: 100vh; display: flex; align-items: center; justify-content: center; color: #fff; font-size: 5em; text-shadow: 0px 2px 0px #b2a98f, 0px 4px 3px rgba(0,0,0,0.15), 0px 8px 1px rgba(0,0,0,0.1); } .bg-red { background: #de5448; } .bg-blue { background: #4267b2; } .bg-green { background: #4CAF50; } </code></pre> </li> <li>Center: Easily center anything, horizontally and vertically, with 3 lines of CSS.<br> CSS: <pre><code>.center { display: flex; align-items: center; justify-content: center; } </code></pre> </li> <li>Cursors (as in, mouse cursor):<br> HTML: <pre><code>&lt;div class="wrapper"&gt; &lt;div class="tile"&gt; Default &lt;/div&gt; &lt;div class="tile tile--image-cursor"&gt; Image Cursor &lt;/div&gt; &lt;div class="tile tile--emoji-cursor"&gt; Emoji Cursor &lt;/div&gt; &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>.wrapper { display: flex; height: 100vh; align-items: center; justify-content: center; background: #4776e6; background: -webkit-linear-gradient(to right, #4776e6, #8e54e9); background: linear-gradient(to right, #4776e6, #8e54e9); padding: 0 10px; } .tile { width: 200px; height: 200px;display: flex; align-items: center; justify-content: center; background-color: #de5448; margin-right: 10px;color: #fff; font-size: 1.4em; text-align: center; } .tile--image-cursor { background-color: #1da1f2; cursor: url(https://picsum.photos/20/20), auto; } .tile--emoji-cursor { background-color: #4267b2; cursor: url("data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' width='40' height='48' viewport='0 0 100 100' style='fill:black;font-size:24px;'&gt;&lt;text y='50%'&gt;🚀&lt;/text&gt;&lt;/svg&gt;"), auto; } </code></pre> </li> <li>Truncate text</li> <li>Truncate the text to a specific number of lines</li> <li><code>::selection</code> CSS pseudo-element</li> <li>Anything resizable: Did you know that you can make any element resizable, just like <code>&lt;textarea&gt;</code>?</li> <li>CSS modals: You can use the <code>:target</code> pseudo-class to create modals with zero JavaScript.<br> HTML: <pre><code>&lt;div class="wrapper"&gt; &lt;a href="#demo-modal"&gt;Open Demo Modal&lt;/a&gt; &lt;/div&gt; &lt;div id="demo-modal" class="modal"&gt; &lt;div class="modal__content"&gt; &lt;h1&gt;CSS Only Modal&lt;/h1&gt; &lt;p&gt; You can use the :target pseudo-class to create a modals with Zero JavaScript. Enjoy! &lt;/p&gt; &lt;div class="modal__footer"&gt; Made with &lt;i class="fa fa-heart"&gt;&lt;/i&gt;, by &lt;a href="https://twitter.com/denicmarko" target="_blank"&gt;@denicmarko&lt;/a&gt; &lt;/div&gt; &lt;a href="#" class="modal__close"&gt;&amp;times;&lt;/a&gt; &lt;/div&gt; &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>.wrapper { height: 100vh; /* This part is important for centering the content */ display: flex; align-items: center; justify-content: center; /* End center */ background: -webkit-linear-gradient(to right, #834d9b, #d04ed6); background: linear-gradient(to right, #834d9b, #d04ed6); } .wrapper a { display: inline-block; text-decoration: none; padding: 15px; background-color: #fff; border-radius: 3px; text-transform: uppercase; color: #585858; font-family: 'Roboto', sans-serif; } .modal { visibility: hidden; opacity: 0; position: absolute; top: 0; right: 0; bottom: 0; left: 0; display: flex; align-items: center; justify-content: center; background: rgba(77, 77, 77, .7); transition: all .4s; } .modal:target { visibility: visible; opacity: 1; } .modal__content { border-radius: 4px; position: relative; width: 500px; max-width: 90%; background: #fff; padding: 1em 2em; } .modal__footer { text-align: right; a { color: #585858; } i { color: #d02d2c; } } .modal__close { position: absolute; top: 10px; right: 10px; color: #585858; text-decoration: none; } </code></pre> </li> <li><code>calc()</code>: The <code>calc()</code> CSS function lets you perform calculations when specifying CSS property values.</li> <li>Style empty elements</li> <li>Create custom scrollbar</li> <li>position sticky</li> <li>CSS Scroll Snap</li> <li>Dynamic Tooltips</li> <li>caret-color</li> <li><code>::in-range</code> and <code>::out-of-range</code> pseudo-classes</li> <li>Fancy text</li> <li>Flex gap</li> <li><code>grayscale()</code> function: Use the <code>grayscale()</code> filter function to convert the input image to grayscale.</li> <li>Rounded, gradiant borders:<br> HTML: <pre><code>&lt;div class="box gradient-border"&gt; This is a div with rounded gradient border. &lt;/div&gt; </code></pre> <p>CSS:</p> <pre><code>.gradient-border { border: solid 5px transparent; border-radius: 10px; background-image: linear-gradient(white, white), linear-gradient(315deg,#833ab4,#fd1d1d 50%,#fcb045); background-origin: border-box; background-clip: content-box, border-box; } /* Demo code, unrelated to the feature. */ .box { width: 350px; height: 100px; display: flex; align-items: center; justify-content: center; margin: 100px auto; } </code></pre> </li> </ul> https://tedneward.github.io/Research/presentation/derbyjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/derbyjs/index.html <p>"Derby includes a powerful data synchronization engine called Racer that automatically syncs data among browsers, servers, and a database. Models subscribe to changes on specific objects, enabling granular control of data propagation without defining channels. Racer supports offline usage and conflict resolution out of the box, which greatly simplifies writing multi-user applications.</p> <p>"Derby applications load immediately and can be indexed by search engines, because the same templates render on both server and client. In addition, templates define bindings, which instantly update the view when the model changes and vice versa. Derby makes it simple to write applications that load as fast as a search engine, are as interactive as a document editor, and work offline."</p> <p><a href="https://derbyjs.com/">Website</a> | <a href="https://github.com/derbyjs">Source</a></p> https://tedneward.github.io/Research/presentation/divkit/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/divkit/index.html <p><a href="https://divkit.tech/en/">Website</a> | <a href="https://github.com/divkit/divkit">Source</a> | <a href="https://divkit.tech/doc">Docs</a></p> <p><strong>DivKit</strong> 🐋 is an open source Server-Driven UI (SDUI) framework. It allows you to roll out server-sourced updates to different app versions. Also, it can be used for fast UI prototyping, allowing you to write a layout once and then ship it to iOS, Android, and Web platforms. DivKit is an excellent choice to start using server-driven UI in your project because it can be easily integrated as a simple view in any part of your app. At the starting point, you don’t need a server integration. You can include all JSON on the client-side to try it in a real-world application.</p> <p>Also, we’ve made a <a href="https://divkit.tech/playground">sandbox</a> for you to experiment with. You can try different samples in the web editor and see the results on the web or in the Android <a href="https://play.google.com/store/apps/details?id=com.yandex.divkit.demo">demo app</a>, both of which are available on Google Play. We’ll publish the iOS demo app shortly. The UI in the demo can be updated live: the sandbox connects to the demo app via web sockets.</p> https://tedneward.github.io/Research/presentation/drawflow/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/drawflow/index.html <p><a href="https://jerosoler.github.io/Drawflow/">Website</a> | <a href="https://github.com/jerosoler/Drawflow">Source</a></p> https://tedneward.github.io/Research/presentation/elmish/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/elmish/index.html <p>Bringing <a href="../languages/elm">Elm</a>-like constructs to the <a href="../languages/fable">Fable</a> world.</p> <p><a href="https://elmish.github.io/">Website</a></p> <p><a href="https://www.compositional-it.com/news-blog/ui-programming-with-elmish-in-f/">Blog post</a> describing it.</p> https://tedneward.github.io/Research/presentation/expo/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/expo/index.html <p><a href="https://expo.dev/">Website</a> | <a href="https://github.com/expo/expo">Source</a> | <a href="https://docs.expo.dev/">Docs</a></p> https://tedneward.github.io/Research/presentation/fluentui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fluentui/index.html <p><a href="https://storybooks.fluentui.dev/react/">Website</a> | <a href="https://github.com/microsoft/fluentui">Source</a></p> https://tedneward.github.io/Research/presentation/foundation/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/foundation/index.html <p><a href="https://get.foundation/">Website</a> | <a href="https://github.com/foundation/">Source</a></p> https://tedneward.github.io/Research/presentation/glance/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/glance/index.html <p><a href="https://github.com/glanceapp/glance">Source</a></p> <h2>Features</h2> <h3>Various widgets</h3> <ul> <li>RSS feeds</li> <li>Subreddit posts</li> <li>Hacker News posts</li> <li>Weather forecasts</li> <li>YouTube channel uploads</li> <li>Twitch channels</li> <li>Market prices</li> <li>Docker containers status</li> <li>Server stats</li> <li>Custom widgets</li> </ul> <p>and many more...</p> <h3>Fast and lightweight</h3> <ul> <li>Low memory usage</li> <li>Few dependencies</li> <li>Minimal vanilla JS</li> <li>Single &lt;20mb binary available for multiple OSs &amp; architectures and just as small Docker container</li> <li>Uncached pages usually load within ~1s (depending on internet speed and number of widgets)</li> </ul> <h3>Tons of customizability</h3> <ul> <li>Different layouts</li> <li>As many pages/tabs as you need</li> <li>Numerous configuration options for each widget</li> <li>Multiple styles for some widgets</li> <li>Custom CSS</li> </ul> <h3>Optimized for mobile devices</h3> <p>Because you'll want to take it with you on the go.</p> https://tedneward.github.io/Research/presentation/graphviz/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/graphviz/index.html <p><a href="https://graphviz.org/">Website</a> | <a href="https://gitlab.com/graphviz/graphviz">Source</a></p> https://tedneward.github.io/Research/presentation/hibiki/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/hibiki/index.html <h2><a href="https://hibikihtml.com/">Website</a> | <a href="https://github.com/dashborg/hibiki">GitHub</a> | <a href="https://playground.hibikihtml.com/tutorial/">Tutorial</a> | <a href="https://playground.hibikihtml.com/">Playground</a> | <a href="https://docs.hibikihtml.com/">Docs</a></h2> <h2>Getting Started</h2> <p>Getting started is easy. There is no JavaScript stack to set up, no boilerplate/scaffolding, and<br> no build tools to download and run.<br> Just add one script tag to the top of your page or template:</p> <pre><code>&lt;script src="https://cdn.hibikihtml.com/hibiki/v0.3.2/hibiki-prod.min.js"&gt;&lt;/script&gt; </code></pre> <p>Wrap any portion of your content with a &lt;template hibiki&gt; tag and you have your first<br> Hibiki HTML app. All plain HTML content is rendered as is, and because Hibiki HTML uses the browser's HTML parser,<br> it follows the same rules as standard browser HTML.<br> Note that all these code examples can be viewed and edited in the<br> <a href="https://playground.hibikihtml.com">Hibiki HTML Playground</a>.</p> <pre><code>&lt;template hibiki&gt; &lt;h1 class="title"&gt;&amp;#x1f338; Hibiki HTML&lt;/h1&gt; &lt;p&gt;Hibiki HTML &lt;i&gt;is&lt;/i&gt; HTML&lt;/p&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-gs">https://playground.hibikihtml.com/?codeid=readme-gs</a>)</p> <h2>Data / Rendering Dynamic Content</h2> <p>Hibiki HTML applications have a built-in frontend data model. You can initialize it with<br> any JSON object using the <code>&lt;hibiki-data&gt;</code> tag. To render text use<br> the <code>&lt;h-text&gt;</code> tag or inline <code>{{ ... }}</code> syntax. Attributes and style properties<br> are evaluated dynamically if they start with a <code>*</code>.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt; {"color": "blue", "name": "Mike"} &lt;/hibiki-data&gt; &lt;p&gt; {{ $.name }}'s favorite color is &lt;span style="font-weight: bold; color: *$.color"&gt;{{ $.color }}&lt;/span&gt; &lt;/p&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-data-1">https://playground.hibikihtml.com/?codeid=readme-data-1</a>)<br> Hibiki HTML supports a full expression language, including all of the standard conditional and arithmetic<br> operators. Additional classes can be turned on/off using the shorthand<br> attribute syntax <code>class.[class-name]="true/false expression"</code>.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt; {"numlights": 4, "selected": true, "index": 5, "isprimary": true} &lt;/hibiki-data&gt; &lt;div class="box" style="font-weight: *($.selected ? 'bold' : 'normal')"&gt; Bold Text (if selected) &lt;/div&gt; &lt;div class="box"&gt;Index: &lt;h-text bind="$.index + 1"&gt;&lt;/h-text&gt;&lt;/div&gt; &lt;div class="box"&gt; There are {{$.numlights}} light{{$.numlights &gt; 1 ? 's' : ''}} &lt;/div&gt; &lt;div class="box"&gt; &lt;button class="button" class.is-primary="*$.isprimary"&gt;Primary Button&lt;/button&gt; &lt;/div&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-data-2">https://playground.hibikihtml.com/?codeid=readme-data-2</a>)<br> You can conditionally include elements using the <code>if</code> attribute. Looping<br> is handled by the <code>foreach</code> attribute. You can loop over arrays or<br> objects. The foreach attribute uses a special syntax of <code>@item in $.array</code>,<br> where every iteration <code>@item</code> will be assigned to an element of the array or object.<br> If you provide a second variable it will capture the array index or element key:<br> <code>(@item, @key) in $.object</code> or <code>(@item, @idx) in $.array</code>.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt; {"fruits": [{"name": "apple", "emoji": "&amp;#127822;"}, {"name": "banana", "emoji": "&amp;#127820;"}, {"name": "blueberry", "emoji": "&amp;#129744;"}], "selected": "banana"} &lt;/hibiki-data&gt; &lt;div class="content box"&gt; &lt;ul&gt; &lt;li foreach="@fruit in $.fruits"&gt; {{ @fruit.name }} {{ @fruit.emoji }} &lt;span if="@fruit.name == $.selected"&gt;(selected)&lt;/span&gt; &lt;/li&gt; &lt;/ul&gt; &lt;/div&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-data-3">https://playground.hibikihtml.com/?codeid=readme-data-3</a>)</p> <h2>Handlers</h2> <p>To update data (and dynamically change content), Hibiki HTML supports <em>handlers</em>. Basic handlers respond to<br> events like <em>click</em>, <em>mount</em>, <em>submit</em>, <em>change</em>, etc.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt;{"color": "blue"}&lt;/hibiki-data&gt; &lt;div class="box" style="background-color: *$.color; color: white"&gt; The color is {{$.color}} &lt;/div&gt; &lt;button class="button" click.handler="$.color = 'green'"&gt;Change Color&lt;/button&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-handlers-1">https://playground.hibikihtml.com/?codeid=readme-handlers-1</a>)<br> You can make remote AJAX calls for JSON data using a remote handler. In this example<br> the call to <a href="https://testapi.hibikihtml.com/api/get-color-1">https://testapi.hibikihtml.com/api/get-color-1</a> will return the JSON<br> response <code>{"color": "purple"}</code>. We assign it to a <em>context variable</em> <code>@resp</code> and<br> then assign <code>$.color = @resp.color</code>.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt;{"color": "blue"}&lt;/hibiki-data&gt; &lt;div class="box" style="background-color: *$.color; color: white"&gt; The color is {{$.color}} &lt;/div&gt; &lt;button class="button" click.handler="$.color = 'green'"&gt;Change Color&lt;/button&gt; &lt;button class="button" click.handler="@resp = GET https://testapi.hibikihtml.com/api/get-color-1; $.color = @resp.color;"&gt; GET /api/get-color-1 &lt;/button&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-handlers-2">https://playground.hibikihtml.com/?codeid=readme-handlers-2</a>)<br> You can make GET, POST, PUT, PATCH, or DELETE requests by changing the verb in front of the URL. You can<br> pass query arguments (or JSON data bodies) by passing arguments to your handler by using <code>(arg1=val1, arg2=val2...)</code> (complex arguments will be JSON encoded), and you can define local handlers using the <code>define-handler</code> tag for convenience and reuse.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt;{"color": "blue"}&lt;/hibiki-data&gt; &lt;define-handler name="//@local/get-color"&gt; @resp = GET https://testapi.hibikihtml.com/api/get-color-1(name="Michelle"); $.color = @resp.color; &lt;/define-handler&gt; &lt;div class="box" style="background-color: *$.color; color: white"&gt; The color is {{$.color}} &lt;/div&gt; &lt;button class="button" click.handler="$.color = 'green'"&gt;Change Color&lt;/button&gt; &lt;button class="button" click.handler="//@local/get-color"&gt; Get Michelle's Color &lt;/button&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-handlers-3">https://playground.hibikihtml.com/?codeid=readme-handlers-3</a>)<br> Connecting to an existing API? Don't worry, Hibiki HTML handlers support advanced options<br> like CORS, CSRF, parameter encodings, BLOB results, and multipart file uploads.</p> <h2>Hibiki Actions</h2> <p>The magic of Hibiki HTML is that every handler is really just a series of actions. <em>Hibiki Actions</em><br> are primitives like <em>setdata</em>, <em>if</em>, <em>callhandler</em>, <em>fireevent</em>, <em>invalidate</em>, <em>html</em>, etc. Every action<br> that can you can write in a handler, also has a JSON representation.<br> This means we can write a <em>backend</em> handler that returns a JSON response that scripts<br> and updates the frontend! Here's an example JSON response that is equivalent to running<br> <code>$.color = 'DeepSkyBlue'</code> on the frontend:</p> <pre><code>{"hibikiactions": [ {"actiontype": "setdata", "setpath": "$.color", "data": "DeepSkyBlue"} ]} </code></pre> <p>I've set up <a href="https://testapi.hibikihtml.com/api/set-color-action">https://testapi.hibikihtml.com/api/set-color-action</a> to return just that. Now<br> if we have our click handler call that URL, we'll see the color change automatically.</p> <pre><code>&lt;template hibiki&gt; &lt;hibiki-data&gt;{"color": "red"}&lt;/hibiki-data&gt; &lt;div class="box" style="background-color: *$.color; color: white"&gt; The color is {{$.color}} &lt;/div&gt; &lt;button class="button" click.handler="$.color = 'green'"&gt;Change Color&lt;/button&gt; &lt;button class="button" click.handler="GET https://testapi.hibikihtml.com/api/set-color-action;"&gt; Backend Set Color Action &lt;/button&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-actions-1">https://playground.hibikihtml.com/?codeid=readme-actions-1</a>)<br> Backend handlers are <em>very</em> powerful. You can set return values, return data and BLOBs (like images) in<br> one request, mix frontend and backend data with expressions, conditionally execute actions, and more.<br> You can also use backend handlers to create multi-page Hibiki HTML applications by returning a<br> new HTML template to be rendered.</p> <h2>Components and Libraries</h2> <p>Hibiki HTML makes it easy to use, share, and bundle components for easy reuse. The ecosystem is just getting<br> started, but you can write native Hibiki HTML components, link to 3rd party JavaScript (D3, CodeMirror, etc.),<br> and import ReactJS components.</p> <pre><code>&lt;template hibiki&gt; &lt;define-component name="colorbox"&gt; &lt;div class="box" style="background-color: *$args.color; color: white"&gt; The color is {{$args.color}} &lt;/div&gt; &lt;/define-component&gt; &lt;local-colorbox color="green"&gt;&lt;/local-colorbox&gt; &lt;local-colorbox color="blue"&gt;&lt;/local-colorbox&gt; &lt;local-colorbox color="purple"&gt;&lt;/local-colorbox&gt; &lt;/template&gt; </code></pre> <p>(Playground Link - <a href="https://playground.hibikihtml.com/?codeid=readme-comps-1">https://playground.hibikihtml.com/?codeid=readme-comps-1</a>)</p> https://tedneward.github.io/Research/presentation/htmlx/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/htmlx/index.html <p>htmx allows you to access AJAX, CSS Transitions, WebSockets and Server Sent Events directly in HTML, using attributes, so you can build modern user interfaces with the simplicity and power of hypertext.</p> <p>htmx is the successor to <a href="http://intercoolerjs.org/">intercooler.js</a>.</p> <p><a href="https://htmx.org/">Website</a> | <a href="https://github.com/bigskysoftware/htmx">Source</a> | <a href="https://htmx.org/essays/">Essays</a></p> <h3>AJAX</h3> <p>The core of htmx is a set of attributes that allow you to issue AJAX requests directly from HTML:</p> <table> <thead> <tr> <th>Attribute </th> <th> Description</th> </tr> </thead> <tbody> <tr> <td>hx-get </td> <td> Issues a GET request to the given URL</td> </tr> <tr> <td>hx-post </td> <td> Issues a POST request to the given URL</td> </tr> <tr> <td>hx-put </td> <td> Issues a PUT request to the given URL</td> </tr> <tr> <td>hx-patch </td> <td> Issues a PATCH request to the given URL</td> </tr> <tr> <td>hx-delete </td> <td> Issues a DELETE request to the given URL</td> </tr> </tbody> </table> <p>Each of these attributes takes a URL to issue an AJAX request to. The element will issue a request of the specified type to the given URL when the element is triggered:</p> <pre><code class="language-html"> &lt;div hx-put="/messages"&gt; Put To Messages &lt;/div&gt; </code></pre> <p>This tells the browser:</p> <blockquote> <p>When a user clicks on this div, issue a PUT request to the URL /messages and load the response into the div</p> </blockquote> https://tedneward.github.io/Research/presentation/isis/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/isis/index.html <p><a href="http://isis.apache.org/">Website</a> | <a href="https://github.com/apache/isis.git">Source</a></p> <hr> <p>Generally the preferred way to get started is with a starter template</p> <p>"helloworld" starter app instructions:</p> <pre><code>curl https://codeload.github.com/apache/isis-app-helloworld/zip/2.0.0-M3 | jar xv cd isis-app-helloworld-2.0.0-M3 mvn clean install mvn spring-boot:run </code></pre> <p>"simpleapp" more structured start app instructions:</p> <pre><code>curl https://codeload.github.com/apache/isis-app-simpleapp/zip/2.0.0-M3 | jar xv cd isis-app-simpleapp-2.0.0-M3 mvn clean install mvn -pl webapp spring-boot:run </code></pre> <p>An example/demo app is available via Docker:</p> <pre><code>docker pull apacheisis/demo-springboot:latest docker run \ --name demo \ --detach \ -p8080:8080 \ apacheisis/demo-springboot:latest </code></pre> <p>... and then accessible at <a href="http://localhost:8080/wicket/">http://localhost:8080/wicket/</a> ; login with "sven"/"pass".</p> <hr> <p>See also: <a href="/distribution/restfulobjects.html">Restful Objects</a></p> <hr> <h2>Documentation</h2> <p><a href="http://isis.apache.org/setupguide/2.0.0-M3/about.html">Setup Guide</a> <a href="http://isis.apache.org/userguide/2.0.0-M3/about.html">User Guide</a> | <a href="http://isis.apache.org/refguide/2.0.0-M3/about.html">Reference Guide</a> | <a href="http://isis.apache.org/testing/2.0.0-M3/about.html">Testing</a></p> https://tedneward.github.io/Research/presentation/json-render/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/json-render/index.html <p><a href="https://json-render.dev/">Website</a> | <a href="https://github.com/vercel-labs/json-render">Source</a> | <a href="https://json-render.dev/docs">Docs</a></p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://thenewstack.io/vercels-json-render-a-step-toward-generative-ui/">https://thenewstack.io/vercels-json-render-a-step-toward-generative-ui/</a></li> </ul> https://tedneward.github.io/Research/presentation/console/jexer/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/jexer/index.html <p><a href="https://jexer.sourceforge.io/index.html">Website</a> | <a href="https://gitlab.com/AutumnMeowMeow/jexer">Source</a></p> <p>Maven:</p> <pre><code> &lt;dependency&gt; &lt;groupId&gt;com.gitlab.klamonte&lt;/groupId&gt; &lt;artifactId&gt;jexer&lt;/artifactId&gt; &lt;version&gt;1.6.0&lt;/version&gt; &lt;/dependency&gt; </code></pre> https://tedneward.github.io/Research/presentation/console/spectre-console/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/spectre-console/index.html <p><a href="https://github.com/spectreconsole">Source</a></p> <p>Almost anything a console can do, Spectre.Console can provide. Wow.</p> https://tedneward.github.io/Research/presentation/console/vtmc/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/vtmc/index.html <p><a href="https://github.com/jclulow/vtmc">Source</a></p> https://tedneward.github.io/Research/presentation/css/bootstrap/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/css/bootstrap/index.html <p><a href="http://www.getbootstrap.com/">Website</a></p> <h2>Related resources</h2> <p><a href="https://bootsnipp.com/#google_vignette">Bootsnipp</a>: Design elements, playground, and code snippets for Bootstrap</p> <p><a href="https://bootswatch.com/">Bootswatch</a>: Free themes</p> <p><a href="https://bootswatchr.com/">BootsWatchr</a>: Getting to know Bootstrap Grid System</p> <p><a href="https://shapebootstrap.net/">ShapeBootstrap</a>: Your favorite web design school</p> <p><a href="https://startbootstrap.com/#google_vignette">StartBootstrap</a>: Themes, templates, UI tools</p> <p><a href="https://stylebootstrap.info/#">StyleBootstrap</a>: Themes, templates, styles, generator, customize</p> <h2>Articles</h2> <ul> <li><a href="https://www.smashingmagazine.com/2013/03/customizing-bootstrap/">How to Customize Bootstrap</a></li> <li><a href="https://www.queness.com/post/13029/extend-twitter-bootstrap-javascript-plugins">Extend Boostrap JS plugins</a></li> <li><a href="https://www.queness.com/post/11632/18-useful-twitter-boostrap-goodies-you-should-know">18 Useful Bootstrap Goodies You Should Know</a></li> </ul> <h2>Books</h2> <ul> <li><a href="https://www.syncfusion.com/resources/techportal/details/ebooks/twitterbootstrap3">Twitter Bootstrap 3 Succinctly</a> - Peter Shaw</li> <li><a href="https://www.syncfusion.com/ebooks/twitterbootstrap4-succinctly">Twitter Bootstrap 4 Succinctly</a> - Peter Shaw</li> <li><a href="https://www.syncfusion.com/resources/techportal/details/ebooks/twitterbootstrap">Twitter Bootstrap Succinctly</a> - Peter Shaw</li> </ul> https://tedneward.github.io/Research/presentation/css/specs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/css/specs/index.html <h2>Specs</h2> <p><a href="https://www.w3.org/TR/CSS/">CSS Snapshot</a> (list of links to all current specs) | <a href="https://www.w3.org/TR/CSS/#terms">Index to all terms/Glossary</a></p> <p>As of 2021, Cascading Style Sheets (CSS) is defined by the following specifications.</p> <p><a href="https://www.w3.org/TR/CSS2/">CSS Level 2</a>, latest revision (including errata) <strong>[CSS2]</strong>: This defines the core of CSS, parts of which are overridden by later specifications. We recommend in particular reading Chapter 2, which introduces some of the basic concepts of CSS and its design principles.</p> <p><a href="https://www.w3.org/TR/css-syntax-3/">CSS Syntax Level 3</a> <strong>[CSS-SYNTAX-3]</strong>: Replaces CSS2§4.1, CSS2§4.2, CSS2§4.4, and CSS2§G, redefining how CSS is parsed.</p> <p><a href="https://www.w3.org/TR/css-style-attr/">CSS Style Attributes</a> <strong>[CSS-STYLE-ATTR]</strong>: Defines how CSS declarations can be embedded in markup attributes.</p> <p><a href="https://www.w3.org/TR/css3-mediaqueries/">Media Queries Level 3</a> <strong>[CSS3-MEDIAQUERIES]</strong>: Replaces CSS2§7.3 and expands on the syntax for media-specific styles.</p> <p><a href="https://www.w3.org/TR/css-conditional-3/">CSS Conditional Rules Level 3</a> <strong>[CSS-CONDITIONAL-3]</strong>: Extends and supersedes CSS2§7.2, updating the definition of @media rules to allow nesting and introducing the @supports rule for feature-support queries.</p> <p><a href="https://www.w3.org/TR/selectors-3/">Selectors Level 3</a> <strong>[SELECTORS-3]</strong>: Replaces CSS2§5 and CSS2§6.4.3, defining an extended range of selectors.</p> <p><a href="https://www.w3.org/TR/css-namespaces/">CSS Namespaces</a> <strong>[CSS3-NAMESPACE]</strong>: Introduces an @namespace rule to allow namespace-prefixed selectors.</p> <p><a href="https://www.w3.org/TR/css-cascade-4/">CSS Cascading and Inheritance Level 4</a> <strong>[CSS-CASCADE-4]</strong>: Extends and supersedes CSS2§1.4.3 and CSS2§6, as well as [CSS-CASCADE-3]. Describes how to collate style rules and assign values to all properties on all elements. By way of cascading and inheritance, values are propagated for all properties on all elements.</p> <p><a href="https://www.w3.org/TR/css-values-3/">CSS Values and Units Level 3</a> <strong>[CSS-VALUES-3]</strong>: Extends and supersedes CSS2§1.4.2.1, CSS2§4.3, and CSS2§A.2.1–3, defining CSS’s property definition syntax and expanding its set of units.</p> <p><a href="https://www.w3.org/TR/css-variables-1/">CSS Custom Properties for Cascading Variables Module Level 1</a> <strong>[CSS-VARIABLES-1]</strong>: Introduces cascading variables as a new primitive value type that is accepted by all CSS properties, and custom properties for defining them.</p> <p><a href="https://www.w3.org/TR/css-box-3/">CSS Box Model Level 3</a> <strong>[CSS-BOX-3]</strong>: Replaces CSS2§8.1, §8.2, §8.3 (but not §8.3.1), and §8.4.</p> <p><a href="https://www.w3.org/TR/css-color-3/">CSS Color Level 3</a> <strong>[CSS-COLOR-3]</strong>: Extends and supersedes CSS2§4.3.6, CSS2§14.1, and CSS2§18.2, introducing an extended range of color values. Also introduces the opacity property.</p> <p><a href="https://www.w3.org/TR/css-backgrounds-3/">CSS Backgrounds and Borders Level 3</a> <strong>[CSS-BACKGROUNDS-3]</strong>: Extends and supersedes CSS2§8.5 and CSS2§14.2, providing more control of backgrounds and borders, including layered background images, image borders, and drop shadows.</p> <p><a href="https://www.w3.org/TR/css-images-3/">CSS Images Level 3</a> <strong>[CSS-IMAGES-3]</strong>: Redefines and incorporates the external 2D image value type, introduces native 2D gradients, and adds additional controls for replaced element sizing and rendering.</p> <p><a href="https://www.w3.org/TR/css-fonts-3/">CSS Fonts Level 3</a> <strong>[CSS-FONTS-3]</strong>: Extends and supersedes CSS2§15 and provides more control over font choice and feature selection.</p> <p><a href="https://www.w3.org/TR/css-writing-modes-3/">CSS Writing Modes Level 3</a> <strong>[CSS-WRITING-MODES-3]</strong>: Defines CSS support for various international writing modes, such as left-to-right (e.g. Latin or Indic), right-to-left (e.g. Hebrew or Arabic), bidirectional (e.g. mixed Latin and Arabic) and vertical (e.g. Asian scripts). Replaces and extends CSS2§8.6 and §9.10.</p> <p><a href="https://www.w3.org/TR/css-multicol-1/">CSS Multi-column Layout Level 1</a> <strong>[CSS-MULTICOL-1]</strong>: Introduces multi-column flows to CSS layout.</p> <p><a href="https://www.w3.org/TR/css-flexbox-1/">CSS Flexible Box Module Level 1</a> <strong>[CSS-FLEXBOX-1]</strong>: Introduces a flexible linear layout model for CSS.</p> <p><a href="https://www.w3.org/TR/css-ui-3/">CSS User Interface Module Level 3</a> <strong>[CSS-UI-3]</strong>: Extends and supersedes CSS2§18.1 and CSS2§18.4, defining cursor, outline, and several new CSS features that also enhance the user interface.</p> <p><a href="https://www.w3.org/TR/css-contain-1/">CSS Containment Module Level 1</a> <strong>[CSS-CONTAIN-1]</strong>: Introduces the contain property, which enforces the independent CSS processing of an element’s subtree in order to enable heavy optimizations by user agents when used well.</p> <p><a href="https://www.w3.org/TR/css-transforms-1/">CSS Transforms Level 1</a> <strong>[CSS-TRANSFORMS-1]</strong>: Introduces coordinate-based graphical transformations to CSS.</p> <p><a href="https://www.w3.org/TR/compositing-1/">CSS Compositing and Blending Level 1</a> <strong>[COMPOSITING]</strong>: Defines the compositing and blending of overlaid content and introduces features to control their modes.</p> <p><a href="https://www.w3.org/TR/css-easing-1/">CSS Easing Functions Level 1</a> <strong>[CSS-EASING-1]</strong>: Describes a way for authors to define a transformation that controls the rate of change of some value. Applied to animations, such transformations can be used to produce animations that mimic physical phenomena such as momentum or to cause the animation to move in discrete steps producing robot-like movement.</p> <p><a href="https://www.w3.org/TR/css-counter-styles-3/">CSS Counter Styles Level 3</a> <strong>[CSS-COUNTER-STYLES-3]</strong>: Introduces the @counter-style rule, which allows authors to define their own custom counter styles for use with CSS list-marker and generated-content counters [CSS-LISTS-3]. It also predefines a set of common counter styles, including the ones present in CSS2 and CSS2.1.</p> <p><a href="https://www.w3.org/Style/CSS/specs.en.html">List of all current-work CSS specifications</a></p> <h4>Older specs</h4> <p><a href="https://www.w3.org/TR/CSS1/">CSS Level 1</a> (1996, revised 2008; superseded in 2016)</p> https://tedneward.github.io/Research/presentation/dflex/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/dflex/index.html <p><a href="https://www.dflex.dev/">Website</a> | <a href="https://github.com/dflex-js/dflex">Source</a></p> https://tedneward.github.io/Research/presentation/docbook/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/docbook/index.html <p><a href="https://docbook.org/">Website</a></p> https://tedneward.github.io/Research/presentation/dzslides/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/dzslides/index.html <p><a href="http://paulrouget.com/dzslides">Website</a> | <a href="https://github.com/paulrouget/dzslides">Source</a></p> https://tedneward.github.io/Research/presentation/emberjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/emberjs/index.html <p><a href="https://emberjs.com/">Website</a> | <a href="https://github.com/emberjs/ember.js">Source</a></p> https://tedneward.github.io/Research/presentation/fabulous/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fabulous/index.html <p><a href="https://fsprojects.github.io/Fabulous/">Website</a> | <a href="https://github.com/fsprojects/Fabulous">Source</a></p> <p><em>F# Functional App Development, using declarative dynamic UI</em></p> <h2>Documentation</h2> <ul> <li> <p><a href="https://fsprojects.github.io/Fabulous/Fabulous.XamarinForms/index.html#getting-started">Getting started</a></p> </li> <li> <p><a href="https://fsprojects.github.io/Fabulous/">Fabulous documentation</a></p> </li> </ul> <h2>About Fabulous</h2> <p>Fabulous aims to provide all the tools to let you create your own mobile and desktop apps using only F# and the <a href="https://guide.elm-lang.org/architecture/">Model-View-Update architecture</a> (shorten to MVU), with a great F# DSL for building dynamic UIs.<br> The combination of F# and MVU makes for a great development experience.</p> <p>Note that Fabulous itself does not provide UI controls, so you'll need to combine it with another framework like Xamarin.Forms.</p> <h3>Fabulous for Xamarin.Forms</h3> <p>Fabulous for Xamarin.Forms combines both frameworks with a tailored DSL to let you take advantage of everything Xamarin.Forms has to offer while keeping all the benefits of Fabulous.</p> <p>With Fabulous for Xamarin.Forms, you will be able to write complete applications in F# like this:</p> <pre><code class="language-fsharp">type Model = { Text: string } type Msg = ButtonClicked let init () = { Text = "Hello Fabulous!" } let update msg model = match msg with | ButtonClicked -&gt; { model with Text = "Thanks for using Fabulous!" } let view model dispatch = View.ContentPage( View.StackLayout( children = [ View.Image(source = Image.fromPath "fabulous.png") View.Label(text = model.Text) View.Button(text = "Click me", command = fun () -&gt; dispatch ButtonClicked) ] ) ) </code></pre> <p><a href="https://github.com/fsprojects/Fabulous/tree/master/Fabulous.XamarinForms">Learn more about Fabulous for Xamarin.Forms</a></p> <h3>Can I use Fabulous with other UI frameworks?</h3> <p>Fabulous is not tied to Xamarin.Forms.<br> If you want to use your favorite UI framework instead, it's possible.</p> <p>To help you with that, Fabulous comes with its own code generator, called Fabulous.CodeGen, to automate the creation of a DSL specific to your UI framework.</p> <p><a href="Fabulous.CodeGen/README.md">Learn more about Fabulous.CodeGen</a></p> https://tedneward.github.io/Research/presentation/flyde/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/flyde/index.html <p><a href="https://www.flyde.dev/">Website</a> | <a href="https://github.com/flydelabs/flyde">Source</a></p> <p>Flyde is a holistic solution for prototyping, integrating, evaluating and iterating on AI-heavy backend logic. It's a visual extension of TypeScript that runs in-codebase, providing the missing link between developers and non-developers working on backend AI workflows.</p> <p>Flyde allows you to create visual flows for backend services like AI agents, prompt chains, API orchestration, and agentic workflows - directly integrated with your existing codebase. It includes a VSCode extension and seamlessly integrates with existing TypeScript/JavaScript code.</p> <h4>Who is Flyde for?</h4> <ul> <li> <p>Product Teams &amp; Developers: Flyde bridges the gap between technical and non-technical team members. Product managers, designers, and backend developers can collaborate on the same visual flows, making complex backend logic transparent and enabling everyone to contribute meaningfully to the development process.</p> </li> <li> <p>Developers Building AI-Powered Backends: If you're dealing with complex prompt chains, AI agents, or multi-step backend AI workflows, Flyde provides a more manageable way to build, debug, and maintain these systems while keeping everything in your codebase.</p> </li> <li> <p>Teams Seeking Visual Clarity: Whether you're prototyping new backend features, managing complex business logic, API orchestration, or simply looking for a better way to document and understand your backend application flow, Flyde's visual approach helps you see the big picture while maintaining the power of code.</p> </li> </ul> https://tedneward.github.io/Research/presentation/fox/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fox/index.html <p><a href="http://www.fox-toolkit.org/">Website</a> | <a href="http://www.fox-toolkit.org/download.html">Downloads</a> (source zips available)</p> https://tedneward.github.io/Research/presentation/glimesh/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/glimesh/index.html <p><a href="https://glimesh.tv/">Website</a> | <a href="https://github.com/glimesh/glimesh.tv">Source</a></p> https://tedneward.github.io/Research/presentation/gtk/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/gtk/index.html <p><a href="https://www.gtk.org/">Website</a> | <a href="https://gitlab.gnome.org/GNOME/gtk/">Source</a></p> <ul> <li><a href="https://www.gtkmm.org/en/">gtkmm</a>: the official C++ interface for the popular GUI library GTK+. Highlights include typesafe callbacks, and a comprehensive set of widgets that are easily extensible via inheritance. You can create user interfaces either in code or with the <a href="https://glade.gnome.org/">Glade User Interface designer</a>, using Gtk::Builder. There's extensive documentation, including API reference and a tutorial.</li> </ul> https://tedneward.github.io/Research/presentation/hilla/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/hilla/index.html <p><a href="https://hilla.dev/">Website</a> | <a href="https://github.com/vaadin/hilla">Source</a> (part of <a href="../vaadin">Vaadin</a>)</p> https://tedneward.github.io/Research/presentation/hugo/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/hugo/index.html <p><a href="https://gohugo.io/">Website</a> | <a href="https://github.com/gohugoio/hugo">Source</a></p> https://tedneward.github.io/Research/presentation/jamstack/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/jamstack/index.html <h1>Rationale</h1> <p><strong>Better Performance</strong><br> Why wait for pages to build on the fly when you can generate them at deploy time? When it comes to minimizing the time to first byte, nothing beats pre-built files served over a CDN.</p> <p><strong>Higher Security</strong><br> With server-side processes abstracted into microservice APIs, surface areas for attacks are reduced. You can also leverage the domain expertise of specialist third-party services.</p> <p><strong>Cheaper, Easier Scaling</strong><br> When your deployment amounts to a stack of files that can be served anywhere, scaling is a matter of serving those files in more places. CDNs are perfect for this, and often include scaling in all of their plans.</p> <p><strong>Better Developer Experience</strong><br> Loose coupling and separation of controls allow for more targeted development and debugging, and the expanding selection of CMS options for site generators remove the need to maintain a separate stack for content and marketing.</p> <p><a href="https://jamstack.org/">Website</a></p> https://tedneward.github.io/Research/presentation/juce/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/juce/index.html <p><a href="https://juce.com/">Website</a></p> <p>Pros: battle tested / production ready, fancy</p> <p>Cons: more of an audio framework with UIs, heavy framework (a-la Qt)</p> https://tedneward.github.io/Research/presentation/console/kotter/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/kotter/index.html <p><a href="https://github.com/varabyte/kotter">Source</a></p> https://tedneward.github.io/Research/presentation/console/terminalgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/terminalgui/index.html <p>Terminal.GUI.NET: <a href="https://github.com/migueldeicaza/gui.cs">Source</a></p> <p>TerminalKit for Swift: <a href="https://github.com/migueldeicaza/TermKit">Source</a></p> https://tedneward.github.io/Research/presentation/cordova/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/cordova/index.html <p><a href="https://cordova.apache.org/">Website</a> | <a href="https://github.com/apache?utf8=%E2%9C%93&amp;q=cordova-">Source</a></p> <hr> <p><a href="https://volt.build/">Volt</a> is a build system for Cordova</p> https://tedneward.github.io/Research/presentation/css/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/css/index/index.html <h2>Useful locations</h2> <p>Cheat sheets/references: <a href="https://htmlcheatsheet.com/css/">Interactive cheat sheet</a> | <a href="https://adam-marsden.co.uk/css-cheat-sheet">Another</a></p> <h2>Most useful debugging trick for CSS ever</h2> <p><code>selector { outline: 1px solid red; }</code></p> <h2>CSS Levels</h2> <p>From <a href="https://www.w3.org/TR/CSS/">the CSS Snapshot doc</a>:</p> <ul> <li> <p><strong>CSS Level 1</strong>: The CSS Working Group considers the CSS1 specification to be obsolete. CSS Level 1 is defined as all the features defined in the CSS1 specification (properties, values, at-rules, etc), but using the syntax and definitions in the CSS2.1 specification. CSS Style Attributes defines its inclusion in element-specific style attributes.</p> </li> <li> <p><strong>CSS Level 2</strong>: Although the CSS2 specification is technically a W3C Recommendation, it passed into the Recommendation stage before the W3C had defined the Candidate Recommendation stage. Over time implementation experience and further review has brought to light many problems in the CSS2 specification, so instead of expanding an already unwieldy errata list, the CSS Working Group chose to define CSS Level 2 Revision 1 (CSS2.1). In case of any conflict between the two specs CSS2.1 contains the definitive definition. Once CSS2.1 became Candidate Recommendation--effectively though not officially the same level of stability as CSS2--obsoleted the CSS2 Recommendation. Features in CSS2 that were dropped from CSS2.1 should be considered to be at the Candidate Recommendation stage, but note that many of these have been or will be pulled into a CSS Level 3 working draft, in which case that specification will, once it reaches CR, obsolete the definitions in CSS2. The CSS2.1 specification defines CSS Level 2 and the CSS Style Attributes specification defines its inclusion in element-specific style attributes.</p> </li> <li> <p><strong>CSS Level 3</strong>: builds on CSS Level 2 module by module, using the CSS2.1 specification as its core. Each module adds functionality and/or replaces part of the CSS2.1 specification. The CSS Working Group intends that the new CSS modules will not contradict the CSS2.1 specification: only that they will add functionality and refine definitions. As each module is completed, it will be plugged in to the existing system of CSS2.1 plus previously-completed modules. From this level on modules are levelled independently: for example Selectors Level 4 may well be completed before CSS Line Module Level 3. Modules with no CSS Level 2 equivalent start at Level 1; modules that update features that existed in CSS Level 2 start at Level 3.</p> </li> <li> <p><strong>CSS Level 4 and beyond</strong>: There is no CSS Level 4. Independent modules can reach level 4 or beyond, but CSS the language no longer has levels. ("CSS Level 3" as a term is used only to differentiate it from the previous monolithic versions.)</p> </li> </ul> <h2>Tips, Tricks, etc</h2> <p><a href="https://www.joshwcomeau.com/css/understanding-layout-algorithms">"Understanding layout algorithms"</a>: "The key realization I had is that CSS is so much more than a collection of properties. It's a constellation of inter-connected layout algorithms. Each algorithm is a complex system with its own rules and secret mechanisms. It's not enough to learn what specific properties do. We need to learn how the layout algorithms work, and how they use the properties we provide to them."</p> <p><a href="https://flexboxfroggy.com/">Flexbox Froggy</a>: A cute Frogger-themed series of exercises practicing with CSS Flexbox styling.</p> <h2>Features</h2> <h3>CSS Masonry</h3> <ul> <li><a href="https://www.smashingmagazine.com/2025/12/masonry-things-you-wont-need-library-anymore/">"Masonry: Things Won't Need a Library For Anymore"</a></li> </ul> <h2>Reading</h2> <h3>Books</h3> <ul> <li><a href="http://learn.shayhowe.com/html-css/">A beginner's guide to HTML&amp;CSS</a></li> <li><a href="http://marksheet.io">A free guide to learn HTML and CSS</a></li> <li><a href="http://adaptivewebdesign.info/1st-edition/">Adaptive Web Design</a> - Aaron Gustafson</li> <li><a href="https://github.com/airbnb/css">Airbnb CSS / Sass Styleguide</a> - Airbnb</li> <li><a href="https://airbnb.io/javascript/css-in-javascript/">Airbnb CSS-in-JavaScript Style Guide</a> - Airbnb</li> <li><a href="http://learn.shayhowe.com/advanced-html-css/">An advanced guide to HTML&amp;CSS</a></li> <li><a href="https://atomicdesign.bradfrost.com">Atomic Design</a> - Brad Frost</li> <li><a href="https://web.archive.org/web/20160505010319/http://learnjs.io/canvassing/read/">Canvassing</a></li> <li><a href="http://mdo.github.io/code-guide/">Code Guide: Standards for developing flexible, durable, and sustainable HTML and CSS</a> - Mark Otto</li> <li><a href="https://github.com/cssanimation/css-animation-101">CSS Animation 101</a></li> <li><a href="http://goalkicker.com/CSSBook">CSS Notes for Professionals</a> - Compiled from StackOverflow Documentation (PDF)</li> <li><a href="http://domenlightenment.com">DOM Enlightenment</a> - Cody Lindley (HTML)</li> <li><a href="https://dash.generalassemb.ly">GA Dash</a></li> <li><a href="https://google.github.io/styleguide/htmlcssguide.html">Google's HTML/CSS Style Guide</a></li> <li><a href="https://www.digitalocean.com/community/books/how-to-build-a-website-with-html-ebook">How To Build a Website with HTML</a> - Erin Glass (PDF, EPUB)</li> <li><a href="https://web.archive.org/web/20180816174417/http://howtocodeinhtml.com/HowToCodeInHTML5AndCSS3.pdf">How to Code in HTML5 and CSS3</a> - Damian Wielgosik (PDF)</li> <li><a href="https://www.internetingishard.com">Interneting is Hard (But it Doesn't Have to Be)</a> - Oliver James</li> <li><a href="http://learnlayout.com">Learn CSS Layout</a></li> <li><a href="http://book.mixu.net/css/">Learn CSS Layout the pedantic way</a></li> <li><a href="https://learn.shayhowe.com">Learn to Code HTML &amp; CSS</a> - Shay Howe</li> <li><a href="https://adamschwartz.co/magic-of-css/">Magic of CSS</a> - Adam Schwartz (HTML) (:construction: <em>in process</em>)</li> <li><a href="http://maintainablecss.com">MaintainableCSS</a></li> <li><a href="https://svgpocketguide.com">Pocket Guide to Writing SVG</a> - Joni Trythall</li> <li><a href="https://resilientwebdesign.com/#Resilientweb%20design">Resilient Web Design</a> - Jeremy Keith</li> <li><a href="https://rtlstyling.com">RTL Styling 101</a> - Ahmad Shadeed</li> <li><a href="https://web.archive.org/web/20191116073929/http://smacss.com/">Scalable and Modular Architecture for CSS</a> - Jonathan Snook</li> <li><a href="https://flaviocopes.com/page/css-handbook/">The CSS Handbook</a> - Flavio Copes (PDF, EPUB, Kindle) <em>(email address requested)</em></li> <li><a href="https://ohansemmanuel.github.io/uf_download.html">Understanding Flexbox: Everything you need to know</a> - Ohans Emmanuel</li> <li><a href="https://www.syncfusion.com/ebooks/w3_css_succinctly">W3.CSS Succinctly</a> - Joseph D. Booth</li> <li><a href="http://chimera.labs.oreilly.com/books/1234000001552">Web Audio API</a> - Boris Smus</li> <li><a href="https://leanpub.com/web-visual-effects-with-css3/read">Web Visual Effects with CSS3</a> - Thomas Mak</li> </ul> <h3>Articles</h3> <ul> <li><a href="https://www.howtogeek.com/add-a-smooth-typewriter-animation-to-your-website-without-any-javascript/">https://www.howtogeek.com/add-a-smooth-typewriter-animation-to-your-website-without-any-javascript/</a></li> <li><a href="https://tympanus.net/codrops/2025/11/10/crafting-generative-css-worlds/">Crafting Generative CSS Worlds: Learn how stacked grids and 3D transforms can bring heightmaps to life using nothing but the power of CSS.</a></li> </ul> https://tedneward.github.io/Research/presentation/cyclejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/cyclejs/index.html <p><a href="https://cycle.js.org/">Website</a></p> https://tedneward.github.io/Research/presentation/dillinger/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/dillinger/index.html <p><a href="https://dillinger.io/">Website</a> | <a href="https://github.com/joemccann/dillinger.git">Source</a></p> https://tedneward.github.io/Research/presentation/docmost/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/docmost/index.html <p><a href="https://docmost.com/">Website</a> | <a href="https://github.com/docmost/docmost">Source</a></p> https://tedneward.github.io/Research/presentation/electron/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/electron/index.html <p><a href="https://www.electronjs.org/">Website</a> | <a href="https://www.electronjs.org/docs">Docs</a> | <a href="https://github.com/electron">Source</a></p> <p><a href="https://github.com/electron/electron-quick-start">Quick start app</a>, commonly used as the start scaffolding for a new project.</p> https://tedneward.github.io/Research/presentation/epicstack/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/epicstack/index.html <p><a href="https://www.epicweb.dev/epic-stack">Website</a> | <a href="https://github.com/epicweb-dev/epic-stack">Source</a></p> <p>The primary goal of the Epic Stack is to help your team get over analysis paralysis by giving you solid opinions for technologies to use to build your web application. The Epic Stack is really two things:</p> <ul> <li>A project starter</li> <li>A reference implementation</li> </ul> <p>For those of you starting a new project, you can use the <code>create-remix CLI</code> custom stacks feature to bootstrap your app with <code>npx create-remix@latest --typescript --install --template epicweb-dev/epic-stack</code> and you’ll be set.</p> <p>With that context, here are a few things you get today:</p> <ul> <li>Remix is the Web Framework of choice</li> <li>Fly app deployment with Docker</li> <li>Multi-region, distributed, production-ready SQLite Database with LiteFS.</li> <li>Healthcheck endpoint for Fly backups region fallbacks</li> <li>Grafana Dashboards of the running app</li> <li>GitHub Actions with testing and deploy on merge for both production and staging environments</li> <li>Email/Password Authentication with cookie-based sessions</li> <li>Transaction email with Mailgun and forgot password/password reset support.</li> <li>Progressively Enhanced and fully type safe forms with Conform</li> <li>Database ORM with Prisma</li> <li>Role-based User Permissions.</li> <li>Custom built image hosting</li> <li>Caching via cachified: Both in-memory and SQLite-based (with better-sqlite3)</li> <li>Styling with Tailwind</li> <li>An excellent, customizable component library with Radix UI</li> <li>End-to-end testing with Playwright</li> <li>Local third party request mocking with MSW</li> <li>Unit testing with Vitest and Testing Library with pre-configured Test Database</li> <li>Code formatting with Prettier</li> <li>Linting with ESLint</li> <li>Static Types with TypeScript</li> <li>Runtime schema validation with zod</li> </ul> https://tedneward.github.io/Research/presentation/fenster/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fenster/index.html <p><a href="https://github.com/zserge/fenster">Source</a></p> https://tedneward.github.io/Research/presentation/fonoster/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fonoster/index.html <p><a href="https://fonoster.com/">Website</a> | <a href="https://github.com/fonoster/fonoster">Source</a></p> https://tedneward.github.io/Research/presentation/fractalide/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/fractalide/index.html <p><a href="https://github.com/fractalide/fractalide">Source</a></p> https://tedneward.github.io/Research/presentation/gluon/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/gluon/index.html <p><a href="https://gluonhq.com/">Website</a> | <a href="https://github.com/gluonhq">Source</a> | <a href="https://docs.gluonhq.com/#_introduction">Docs</a></p> <p>Gluon provides an easy and modern approach for developing Java Client applications. These applications can run on the JVM or can be converted to a platform specific native-images which have lighting fast startup and takes a fraction of space. Moreover, applications can also be targeted to Android, iOS, and embedded apart from all the desktop environments.</p> <p>Gluon provides all the tools necessary to build these applications including, but not limited to, build tools, IDE plugins, UI library, cloud connectivity, data-binding etc.</p> <h2>Reading</h2> <h3>Articles</h3> <ul> <li><a href="https://foojay.io/today/creating-mobile-apps-with-javafx-part-1/">https://foojay.io/today/creating-mobile-apps-with-javafx-part-1/</a></li> <li><a href="https://foojay.io/today/creating-mobile-apps-with-javafx-part-2/">https://foojay.io/today/creating-mobile-apps-with-javafx-part-2/</a></li> <li><a href="https://foojay.io/today/creating-mobile-apps-with-javafx-part-3/">https://foojay.io/today/creating-mobile-apps-with-javafx-part-3/</a></li> </ul> https://tedneward.github.io/Research/presentation/guizero/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/guizero/index.html <p><a href="https://lawsie.github.io/guizero/">Website</a> | <a href="https://github.com/lawsie/guizero">Source</a> | <a href="">Book</a></p> <p>The aim of guizero is to make the process of creating simple GUIs quick, accessible and understandable for new learners.</p> <ul> <li>Works with standard Python Tkinter GUI library (and no need to install other libraries)</li> <li>Abstracts away details new learners find difficult to understand (such as Tkinter StringVar() objects)</li> <li>Accessible widget naming system to help new learners to build up a mental model</li> <li>Flexible enough to be used for projects up to A-Level standard, yet accessible to primary school children</li> <li>Comprehensive and accessible documentation with examples</li> <li>Generates helpful additional error messages</li> </ul> <h2>Examples</h2> <p>Hello, world</p> <pre><code>from guizero import App, Text, PushButton app = App(title="guizero") intro = Text(app, text="Have a go with guizero and see what you can create.") ok = PushButton(app, text="Ok") app.display() </code></pre> <p>Get some text</p> <pre><code>from guizero import App, TextBox app = App() name = TextBox(app, text="Enter your name") app.display() </code></pre> <p>Use a PushButton to display a message when the button is pressed.</p> <pre><code>from guizero import App, TextBox, PushButton, Text def update_text(): label.value = name.value app = App() label = Text(app, text="What's your name?") name = TextBox(app) button = PushButton(app, command=update_text) app.display() </code></pre> <p>Use a Picture object to display an image.</p> <pre><code>from guizero import App, Picture app = App() pic = Picture(app, image="myimage.gif") app.display() </code></pre> <p>Create a picture PushButton with an image like this:</p> <pre><code>from guizero import App, PushButton def do_nothing(): print("A picture button was pressed") app = App() button = PushButton(app, image="button.gif", command=do_nothing) app.display() </code></pre> <p>The image button.gif should be stored in the folder as your program. Alternatively you can provide the path to your image.</p> <p>Have 2 buttons, start and stop with them changing the enabled state of each other.</p> <pre><code>from guizero import App, PushButton def start(): start_button.disable() stop_button.enable() def stop(): start_button.enable() stop_button.disable() app = App() start_button = PushButton(app, command=start, text="start") stop_button = PushButton(app, command=stop, text="stop", enabled=False) app.display() </code></pre> <p>Your app doesn't have to use the standard colors and text, let your user pick the background and text color from 2 combo's.</p> <pre><code> from guizero import App, Combo, Text def update_bg(): app.bg = bg_combo.value def update_text(): app.text_color = text_combo.value colors = ["black", "white", "red", "green", "blue"] app = App() app.bg = "black" app.text_color = "white" title1 = Text(app, text="Background color") bg_combo = Combo(app, options=colors, selected=app.bg, command=update_bg) title2 = Text(app, text="Text color") text_combo = Combo(app, options=colors, selected=app.text_color, command=update_text) app.display() </code></pre> <p>Display an image on the screen with 2 sliders, 1 for height and 1 for width.</p> <pre><code>from guizero import App, Slider, Picture def resize(): picture.width = width.value picture.height = height.value app = App(layout="grid") picture = Picture(app, image="image.gif", grid=[0,1]) width = Slider(app, command=resize, grid=[0,0], start=1, end=picture.width) width.width = picture.width width.value = picture.width height = Slider(app, command=resize, horizontal=False, grid=[1,1], start=1, end=picture.height) height.height = picture.height height.value = picture.height app.display() </code></pre> <p>To be able to react when a user double click's you will need to use events.</p> <pre><code>from guizero import App, Text def double_click(): double_click_me.value = "Thanks" app = App() double_click_me = Text(app, text="Double click me") double_click_me.when_double_clicked = double_click app.display() </code></pre> <p>You can use repeat to periodically update your application. For example, a timer which updates every 1 second (1000 ms).</p> <pre><code>from guizero import App, Text # Update the counter def counter(): text.value = int(text.value) + 1 app = App("Hello world") text = Text(app, text="1") # Schedule call to counter() every 1000ms text.repeat(1000, counter) app.display() </code></pre> <p>You can use a yesno box to check whether someone really wants to exit your app. If they click yes, the app is closed, if not, nothing happens and they can continue with what they were doing.</p> <pre><code>from guizero import App, Text # Ask the user if they really want to close the window def do_this_when_closed(): if app.yesno("Close", "Do you want to quit?"): app.destroy() app = App() title = Text(app, text="blank app") # When the user tries to close the window, run the function do_this_when_closed() app.when_closed = do_this_when_closed app.display() </code></pre> https://tedneward.github.io/Research/presentation/hmpl/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/hmpl/index.html <p><a href="https://hmpl-lang.dev/">Website</a> | <a href="https://github.com/hmpl-language/hmpl">Source</a></p> <p>Alternative to HTMX and Alpine.js.</p> <h4>Example</h4> <pre><code>&lt;div&gt; {{#request src="/api/my-component.html"}} {{#indicator trigger="pending"}} &lt;p&gt;Loading...&lt;/p&gt; {{/indicator}} {{/request}} &lt;/div&gt; </code></pre> <p>Basic usage:</p> <pre><code>import hmpl from "hmpl-js"; const templateFn = hmpl.compile( `&lt;div&gt; &lt;button data-action="increment" id="btn"&gt;Click!&lt;/button&gt; &lt;div&gt;Clicks: {{#request src="/api/clicks" after="click:#btn"}}{{/request}}&lt;/div&gt; &lt;/div&gt;` ); const clicker = templateFn(({ request: { event } }) =&gt; ({ body: JSON.stringify({ action: event.target.getAttribute("data-action") }) })).response; document.querySelector("#app").append(clicker); </code></pre> https://tedneward.github.io/Research/presentation/imgui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/imgui/index.html <p>Library</p> https://tedneward.github.io/Research/presentation/jitsi/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/jitsi/index.html <p><a href="https://jitsi.org/meet">Website</a> | <a href="https://github.com/jitsi/jitsi-meet">Source</a></p> https://tedneward.github.io/Research/platforms/web/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/index/index.html <p>The Web browsers (<a href="/platforms/chrome">Chrome</a>, <a href="/platforms/safari">Safari</a>, IE, <a href="/platforms/firefox">Firefox</a>, etc) form a platform in of themselves, given how much is targeted specifically to them. That said, there is a great deal of overlap between the browser ecosystem and the <a href="/platforms/nodejs">NodeJS</a> platform, owing to the ECMAScript-based nature of each.</p> <p>Other browsers:</p> <ul> <li><a href="https://browser.kagi.com/">Orion</a> (macOS, iOS/iPadOS)</li> </ul> <h2>Browser insights</h2> <ul> <li><a href="https://taligarsiel.com/Projects/howbrowserswork1.htm">"How browsers work"</a></li> </ul> <h2>Web APIs</h2> <p><a href="https://developer.mozilla.org/en-US/docs/Web/API">MDN list</a>:</p> <ul> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Background_Tasks_API">Background tasks</a>: provides the ability to queue tasks to be executed automatically by the user agent when it determines that there is free time to do so.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Background_Fetch_API">Background Fetch API</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Bluetooth_API">Bluetooth</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API">Broadcast Channel</a>: broadcast messages across windows/frames/etc</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API">Canvas</a>: a means for drawing 2D graphics via JavaScript and the HTML <code>&lt;canvas&gt;</code> element.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Channel_Messaging_API">Channel Messaging API</a>: allows two separate scripts running in different browsing contexts attached to the same document (e.g., two IFrames, or the main document and an IFrame, two documents via a SharedWorker, or two workers) to communicate directly, passing messages between one another through two-way channels (or pipes) with a port at each end.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Console_API">Console</a>: functionality to allow developers to perform debugging tasks, such as logging messages or the values of variables at set points in your code, or timing how long an operation takes to complete.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Credential_Management_API">Credential Management API</a>: enables a website to create, store, and retrieve credentials. A credential is an item which enables a system to make an authentication decision: for example, to decide whether to sign a user into an account.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model">DOM</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API">Fetch</a>: generalized replacement for XMLHttpRequest</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Fullscreen_API">Fullscreen</a>: adds methods to present a specific Element (and its descendants) in full-screen mode, and to exit full-screen mode once it is no longer needed.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Geolocation_API">Geolocation</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/HTML_Drag_and_Drop_API">HTML Drag and Drop</a></li> <li><a href="/storage/indexeddb.html">IndexedDB</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Intersection_Observer_API">Intersection Observer</a>: provides a way to asynchronously observe changes in the intersection of a target element with an ancestor element or with a top-level document's viewport.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Navigation_timing_API">Navigation Timing</a>: measure the performance of a web site.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Page_Visibility_API">Page Visibility</a>: determine whether a page is actually visible to the user</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Push_API">Push API</a>: gives web applications the ability to receive messages pushed to them from a server, whether or not the web app is in the foreground, or even currently loaded, on a user agent. This lets developers deliver asynchronous notifications and updates to users that opt in</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events">Server Sent Events</a>: It's possible for a server to send new data to a web page at any time, by pushing messages to the web page. These incoming messages can be treated as Events + data inside the web page.</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API">Service Workers</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Storage_API">Storage</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Storage_Access_API">Storage Access</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Touch_events">Touch Events</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/URL_API">URL</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Notifications_API">Web Notifications</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API">Web Storage</a></li> <li><a href="../webworkers">Web Workers</a></li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/WebGL_API">WebGL</a> (This should be its own page)</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API">WebRTC</a> (This should be its own page)</li> <li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Websockets_API">Websockets API</a> and <a href="/distribution/websockets.html">here</a></li> </ul> <p>Reamining list:</p> <ul> <li>Barcode Detection API</li> <li>Battery API</li> <li>Beacon</li> <li>CSS Counter Styles</li> <li>CSS Font Loading API</li> <li>CSS Painting API</li> <li>CSS Typed Object Model API</li> <li>CSSOM</li> <li>Clipboard API</li> <li>Contact Picker API</li> <li>Content Index API</li> <li>Encoding API</li> <li>Encrypted Media Extensions</li> <li>EyeDropper API</li> <li>File System Access API</li> <li>File and Directory Entries API</li> <li>Gamepad API</li> <li>HTML DOM</li> <li>HTML Sanitizer API</li> <li>High Resolution Time</li> <li>History API</li> <li>Image Capture API</li> <li>Layout Instability API</li> <li>Long Tasks API</li> <li>Media Capabilities API</li> <li>Media Capture and Streams</li> <li>Media Session API</li> <li>Media Source Extensions</li> <li>MediaStream Recording</li> <li>Network Information API</li> <li>Payment Request API</li> <li>Performance API</li> <li>Performance Timeline API</li> <li>Periodic Background Sync</li> <li>Permissions API</li> <li>Picture-in-Picture API</li> <li>Pointer Events</li> <li>Pointer Lock API</li> <li>Presentation API</li> <li>Proximity Events</li> <li>Resize Observer API</li> <li>Resource Timing API</li> <li>Screen Capture API</li> <li>Screen Orientation API</li> <li>Screen Wake Lock API</li> <li>Sensor API</li> <li>Streams</li> <li>URL Pattern API</li> <li>Vibration API</li> <li>Visual Viewport</li> <li>Web Animations</li> <li>Web Audio API</li> <li>Web Authentication API</li> <li>Web Crypto API</li> <li>Web MIDI API</li> <li>Web Share API</li> <li>Web Speech API</li> <li>WebCodecs API</li> <li>WebHID API</li> <li>WebVR API</li> <li>WebVTT</li> <li>WebXR Device API</li> <li>Window Controls Overlay API</li> <li>XMLHttpRequest</li> </ul> <hr> <h2>Web/browser session replay</h2> <ul> <li><a href="https://openreplay.com/">OpenReplay</a>: <a href="https://github.com/openreplay/openreplay">Source</a></li> <li><a href="https://logrocket.com/">LogRocket</a></li> <li><a href="https://www.fullstory.com/">FullStory</a></li> </ul> https://tedneward.github.io/Research/platforms/webflow/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/webflow/index.html <p><a href="https://webflow.com/">Website</a></p> <p>Provides hosting, visual designer.</p> https://tedneward.github.io/Research/platforms/windows/development/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/development/index.html <p><a href="https://www.randallhyde.com/AssemblyLanguage/Win32Asm/index.html">Win32 Assembly Language</a></p> <hr> <p>Brief bit of Windows lore: <a href="https://reddit.com/r/programming/comments/gnazif/ray_tracing_in_notepadexe_at_30_fps/fr8uy2l">Using Notepad as a debug console for Windows programs</a>:</p> <pre><code class="language-c">static void nlog(char *str, ...) { HWND notepad, edit; va_list ap; char buf[256]; va_start(ap, str); vsprintf(buf, str, ap); va_end(ap); strcat(buf, "\r\n"); notepad = FindWindow(NULL, "Untitled - Notepad"); edit = FindWindowEx(notepad, NULL, "EDIT", NULL); SendMessage(edit, EM_REPLACESEL, TRUE, (LPARAM)buf); } </code></pre> <p>... which is kinda a cool trick.</p> <hr> https://tedneward.github.io/Research/platforms/windows/winslopr/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/winslopr/index.html <p><a href="https://github.com/builtbybel/Winslopr">Source</a></p> <p>59 community-sourced annoyances. Filterable. With fixes.</p> https://tedneward.github.io/Research/platforms/xv6/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/xv6/index.html <p><a href="https://pdos.csail.mit.edu/6.828/2021/xv6.html">Website</a> | <a href="https://github.com/mit-pdos/xv6-riscv.git">Source</a> | <a href="https://pdos.csail.mit.edu/6.828/2021/xv6/book-riscv-rev2.pdf">Book</a> (<a href="https://github.com/mit-pdos/xv6-riscv-book.git">Source</a>)</p> https://tedneward.github.io/Research/presentation/accessibility/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/accessibility/index.html <p><a href="https://wearecolorblind.com/">WeAreColorBlind</a></p> https://tedneward.github.io/Research/presentation/alpinejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/alpinejs/index.html <p><a href="https://alpinejs.dev/">Website</a> | <a href="https://github.com/alpinejs/alpine">Source</a></p> <pre><code>&lt;script src="//unpkg.com/alpinejs" defer&gt;&lt;/script&gt; &lt;div x-data="{ open: false }"&gt; &lt;button @click="open = true"&gt;Expand&lt;/button&gt; &lt;span x-show="open"&gt; Content... &lt;/span&gt; &lt;/div&gt; </code></pre> <p>15 Attributes, 6 Properties, 2 Methods.</p> <h1>Docs: Start Here</h1> <p>Create a blank HTML file somewhere on your computer with a name like: <code>i-love-alpine.html</code></p> <p>Using a text editor, fill the file with these contents:</p> <pre><code class="language-alpine">&lt;html&gt; &lt;head&gt; &lt;script defer src="https://cdn.jsdelivr.net/npm/alpinejs@3.x.x/dist/cdn.min.js"&gt;&lt;/script&gt; &lt;/head&gt; &lt;body&gt; &lt;h1 x-data="{ message: 'I ❤️ Alpine' }" x-text="message"&gt;&lt;/h1&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>Open your file in a web browser, if you see <code>I ❤️ Alpine</code>, you're ready to rumble!</p> <p>Now that you're all set up to play around, let's look at three practical examples as a foundation for teaching you the basics of Alpine. By the end of this exercise, you should be more than equipped to start building stuff on your own. Let's goooooo.</p> <h2>Building a counter</h2> <p>Let's start with a simple "counter" component to demonstrate the basics of state and event listening in Alpine, two core features.</p> <p>Insert the following into the <code>&lt;body&gt;</code> tag:</p> <pre><code class="language-alpine">&lt;div x-data="{ count: 0 }"&gt; &lt;button x-on:click="count++"&gt;Increment&lt;/button&gt; &lt;span x-text="count"&gt;&lt;/span&gt; &lt;/div&gt; </code></pre> <p>Now, you can see with 3 bits of Alpine sprinkled into this HTML, we've created an interactive "counter" component.</p> <p>Let's walk through what's happening briefly:</p> <h3>Declaring data</h3> <pre><code class="language-alpine">&lt;div x-data="{ count: 0 }"&gt; </code></pre> <p>Everything in Alpine starts with an <code>x-data</code> directive. Inside of <code>x-data</code>, in plain JavaScript, you declare an object of data that Alpine will track.</p> <p>Every property inside this object will be made available to other directives inside this HTML element. In addition, when one of these properties changes, everything that relies on it will change as well.</p> <blockquote> <p><code>x-data</code> is required on a parent element for most Alpine directives to work.</p> </blockquote> <h3>Listening for events</h3> <pre><code class="language-alpine">&lt;button x-on:click="count++"&gt;Increment&lt;/button&gt; </code></pre> <p><code>x-on</code> is a directive you can use to listen for any event on an element. We're listening for a <code>click</code> event in this case, so ours looks like <code>x-on:click</code>.</p> <p>You can listen for other events as you'd imagine. For example, listening for a <code>mouseenter</code> event would look like this: <code>x-on:mouseenter</code>.</p> <p>When a <code>click</code> event happens, Alpine will call the associated JavaScript expression, <code>count++</code> in our case. As you can see, we have direct access to data declared in the <code>x-data</code> expression.</p> <blockquote> <p>You will often see <code>@</code> instead of <code>x-on:</code>. This is a shorter, friendlier syntax that many prefer. From now on, this documentation will likely use <code>@</code> instead of <code>x-on:</code>.</p> </blockquote> <h3>Reacting to changes</h3> <pre><code class="language-alpine">&lt;span x-text="count"&gt;&lt;/span&gt; </code></pre> <p><code>x-text</code> is an Alpine directive you can use to set the text content of an element to the result of a JavaScript expression.</p> <p>In this case, we're telling Alpine to always make sure that the contents of this <code>span</code> tag reflect the value of the <code>count</code> property.</p> <p>In case it's not clear, <code>x-text</code>, like most directives accepts a plain JavaScript expression as an argument. So for example, you could instead set its contents to: <code>x-text="count * 2"</code> and the text content of the <code>span</code> will now always be 2 times the value of <code>count</code>.</p> <h2>Building a dropdown</h2> <p>Now that we've seen some basic functionality, let's keep going and look at an important directive in Alpine: <code>x-show</code>, by building a contrived "dropdown" component.</p> <p>Insert the following code into the <code>&lt;body&gt;</code> tag:</p> <pre><code class="language-alpine">&lt;div x-data="{ open: false }"&gt; &lt;button @click="open = ! open"&gt;Toggle&lt;/button&gt; &lt;div x-show="open" @click.outside="open = false"&gt;Contents...&lt;/div&gt; &lt;/div&gt; </code></pre> <p>If you load this component, you should see that the "Contents..." are hidden by default. You can toggle showing them on the page by clicking the "Toggle" button.</p> <p>The <code>x-data</code> and <code>x-on</code> directives should be familiar to you from the previous example, so we'll skip those explanations.</p> <h3>Toggling elements</h3> <pre><code class="language-alpine">&lt;div x-show="open" ...&gt;Contents...&lt;/div&gt; </code></pre> <p><code>x-show</code> is an extremely powerful directive in Alpine that can be used to show and hide a block of HTML on a page based on the result of a JavaScript expression, in our case: <code>open</code>.</p> <p><a name="listening-for-a-click-outside"></a></p> <h3>Listening for a click outside</h3> <pre><code class="language-alpine">&lt;div ... @click.outside="open = false"&gt;Contents...&lt;/div&gt; </code></pre> <p>You'll notice something new in this example: <code>.outside</code>. Many directives in Alpine accept "modifiers" that are chained onto the end of the directive and are separated by periods.</p> <p>In this case, <code>.outside</code> tells Alpine to instead of listening for a click INSIDE the <code>&lt;div&gt;</code>, to listen for the click only if it happens OUTSIDE the <code>&lt;div&gt;</code>.</p> <p>This is a convenience helper built into Alpine because this is a common need and implementing it by hand is annoying and complex.</p> <h2>Building a search input</h2> <p>Let's now build a more complex component and introduce a handful of other directives and patterns.</p> <p>Insert the following code into the <code>&lt;body&gt;</code> tag:</p> <pre><code class="language-alpine">&lt;div x-data="{ search: '', items: ['foo', 'bar', 'baz'], get filteredItems() { return this.items.filter( i =&gt; i.startsWith(this.search) ) } }" &gt; &lt;input x-model="search" placeholder="Search..."&gt; &lt;ul&gt; &lt;template x-for="item in filteredItems" :key="item"&gt; &lt;li x-text="item"&gt;&lt;/li&gt; &lt;/template&gt; &lt;/ul&gt; &lt;/div&gt; </code></pre> <p>By default, all of the "items" (foo, bar, and baz) will be shown on the page, but you can filter them by typing into the text input. As you type, the list of items will change to reflect what you're searching for.</p> <p>Now there's quite a bit happening here, so let's go through this snippet piece by piece.</p> <p><a name="multi-line-formatting"></a></p> <h3>Multi line formatting</h3> <p>The first thing I'd like to point out is that <code>x-data</code> now has a lot more going on in it than before. To make it easier to write and read, we've split it up into multiple lines in our HTML. This is completely optional and we'll talk more in a bit about how to avoid this problem altogether, but for now, we'll keep all of this JavaScript directly in the HTML.</p> <h3>Binding to inputs</h3> <pre><code class="language-alpine">&lt;input x-model="search" placeholder="Search..."&gt; </code></pre> <p>You'll notice a new directive we haven't seen yet: <code>x-model</code>.</p> <p><code>x-model</code> is used to "bind" the value of an input element with a data property: "search" from <code>x-data="{ search: '', ... }"</code> in our case.</p> <p>This means that anytime the value of the input changes, the value of "search" will change to reflect that.</p> <p><code>x-model</code> is capable of much more than this simple example.</p> <p><a name="computed-properties-using-getters"></a></p> <h3>Computed properties using getters</h3> <p>The next bit I'd like to draw your attention to is the <code>items</code> and <code>filteredItems</code> properties from the <code>x-data</code> directive.</p> <pre><code class="language-js">{ ... items: ['foo', 'bar', 'baz'], get filteredItems() { return this.items.filter( i =&gt; i.startsWith(this.search) ) } } </code></pre> <p>The <code>items</code> property should be self-explanatory. Here we are setting the value of <code>items</code> to a JavaScript array of 3 different items (foo, bar, and baz).</p> <p>The interesting part of this snippet is the <code>filteredItems</code> property.</p> <p>Denoted by the <code>get</code> prefix for this property, <code>filteredItems</code> is a "getter" property in this object. This means we can access <code>filteredItems</code> as if it was a normal property in our data object, but when we do, JavaScript will evaluate the provided function under the hood and return the result.</p> <p>It's completely acceptable to forgo the <code>get</code> and just make this a method that you can call from the template, but some prefer the nicer syntax of the getter.</p> <p>Now let's look inside the <code>filteredItems</code> getter and make sure we understand what's going on there:</p> <pre><code class="language-js">return this.items.filter( i =&gt; i.startsWith(this.search) ) </code></pre> <p>This is all plain JavaScript. We are first getting the array of items (foo, bar, and baz) and filtering them using the provided callback: <code>i =&gt; i.startsWith(this.search)</code>.</p> <p>By passing in this callback to <code>filter</code>, we are telling JavaScript to only return the items that start with the string: <code>this.search</code>, which like we saw with <code>x-model</code> will always reflect the value of the input.</p> <p>You may notice that up until now, we haven't had to use <code>this.</code> to reference properties. However, because we are working directly inside the <code>x-data</code> object, we must reference any properties using <code>this.[property]</code> instead of simply <code>[property]</code>.</p> <p>Because Alpine is a "reactive" framework. Any time the value of <code>this.search</code> changes, parts of the template that use <code>filteredItems</code> will automatically be updated.</p> <p><a name="looping-elements"></a></p> <h3>Looping elements</h3> <p>Now that we understand the data part of our component, let's understand what's happening in the template that allows us to loop through <code>filteredItems</code> on the page.</p> <pre><code class="language-alpine">&lt;ul&gt; &lt;template x-for="item in filteredItems"&gt; &lt;li x-text="item"&gt;&lt;/li&gt; &lt;/template&gt; &lt;/ul&gt; </code></pre> <p>The first thing to notice here is the <code>x-for</code> directive. <code>x-for</code> expressions take the following form: <code>[item] in [items]</code> where [items] is any array of data, and [item] is the name of the variable that will be assigned to an iteration inside the loop.</p> <p>Also notice that <code>x-for</code> is declared on a <code>&lt;template&gt;</code> element and not directly on the <code>&lt;li&gt;</code>. This is a requirement of using <code>x-for</code>. It allows Alpine to leverage the existing behavior of <code>&lt;template&gt;</code> tags in the browser to its advantage.</p> <p>Now any element inside the <code>&lt;template&gt;</code> tag will be repeated for every item inside <code>filteredItems</code> and all expressions evaluated inside the loop will have direct access to the iteration variable (<code>item</code> in this case).</p> https://tedneward.github.io/Research/presentation/apperyio/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/apperyio/index.html <p>"Appery.io is a low-code app development platform that makes it easy to create hybrid mobile apps, web apps and PWAs."</p> <p>"Use integrated backend services such as cloud database, server-side scripting, authentication, push notifications and more. Instantly import and use these services in the app builder. We offer scalable, reliable and secure infrastructure. Focus on implementing your idea and we will take care of the rest."</p> <p><a href="https://appery.io/">Website</a></p> https://tedneward.github.io/Research/presentation/avalonia/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/avalonia/index.html <p><a href="https://avaloniaui.net/">Website</a> | <a href="https://github.com/AvaloniaUI/Avalonia">Source</a></p> <h2>Articles</h2> <ul> <li><a href="https://thenewstack.io/avalonia-an-open-source-option-for-cross-platform-ui-work/">"Avalonia: An Open-Source Option for Cross-Platform UI Work"</a></li> </ul> https://tedneward.github.io/Research/presentation/blazor/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/blazor/index.html <h2>Books</h2> <ul> <li><a href="https://www.telerik.com/campaigns/blazor/wp-beginners-guide-ebook">Blazor: A Beginner's Guide</a> - Ed Charbeneau (PDF) (email address <em>requested</em>, not required)</li> <li><a href="https://dotnet.microsoft.com/download/e-book/blazor-for-web-forms-devs/pdf">Blazor for ASP.NET Web Forms Developers</a> - Daniel Roth, Jeff Fritz, Taylor Southwick (PDF)</li> </ul> https://tedneward.github.io/Research/presentation/cegui/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/cegui/index.html <p><a href="http://cegui.org.uk/">Website</a> | <a href="https://github.com/cegui/cegui">Source</a></p> https://tedneward.github.io/Research/presentation/console/dflat/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/console/dflat/index.html <p>Several sources:</p> <ul> <li><a href="https://github.com/alexfru/dflat20">Source</a></li> <li><a href="https://github.com/the-grue/DFlat">Source</a></li> </ul> <p>He also built a C interpreter using it:</p> <ul> <li><a href="https://github.com/the-grue/Quincy">Source</a></li> </ul> https://tedneward.github.io/Research/platforms/web/chromium-os/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/chromium-os/index.html <p><a href="https://www.chromium.org/chromium-os/">Website</a> | <a href="https://www.chromium.org/chromium-os/quick-start-guide/">Source</a></p> https://tedneward.github.io/Research/platforms/web/safari/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/safari/index.html <p><a href="https://developer.apple.com/safari/resources/">Website</a></p> https://tedneward.github.io/Research/platforms/webiny/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/webiny/index.html <p><a href="https://www.webiny.com/">Website</a> | <a href="https://github.com/webiny/webiny-js">Source</a></p> https://tedneward.github.io/Research/platforms/windows/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/index/index.html <h2>Reading</h2> <ul> <li><a href="https://www.zdnet.com/article/how-to-upgrade-your-incompatible-windows-10-pc-to-windows-11-now/">"How to upgrade your Incompatible Windows 10 PC to Windows 11 Now"</a></li> <li><a href="https://www.makeuseof.com/windows-11-create-bootable-usb-drive/">"Create a bootable Windows USB"</a></li> </ul> <h2>WSL</h2> <p><a href="https://x410.dev/">XWindows for WSL subsystem</a>: Commercial</p> <p><a href="https://www.jeremymorgan.com/tutorials/linux/customize-wsl-terminal/">How to customize WSL Terminal</a></p> <h2>Development</h2> <p><a href="http://dada.perl.it/shootout/">The Great Win32 Computer Language Shootout</a></p> <h2>Package Managers</h2> <p>UniGetUI:</p> <ul> <li><a href="https://www.xda-developers.com/unigetui-hands-on-ultimate-package-manager/">"UniGetUI is the ultimate package manager for Windows PCs"</a></li> </ul> <p>winget:</p> <ul> <li><a href="https://www.xda-developers.com/how-use-winget-windows-11/">"How to use winget"</a></li> </ul> https://tedneward.github.io/Research/platforms/wine/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wine/index.html <p><a href="https://www.winehq.org/">Website</a></p> <p>Wine (originally an acronym for "Wine Is Not an Emulator") is a compatibility layer capable of running Windows applications on several POSIX-compliant operating systems, such as Linux, macOS, &amp; BSD. Instead of simulating internal Windows logic like a virtual machine or emulator, Wine translates Windows API calls into POSIX calls on-the-fly, eliminating the performance and memory penalties of other methods and allowing you to cleanly integrate Windows applications into your desktop.</p> https://tedneward.github.io/Research/platforms/yaade/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/yaade/index.html <p><a href="https://github.com/EsperoTech/yaade">Source</a></p> <p>Compare to <a href="/platforms/hoppscotch">Hoppscotch</a>.</p> https://tedneward.github.io/Research/presentation/ace/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/ace/index.html <p><a href="https://ace.c9.io/">Website</a> | <a href="https://github.com/ajaxorg/ace">Source</a> | <a href="https://ace.c9.io/build/kitchen-sink.html">Kitchen sink demo</a></p> <ul> <li>Syntax highlighting for over 110 languages (TextMate/Sublime Text.tmlanguage files can be imported)</li> <li>Over 20 themes (TextMate/Sublime Text .tmtheme files can be imported)</li> <li>Automatic indent and outdent</li> <li>An optional command line</li> <li>Handles huge documents (four million lines seems to be the limit!)</li> <li>Fully customizable key bindings including vim and Emacs modes</li> <li>Search and replace with regular expressions</li> <li>Highlight matching parentheses</li> <li>Toggle between soft tabs and real tabs</li> <li>Displays hidden characters</li> <li>Drag and drop text using the mouse</li> <li>Line wrapping</li> <li>Code folding</li> <li>Multiple cursors and selections</li> <li>Live syntax checker (currently JavaScript/CoffeeScript/CSS/XQuery)</li> <li>Cut, copy, and paste functionality</li> </ul> https://tedneward.github.io/Research/presentation/angular/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/angular/index/index.html <p><a href="https://angular.dev/">Website</a> | <a href="https://github.com/angular/angular.js">Source</a> | <a href="https://angular.io/docs">Docs</a> | <a href="https://angular.io/tutorial">Tutorial</a></p> <h2>Reading</h2> <h2>Resources</h2> <p><a href="https://www.forms-angular.org/#/">forms-angular</a> <a href="https://github.com/forms-angular/forms-angular">Source</a>: Form generator for Angular/MEAN</p> https://tedneward.github.io/Research/presentation/arc/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/arc/index.html <p><a href="https://arc.net/">Website</a> | Appears to be closed-source?</p> https://tedneward.github.io/Research/presentation/backbone/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/backbone/index.html <p><a href="https://backbonejs.org/">Website</a></p> https://tedneward.github.io/Research/presentation/caffeine/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/caffeine/index.html <p><a href="https://caffeine.js.org/">Website</a></p> https://tedneward.github.io/Research/presentation/cfml/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/cfml/index.html <p><a href="https://www.adobe.com/products/coldfusion-family.html">Adobe ColdFusion</a> | <a href="https://helpx.adobe.com/coldfusion/developing-applications/the-cfml-programming-language.html">Adobe CFML reference</a></p> <h1>Implementations</h1> <ul> <li><a href="https://www.lucee.org/">Lucee</a> (<a href="https://github.com/lucee">Source</a>): a high performance, open source, ColdFusion / CFML server engine, written in Java.</li> <li><a href="https://github.com/OpenBD">OpenBD</a>: The original open source Java powered GPL CFML runtime engine. (Archived)</li> </ul> https://tedneward.github.io/Research/platforms/web/chromium/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/chromium/index.html <p><a href="https://www.chromium.org/Home/">Chromium</a> | <a href="https://chromium.googlesource.com/chromium/src.git">Source</a></p> https://tedneward.github.io/Research/platforms/web/webvm/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/webvm/index.html <p><a href="https://webvm.io/">Website</a> | <a href="https://github.com/leaningtech/webvm">Source</a></p> https://tedneward.github.io/Research/platforms/windows/cairoshell/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/cairoshell/index.html <p><a href="https://cairoshell.com/">Website</a> | <a href="https://github.com/cairoshell/cairoshell">Source</a></p> https://tedneward.github.io/Research/platforms/windows/internals/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/internals/index.html <h2>Reading</h2> <h3>Books</h3> <ul> <li>Windows Internals, 7th Ed: <a href="https://www.microsoftpressstore.com/store/windows-internals-part-1-system-architecture-processes-9780735684188">Part 1</a> and <a href="https://www.microsoftpressstore.com/store/windows-internals-part-2-9780135462409">Part 2</a> | <a href="https://github.com/zodiacon/WindowsInternals">Source</a> - The Bible on the Windows OS, hands down</li> </ul> <h3>Articles</h3> <h3>Websites</h3> <ul> <li><a href="https://learn.microsoft.com/en-us/sysinternals/">Sysinternals</a> | <a href="https://learn.microsoft.com/en-us/sysinternals/downloads/">Downloads</a>: AccessChk, AccessEnum, AdExplorer, AdInsight, AdRestore, Autologon, Autoruns, BgInfo, BlueScreen, CacheSet, ClockRes, Contig, Coreinfo, Ctrl2Cap, DebugView, Desktops, Disk2vhd, DiskExt, DiskMon, DiskView, Disk Usage (DU), EFSDump, FindLinks, Handle, Hex2dec, Junction, LDMDump, ListDLLs, LiveKd, LoadOrder, LogonSessions, MoveFile, NotMyFault, NTFSInfo, PendMoves, PipeList, PortMon, ProcDump, Process Explorer, Process Monitor, PsExec, PsFile, PsGetSid, PsInfo, PsKill, PsList, PsLoggedOn, PsLogList, PsPasswd, PsPing, PsService, PsShutdown, PsSuspend, PsTools, RAMMap, RDCMan, RegDelNull, RegHide, RegJump, Registry Usage (RU), SDelete, ShareEnum, ShellRunas, Sigcheck, Streams, Strings, Sync, Sysmon, TCPView, VMMap, VolumeID, WhoIs, WinObj, ZoomIt</li> </ul> <hr> <h1>Notes</h1> <h2>Concepts and Tools</h2> <h2>System Architecture</h2> <h2>Processes and Jobs</h2> <h2>Threads</h2> <h2>Memory Management</h2> <h2>I/O System</h2> <h2>Security</h2> <h2>System mechanisms</h2> <h2>Virtualization technologies</h2> <h2>Management, diagnostics, and tracing</h2> <h2>Caching and file systems</h2> <h2>Startup and shutdown</h2> https://tedneward.github.io/Research/platforms/xlang/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/xlang/index.html <p><a href="https://github.com/microsoft/xlang">Github</a></p> https://tedneward.github.io/Research/platforms/yao/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/yao/index.html <p><a href="https://yaoapps.com/en-US">Website</a> | <a href="https://github.com/YaoApp/yao">Source</a> (Golang) | <a href="https://yaoapps.com/en-US/doc/a.Introduction/a.Getting%20Started">Docs</a> | Demos: <a href="https://demo-crm.yaoapps.com">Customer Relationship Management System</a>: A Customer Relationship Management System; <a href="https://demo-wms.yaoapps.com">Intelligent warehouse management system</a>: An example of cloud + edge IoT application, an unattended intelligent warehouse management system that supports face recognition and RFID.</p> <h2>Features</h2> <p>No Code: Using JSON to create database model, write interface and describe interface is really codeless programming.</p> <p>IoT: Yao supports running on ARM devices and can be used in IoT, edge computing, industrial Internet.</p> <p>Business Intelligence: Easy-to-use BI components and functions, empowering business with more possibilities.</p> <p>Processes: Yao has many built-in processes, and a series of processes that can be created by data flow, Javascirpt, and GRPC plugin.</p> <p>Dataflow: Through data flow, developers can process data in any form and support JavaScript.</p> <p>Plugins: The plug-in mechanism based on grpc supports function expansion using Nodejs, Python and other languages.</p> <p>Dashboard: By writing JSON description interface layout to build various CRM, ERP, and other enterprise internal systems.</p> <p>File System Routing: Every file in the apis folder will become an interface that can be called, this will be helpful for large scale project.</p> <p>Query Engine: Built in query engine comparable to es, and JSON query is realized through DSL, which is simple and easy to use.</p> <p>Performance: The performance is far better than that of Java and PHP. It is distributed and can be done with one line of command.</p> <p>Enterprise Security: Permission classification, separation of administrator and employee accounts, API level permission control.</p> <p>Natural Distribution: Generating binary files for various platforms, which is very convenient for users who need to go to the cloud.</p> <p>"Yao allows developers to create web services by processes. Yao is a low-code engine that creates a database model, writes API services, and describes dashboard interface just by JSON for web &amp; hardware, no code, and 10x productivity. Yao is based on the flow-based programming idea, developed in the Go language, and supports multiple ways to expand the data stream processor. This makes Yao extremely versatile, which can replace programming languages ​​in most scenarios, and is 10 times more efficient than traditional programming languages ​​in terms of reusability and coding efficiency; application performance and resource ratio Better than PHP, JAVA and other languages.</p> <p>"Yao has a built-in data management system. By writing JSON to describe the interface layout, 90% of the common interface interaction functions can be realized. It is especially suitable for quickly making various management background, CRM, ERP and other internal enterprise systems. Special interactive functions can also be implemented by writing extension components or HTML pages. The built-in management system is not coupled with Yao, and any front-end technologies such as VUE and React can be used to implement the management interface."</p> <h4>Install</h4> <p>Run the script under terminal: (MacOS/Linux) <code>curl -fsSL https://website.yaoapps.com/install.sh | bash</code></p> <h4>Step 1: Create a project</h4> <p>Create a new project directory, enter the project directory, and run the yao init command to create a blank Yao application.</p> <pre><code>mkdir -p /data/crm # create project directory cd /data/crm # Enter the project directory yao init # run the initializer </code></pre> <p>After the command runs successfully, the app.json file , db, ui , data and other directories will be created</p> <pre><code>├── data # Used to store files generated by the application, such as pictures, PDFs, etc. ├── db # Used to store SQLite database files │ └── yao.db └── ui # The static file server file directory, where custom front-end products can be placed. The files in this directory can be accessed through http://host:port/filename . └── app.json # Application configuration file, used to define the application name, etc. </code></pre> <h4>Step 2: Create the data table</h4> <p>Use the yao migrate command to create the data table, open the command line terminal, run in the project root directory:</p> <pre><code>yao migrate </code></pre> <p>initialization menu</p> <pre><code>yao run flows.setmenu </code></pre> <h4>Step 3: Start the service</h4> <p>Open a command line terminal, run <code>yao start</code> in the project root directory. Open a browser, visit <a href="https://127.0.0.1:5099/xiang/login/admin">https://127.0.0.1:5099/xiang/login/admin</a>, Enter the default username: <a href="mailto:xiang@iqka.com">xiang@iqka.com</a>, password: A123456p+</p> https://tedneward.github.io/Research/presentation/affine/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/affine/index.html <p><a href="https://affine.pro/">Website</a> | <a href="https://github.com/toeverything/AFFiNE">Source</a></p> https://tedneward.github.io/Research/presentation/animejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/animejs/index.html <p><a href="https://animejs.com/">Website</a> | <a href="https://github.com/juliangarnier/anime">Source</a></p> https://tedneward.github.io/Research/presentation/astro/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/astro/index.html <p><a href="https://astro.build/">Website</a> | <a href="https://github.com/withastro/astro">Source</a></p> <p>Claims a "new" frontend architecture called <a href="https://docs.astro.build/en/concepts/islands/">Islands</a>:</p> <p>"The term “component island” was first coined by Etsy’s frontend architect Katie Sylor-Miller in 2019. This idea was then expanded on and documented in this post by Preact creator Jason Miller on August 11, 2020.</p> <blockquote> <p>The general idea of an “Islands” architecture is deceptively simple: render HTML pages on the server, and inject placeholders or slots around highly dynamic regions […] that can then be “hydrated” on the client into small self-contained widgets, reusing their server-rendered initial HTML.<br> — Jason Miller, Creator of Preact</p> </blockquote> <p>"The technique that this architectural pattern builds on is also known as partial or selective hydration.</p> <p>"In contrast, most JavaScript-based web frameworks hydrate &amp; render an entire website as one large JavaScript application (also known as a single-page application, or SPA). SPAs provide simplicity and power but suffer from page-load performance problems due to heavy client-side JavaScript usage.</p> <p>"SPAs have their place, even embedded inside an Astro page. But, SPAs lack the native ability to selectively and strategically hydrate, making them a heavy-handed choice for most projects on the web today.</p> <p>"Astro became popular as the first mainstream JavaScript web framework with selective hydration built-in, using that component islands pattern first coined by Sylor-Miller."</p> https://tedneward.github.io/Research/presentation/baklavajs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/baklavajs/index.html <p><a href="https://baklava.tech/">Website</a> | <a href="https://github.com/newcat/baklavajs">Source</a></p> https://tedneward.github.io/Research/presentation/capacitorjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/capacitorjs/index.html <p><a href="https://capacitorjs.com/">Website</a> | <a href="https://github.com/ionic-team/capacitor">Source</a></p> <p>99% backward compatibility with Cordova.</p> https://tedneward.github.io/Research/presentation/chartjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/chartjs/index.html <p><a href="https://www.chartjs.org/">Website</a> | <a href="https://github.com/chartjs/Chart.js">Source</a></p> https://tedneward.github.io/Research/platforms/web/firefox/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/firefox/index.html <p><a href="https://developer.mozilla.org/en-US/docs/Tools">Website</a></p> https://tedneward.github.io/Research/platforms/web/webworkers/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/webworkers/index.html <p><a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API">MDN Specification</a></p> <h1>Notes</h1> <ul> <li>Creating a worker is done by calling the <code>Worker("path/to/worker/script")</code> constructor.</li> <li>Workers have no access to the page-specific objects/APIs (like <code>document</code>, etc)</li> <li>We can send data back and forth between the main application and our worker script. Calling the <code>postMessage()</code> function can send JSON, Strings, numbers, and/or arrays. <pre><code>//main.js worker.postMessage("hello world"); // Send this to the worker script. //worker.js postMessage("hi from worker"); // Send this back to the main script. </code></pre> </li> <li> <p>We need to add event listeners to both sides to receive the messages above. We can either use onmessage or addEventListener.</p> <pre><code>//worker.js self.addEventListener('message', function(e) { // Send the message back. self.postMessage('You said: ' + e.data); }, false); </code></pre> </li> </ul> <p>Note that we use self for the worker to give a global reference to web workers because it is not a window object.</p> <p>Event listener in the main script:</p> <pre><code> //main.js let worker = new Worker('worker.js'); worker.addEventListener('message', function(e) { // Log the workers message. console.log(e.data); }, false); </code></pre> <h1>MDN Reference: Functions and classes available to Web Workers</h1> <p>The following functions are available to workers:</p> <ul> <li>WorkerGlobalScope.atob()</li> <li>WorkerGlobalScope.btoa()</li> <li>WorkerGlobalScope.clearInterval()</li> <li>WorkerGlobalScope.clearTimeout()</li> <li>WorkerGlobalScope.createImageBitmap()</li> <li>WorkerGlobalScope.dump() Non-standard</li> <li>WorkerGlobalScope.fetch()</li> <li>WorkerGlobalScope.queueMicrotask()</li> <li>WorkerGlobalScope.reportError()</li> <li>WorkerGlobalScope.setInterval()</li> <li>WorkerGlobalScope.setTimeout()</li> <li>WorkerGlobalScope.structuredClone()</li> <li>DedicatedWorkerGlobalScope.postMessage() (dedicated workers only)</li> <li>DedicatedWorkerGlobalScope.requestAnimationFrame() (dedicated workers only)</li> <li>DedicatedWorkerGlobalScope.cancelAnimationFrame() (dedicated workers only)</li> </ul> <p>The following functions are only available to workers:</p> <ul> <li>WorkerGlobalScope.importScripts()</li> </ul> <p>The following Web APIs are available to workers:</p> <ul> <li>Background Fetch API</li> <li>Background Synchronization API</li> <li>Barcode Detection API</li> <li>Broadcast Channel API</li> <li>Canvas API</li> <li>Channel Messaging API</li> <li>Compression Streams API</li> <li>Compute Pressure API</li> <li>Console API</li> <li>Content Index API</li> <li>Cookie Store API (service workers only)</li> <li>CSS Font Loading API</li> <li>Encoding API</li> <li>Fetch API</li> <li>File API</li> <li>File System API</li> <li>Idle Detection API</li> <li>IndexedDB API</li> <li>Media Capabilities API</li> <li>Media Source Extensions API (dedicated workers only)</li> <li>Network Information API</li> <li>Notifications API</li> <li>Payment Handler API</li> <li>Performance API</li> <li>Permissions API</li> <li>Prioritized Task Scheduling API</li> <li>Push API</li> <li>Reporting API</li> <li>Server-Sent Events</li> <li>Service Worker API</li> <li>Streams API</li> <li>Trusted Types API</li> <li>URL API</li> <li>URL Pattern API</li> <li>User-Agent Client Hints API</li> <li>Web Crypto API</li> <li>Web Locks API</li> <li>Web Serial API</li> <li>Web Periodic Background Synchronization API</li> <li>WebCodecs API</li> <li>WebGL API</li> <li>WebGPU API</li> <li>WebHID API (dedicated and service workers only)</li> <li>WebUSB API</li> <li>WebSockets API</li> <li>XMLHttpRequest API</li> </ul> https://tedneward.github.io/Research/platforms/windows/debugging/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/debugging/index.html <p>Debug Tutorial on CodeProject:</p> <ul> <li><a href="https://www.codeproject.com/Articles/6469/Debug-Tutorial-Part-1-Beginning-Debugging-Using-CD">Part 1: CDB and NTSD</a></li> <li><a href="https://www.codeproject.com/Articles/6470/Debug-Tutorial-Part-2-The-Stack">Part 2: the stack</a></li> <li><a href="https://www.codeproject.com/Articles/6489/Debug-Tutorial-Part-3-The-Heap">Part 3: the heap</a></li> <li><a href="https://www.codeproject.com/Articles/6522/Debug-Tutorial-Part-4-Writing-WINDBG-Extensions">Part 4: WINDBG extensions</a></li> <li><a href="https://www.codeproject.com/Articles/6988/Debug-Tutorial-Part-5-Handle-Leaks">Part 5: Handle leaks</a></li> <li><a href="https://www.codeproject.com/Articles/7913/Debug-Tutorial-Part-6-Navigating-The-Kernel-Debugg">Part 6: Kernel debugging</a></li> <li><a href="https://www.codeproject.com/Articles/7919/Debug-Tutorial-Part-7-Locks-and-Synchronization-Ob">Part 7: Locks and sync</a></li> </ul> <p><a href="https://www.codeproject.com/Articles/6084/Windows-Debuggers-Part-1-A-WinDbg-Tutorial">Windows Debugger WinDbg Tutorial</a></p> https://tedneward.github.io/Research/platforms/windows/seleen/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/windows/seleen/index.html <p><a href="https://seelen.io/">Website</a> | <a href="https://github.com/eythaann/seelen-ui">Source</a></p> https://tedneward.github.io/Research/platforms/xoom/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/xoom/index.html <p><a href="https://vlingo.io/">Website</a> | <a href="https://github.com/vlingo">Source</a> | <a href="https://docs.vlingo.io/">Docs</a></p> <p>Collection of libraries and a source-code-generator (XOOM Designer).</p> https://tedneward.github.io/Research/presentation/960/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/960/index.html <p><a href="https://960.gs/">Website</a> | <a href="http://github.com/nathansmith/960-grid-system/">Source</a></p> https://tedneward.github.io/Research/presentation/aframe/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/aframe/index.html <p><a href="https://aframe.io">Website</a></p> https://tedneward.github.io/Research/presentation/antdesign/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/antdesign/index.html <p><a href="https://ant.design/">Website</a> | <a href="https://github.com/ant-design/ant-design/">Source</a></p> <p>Appears to have components for <a href="https://ng.ant.design/">Angular</a>, <a href="https://ant.design/docs/react/introduce">React</a>, and <a href="https://vue.ant.design/">Vue</a>.</p> https://tedneward.github.io/Research/presentation/aurelia/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/aurelia/index.html <p><a href="https://aurelia.io/">Website</a> | <a href="https://github.com/aurelia">Source</a></p> https://tedneward.github.io/Research/presentation/bce/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/bce/index.html <p><a href="https://bce.design/">Website</a> | <a href="https://github.com/adamBien/bce.design">Source</a></p> <p>Quickstarter and sample application for building non-trivial web applications with minimal tooling, essential dependencies, high productivity, and no migrations.</p> <p>Built on web standards and browser APIs - no framework lock-in, just native Web Components, ES modules, and modern JavaScript</p> https://tedneward.github.io/Research/presentation/cappuccino/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/cappuccino/index.html <p><a href="https://www.cappuccino.dev/">Website</a> | <a href="https://github.com/cappuccino/cappuccino">Source</a></p> <blockquote> <p>Cappuccino, a open-source web application framework in Objective-J (a superset of JavaScript which is transpiled for deployment but can also run in the browser directly), enhances app development by faithfully implementing the NeXTSTEP/Apple Cocoa® APIs for browsers as web browser restrictions permit. Leveraging this well-established architecture, Cappuccino facilitates scalable application development for those usage cases requiring developer productivity, reliability and sophisticated interaction support. Resulting applications are served from any web server and deployed to any modern web browser - without plugins or extensions of any kind.</p> </blockquote> https://tedneward.github.io/Research/presentation/codemirror/index.html Mon, 13 Apr 2026 14:04:55 +0000 presentation/codemirror/index.html <p><a href="https://codemirror.net/">Website</a> | <a href="https://github.com/codemirror/dev/">Source</a></p> https://tedneward.github.io/Research/platforms/wasm/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wasm/index/index.html <p><a href="https://webassembly.org/">Website</a> | <a href="https://bytecodealliance.org/">Bytecode Alliance</a></p> <p><a href="https://webassembly.github.io/spec/">Specifications page</a> (<a href="https://github.com/WebAssembly/spec/">Source</a>):</p> <ul> <li><a href="https://webassembly.github.io/spec/core/_download/WebAssembly.pdf">Core</a></li> <li><a href="https://webassembly.github.io/spec/js-api/index.html">JavaScript Embedding</a></li> <li><a href="https://webassembly.github.io/spec/web-api/index.html">Web Embedding</a></li> <li><a href="https://github.com/WebAssembly/proposals">Proposals</a></li> <li><a href="https://component-model.bytecodealliance.org/introduction.html">Component model</a> | <a href="https://github.com/WebAssembly/component-model">Source</a> <ul> <li>IDL (WIT: Wasm Interface Type): <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md">https://github.com/WebAssembly/component-model/blob/main/design/mvp/WIT.md</a> (reproduced below)</li> <li>text format: <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md">https://github.com/WebAssembly/component-model/blob/main/design/mvp/Explainer.md</a></li> <li>binary format: <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/Binary.md">https://github.com/WebAssembly/component-model/blob/main/design/mvp/Binary.md</a></li> <li>concurrency model: <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md">https://github.com/WebAssembly/component-model/blob/main/design/mvp/Async.md</a></li> <li>canonical ABI: <a href="https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md">https://github.com/WebAssembly/component-model/blob/main/design/mvp/CanonicalABI.md</a></li> </ul> </li> <li> <p><a href="https://wasi.dev/">WebAssembly System Interface</a> | <a href="https://github.com/WebAssembly/WASI/tree/main">Source</a></p> </li> <li><a href="https://github.com/CommonWA/cwa-spec">Common WebAssembly</a></li> <li>Announcements <a href="https://webassembly.org/news/2025-03-20-wasm-2.0/">2.0</a>, <a href="https://webassembly.org/news/2025-09-17-wasm-3.0/">3.0</a></li> </ul> <h2>Features:</h2> <ul> <li> <p>3.0</p> </li> <li> <p><strong>64-bit address space.</strong> Memories and tables can now be declared to use i64 as their address type instead of just i32. That expands the available address space of Wasm applications from 4 gigabytes to (theoretically) 16 exabytes, to the extent that physical hardware allows. While the web will necessarily keep enforcing certain limits — on the web, a 64-bit memory is limited to 16 gigabytes — the new flexibility is especially interesting for non-web ecosystems using Wasm, as they can support much, much larger applications and data sets now.</p> </li> <li><strong>Multiple memories.</strong> Contrary to popular belief, Wasm applications were always able to use multiple memory objects — and hence multiple address spaces — simultaneously. However, previously that was only possible by declaring and accessing each of them in separate modules. This gap has been closed, a single module can now declare (define or import) multiple memories and directly access them, including directly copying data between them. This finally allows tools like wasm-merge, which perform “static linking” on two or more Wasm modules by merging them into one, to work for all Wasm modules. It also paves the way for new uses of separate address spaces, e.g., for security (separating private data), for buffering, or for instrumentation.</li> <li><strong>Garbage collection.</strong> In addition to expanding the capabilities of raw linear memories, Wasm also adds support for a new (and separate) form of storage that is automatically managed by the Wasm runtime via a garbage collector. Staying true to the spirit of Wasm as a low-level language, Wasm GC is low-level as well: a compiler targeting Wasm can declare the memory layout of its runtime data structures in terms of struct and array types, plus unboxed tagged integers, whose allocation and lifetime is then handled by Wasm. But that’s it. Everything else, such as engineering suitable representations for source-language values, including implementation details like method tables, remains the responsibility of compilers targeting Wasm. There are no built-in object systems, nor closures or other higher-level constructs — which would inevitably be heavily biased towards specific languages. Instead, Wasm only provides the basic building blocks for representing such constructs and focuses purely on the memory management aspect.</li> <li><strong>Typed references.</strong> The GC extension is built upon a substantial extension to the Wasm type system, which now supports much richer forms of references. Reference types can now describe the exact shape of the referenced heap value, avoiding additional runtime checks that would otherwise be needed to ensure safety. This more expressive typing mechanism, including subtyping and type recursion, is also available for function references, making it possible to perform safe indirect function calls without any runtime type or bounds check, through the new call_ref instruction.</li> <li><strong>Tail calls.</strong> Tail calls are a variant of function calls that immediately exit the current function, and thereby avoid taking up additional stack space. Tail calls are an important mechanism that is used in various language implementations both in user-visible ways (e.g., in functional languages) and for internal techniques (e.g., to implement stubs). Wasm tail calls are fully general and work for callees both selected statically (by function index) and dynamically (by reference or table).</li> <li><strong>Exception handling.</strong> Exceptions provide a way to locally abort execution, and are a common feature in modern programming languages. Previously, there was no efficient way to compile exception handling to Wasm, and existing compilers typically resorted to convoluted ways of implementing them by escaping to the host language, e.g., JavaScript. This was neither portable nor efficient. Wasm 3.0 hence provides native exception handling within Wasm. Exceptions are defined by declaring exception tags with associated payload data. As one would expect, an exception can be thrown, and selectively be caught by a surrounding handler, based on its tag. Exception handlers are a new form of block instruction that includes a dispatch list of tag/label pairs or catch-all labels to define where to jump when an exception occurs.</li> <li><strong>Relaxed vector instructions.</strong> Wasm 2.0 added a large set of vector (SIMD) instructions, but due to differences in hardware, some of these instructions have to do extra work on some platforms to achieve the specified semantics. In order to squeeze out maximum performance, Wasm 3.0 introduces “relaxed” variants of these instructions that are allowed to have implementation-dependent behavior in certain edge cases. This behavior must be selected from a pre-specified set of legal choices.</li> <li><strong>Deterministic profile.</strong> To make up for the added semantic fuzziness of relaxed vector instructions, and in order to support settings that demand or need deterministic execution semantics (such as blockchains, or replayable systems), the Wasm standard now specifies a deterministic default behavior for every instruction with otherwise non-deterministic results — currently, this includes floating-point operators and their generated NaN values and the aforementioned relaxed vector instructions. Between platforms choosing to implement this deterministic execution profile, Wasm thereby is fully deterministic, reproducible, and portable.</li> <li><strong>Custom annotation syntax.</strong> Finally, the Wasm text format has been enriched with generic syntax for placing annotations in Wasm source code. Analogous to custom sections in the binary format, these annotations are not assigned any meaning by the Wasm standard itself, and can be chosen to be ignored by implementations. However, they provide a way to represent the information stored in custom sections in human-readable and writable form, and concrete annotations can be specified by downstream standards.</li> </ul> <p>In addition to these core features, embeddings of Wasm into JavaScript benefit from a new extension to the JS API:</p> <ul> <li><strong>JS string builtins.</strong> JavaScript string values can already be passed to Wasm as externrefs. Functions from this new primitive library can be imported into a Wasm module to directly access and manipulate such external string values inside Wasm.</li> </ul> <p>With these new features, Wasm has much better support for compiling high-level programming languages. Enabled by this, we have seen various new languages popping up to target Wasm, such as <a href="https://github.com/google/j2cl/blob/master/docs/getting-started-j2wasm.md">Java</a>, <a href="https://dune.readthedocs.io/en/stable/wasmoo.html">OCaml</a>, <a href="https://www.scala-js.org/doc/project/webassembly.html">Scala</a>, <a href="https://kotlinlang.org/docs/wasm-overview.html">Kotlin</a>, <a href="https://spritely.institute/hoot/">Scheme</a>, or <a href="https://dart.dev/web/wasm">Dart</a>, all of which use the new GC feature. On top of all these goodies, Wasm 3.0 also is the first version of the standard that has been produced with the new <a href="https://webassembly.org/news/2025-03-27-spectec/">SpecTec</a> tool chain. We believe that this makes for an even more reliable specification.</p> <h2>Tools</h2> <ul> <li><a href="https://github.com/WebAssembly/wabt">WebAssembly Binary Toolkit (wabt)</a> | <a href="https://github.com/WebAssembly/binaryen">binaryen</a></li> <li><a href="https://github.com/wasm3/wasm-debug">Wasm Debugger</a>: Direct, source-level WebAssembly debugger</li> <li><a href="https://github.com/bytecodealliance/wit-bindgen/tree/main/crates/wasmlink">wasmlink</a>: a CLI that allows us to statically link a module and its dependencies using <a href="https://github.com/WebAssembly/module-linking">module linking</a> and the <a href="https://github.com/WebAssembly/interface-types/pull/140">Canonical Interface Types ABI</a>.</li> </ul> <h2>Examples</h2> <ul> <li><a href="https://emuchip.com/">Chip-8 Emulator</a> <a href="https://github.com/Timmoth/EmuChip">Source</a>: A lightweight interpreted virtual machine originally designed for 8-bit microcomputers. It features a 4K memory space, 16 8-bit registers, a 64×32 monochrome display, a stack for subroutines, and simple timers for delay and sound. This emulator replicates the CHIP-8 architecture in software, written in C# and compiled into WebAssembly for execution directly in the browser.</li> <li><a href="https://github.com/3dgen/cppwasm-book/tree/master/en">WebAssembly friendly programming with C/C++</a> - Ending, Chai Shushan, Yushih (HTML, <a href="https://github.com/3dgen/cppwasm-book/tree/master/examples">examples</a>)</li> </ul> <h2>Languages</h2> <p><a href="https://github.com/appcypher/awesome-wasm-langs">Awesome WASM languages</a>:</p> <ul> <li> <p>.NET</p> <ul> <li><a href="https://github.com/RyanLamansky/dotnet-webassembly">.NET WebAssembly</a>: Create, read, modify, write and execute WebAssembly (WASM) files from .NET-based applications.</li> <li><a href="https://andrewlock.net/running-dotnet-in-the-browser-without-blazor/">"Running .NET in the browser without Blazor"</a></li> <li><a href="https://github.com/mono/mono/tree/master/sdks/wasm">Mono</a> - an open source implementation of Microsoft's .NET Framework based on the ECMA standards for C# and the Common Language Runtime. For a real-work example, see this repository which contains the <a href="https://github.com/nventive/calculator">Windows 10 calculator</a>. The application is built using standard C++ 11 and C++/CX, with a calculation engine that dates back from 1995. Made by possible with mono via <a href="https://platform.uno/a-piece-of-windows-10-is-now-running-on-webassembly-natively-on-ios-and-android/">Uno Platform</a>.</li> <li><a href="https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor">Blazor</a> - a web UI framework using C#/Razor and HTML, running client-side via WebAssembly. Source is maintained on <a href="https://github.com/dotnet/aspnetcore">ASP.Net Core</a> repo.</li> <li><a href="https://fsbolero.io/">Bolero</a> - Bolero brings Blazor to F# developers with an easy to use Model-View-Update architecture, HTML combinators, hot reloaded templates, type-safe endpoints, advanced routing and remoting capabilities, and more.</li> <li><a href="https://github.com/dotnet/runtimelab/tree/feature/NativeAOT-LLVM">NativeAOT-LLVM</a> - an experimental fork of the CoreCLR .NET runtime that compiles .NET applications into single-file executables, with the primary target being WASM</li> </ul> </li> <li> <p><a href="/languages/ada">Ada</a></p> </li> <li><a href="/languages/wasm/assemblyscript">AssemblyScript</a>: An assembler for WebAssembly.</li> <li><a href="/languages/wasm/astro">Astro</a>: a fun safe language for rapid prototyping and high performance applications.</li> <li><a href="/languages/jvm/ballerina">Ballerina</a></li> <li><a href="/languages/basic">BASIC</a></li> <li><a href="/languages/berry">Berry</a>: an ultra-lightweight dynamically typed embedded scripting language. It's designed for lower-performance embedded devices, fast, multi-paradigm, simple, flexible, and has very small RAM footprint.</li> <li><a href="/languages/esoteric/brainfuck">Brainfuck</a></li> <li><a href="/languages/c">C</a></li> <li><a href="https://github.com/kign/c4wa">c4wa</a></li> <li><a href="/languages/clean">Clean</a> - a general purpose, state-of-the-art, pure and lazy functional programming language designed for making real-world applications.</li> <li><a href="/languages/co">Co</a>: A programming language similar to Go and TypeScript. <a href="https://github.com/rsms/co">main repo</a></li> <li><a href="/languages/cobol">COBOL</a></li> <li><a href="/languages/crystal">Crystal</a></li> <li><a href="/languages/cyber">Cyber</a> - Fast, efficient, and concurrent scripting. Dynamic and gradual types; Concurrency with fibers; Multithreaded; Memory safe; FFI and Embeddable.</li> <li><a href="/languages/d">D</a> - D is a general-purpose programming language with static typing, systems-level access, and C-like syntax.</li> <li><a href="/languages/dart">Dart</a> - An approachable, portable, and productive language for high-quality apps on any platform</li> <li><a href="/languages/ecmascript">Duktape</a>: embeddable JS engine capable of being run in the browser via WebAssembly</li> <li><a href="/languages/eclair">Eclair</a>: Eclair is a minimal, fast Datalog implementation that compiles to LLVM IR and WASM.</li> <li><a href="/languages/eel">Eel</a></li> <li><a href="/languages/elixir">Elixir</a></li> <li><a href="/languages/dotnet/fsharp">F#</a>: F# is a mature, open source, cross-platform, functional-first programming language. It empowers users and organizations to tackle complex computing problems with simple, maintainable and robust code. WebAssembly support is achieved through <a href="https://fsbolero.io/">Bolero</a>, a set of free and open-source libraries and tools built on top of <a href="https://dotnet.microsoft.com/apps/aspnet/web-apps/blazor">Blazor</a>.</li> <li><a href="/languages/faust">Faust</a> - Faust (Functional Audio Stream) is a functional programming language specifically designed for real-time signal processing and synthesis. A distinctive characteristic of Faust is to be fully compiled.</li> <li><a href="/languages/forest">Forest</a> - a functional programming language that compiles to WebAssembly. The main repo contains the compiler and core syntaxes, currently implemented in Haskell.</li> <li><a href="/languages/forth">Forth</a> - an interactive, extensible, imperative, untyped, stack-based programming language.</li> <li><a href="/languages/go">Go</a> - a statically typed compiled language in the tradition of C, with memory safety, garbage collection, structural typing, and CSP-style concurrent programming features added.</li> <li><a href="/languages/grain">Grain</a> - a strongly-typed functional programming language built for the modern web.</li> <li><a href="/languages/haskell">Haskell</a> - a standardized, general-purpose purely functional programming language, with non-strict semantics and strong static typing. It is named after logician Haskell Curry. The latest standard of Haskell is Haskell 2010. As of May 2016, a group is working on the next version, Haskell 2020.</li> <li><a href="/languages/idris">Idris</a> - a general purpose pure functional programming language with dependent types. Dependent types allow types to be predicated on values, meaning that some aspects of a program’s behaviour can be specified precisely in the type. It is compiled, with eager evaluation. Its features are influenced by Haskell and ML.</li> <li><a href="/languages/lisp/janet">Janet</a> - a good system scripting language, or a language to embed in other programs.</li> <li><a href="/languages/jvm/java">Java</a></li> <li><a href="/languages/ecmascript">JavaScript</a> - a high-level, interpreted programming language that conforms to the ECMAScript specification. It is a language that is also characterized as dynamic, weakly typed, prototype-based and multi-paradigm.</li> <li><a href="/languages/julia">Julia</a> - Julia was designed from the beginning for high performance. Julia programs compile to efficient native code for multiple platforms via LLVM.</li> <li><a href="/languages/kcl">KCL</a></li> <li><a href="/languages/jvm/kotlin">Kotlin</a></li> <li><a href="/languages/wasm/kou">Kou</a></li> <li>LabVIEW is a development environment for the G dataflow graphical programming language used for data acquisition, instrument control, and industrial automation. <ul> <li><a href="https://www.ni.com/en/shop/electronic-test-instrumentation/programming-environments-for-electronic-test-and-instrumentation/what-is-g-web-development-software.html">G Web Development Software</a> - A standalone development environment implementing a subset of the G dataflow graphical programming language to create web-based user interfaces for test and measurement applications.</li> <li><a href="https://github.com/ni/vireosdk">Vireo</a> - An open-source runtime capable of running the virtual instrument assembly representation of the G dataflow graphical programming language created by G Web Development Software.</li> </ul> </li> <li> <p><a href="/languages/lisp">Lisp</a> - Lisp (historically LISP) is a family of programming languages with a long history and a distinctive, fully parenthesized prefix notation.</p> </li> <li><a href="/languages/lobster">Lobster</a> - a statically typed language with flow-sensitive type inference and specialization, compile time reference counting (lifetime analysis) that looks a bit like Python. It was originally intended specifically for games.</li> <li><a href="/languages/lox">Lox</a> - a language created by Bob Nystrom, used to teach compilers in the book Crafting Interpreters. It is dynamically typed, and supports classes, closures, and first-class functions.</li> <li><a href="/languages/lua">Lua</a> - a lightweight, multi-paradigm programming language designed primarily for embedded systems and clients. Lua is cross-platform, since the interpreter is written in ANSI C, and has a relatively simple C API.</li> <li><a href="/languages/lys">Lys</a> - a typed functional language that compiles directly to WebAssembly.</li> <li><a href="/languages/moonbit">Moonbit</a></li> <li><a href="/languages/nelua">Nelua</a> - Minimal, simple, efficient, statically typed, compiled, metaprogrammable, safe, and extensible systems programming language with a Lua flavor.</li> <li><a href="/languages/nerd">Nerd</a></li> <li><a href="/languages/never">Never</a></li> <li><a href="/languages/nim">Nim</a></li> <li><a href="/languages/ml/ocaml">Ocaml</a> - the main implementation of the programming language Caml, created by Xavier Leroy, Jérôme Vouillon, Damien Doligez, Didier Rémy, Ascánder Suárez and others in 1996. A member of the ML language family, OCaml extends the core Caml language with object-oriented programming constructs.</li> <li><a href="/languages/pascal">Pascal</a> - Pascal is a general purpose imperative, procedural and object-oriented static typing programming language. The Free Pascal compiler targets many processor architectures, including wasm32; operating systems, including WASI; and embedded platforms.</li> <li><a href="/languages/php">PHP</a> - PHP is a general-purpose scripting language that is especially suited to server-side web development, in which case PHP generally runs on a web server. Any PHP code in a requested file is executed by the PHP runtime, usually to create dynamic web page content or dynamic images used on websites or elsewhere.</li> <li><a href="/languages/plorth">Plorth</a> - Plorth is stack based, concatenative, strongly typed functional scripting language which is easy to embed to applications written in C++. It's inspired by Forth and Factor programming languages.</li> <li><a href="/languages/wasm/poetry">Poetry</a> - Poetry is a poetically dynamic and simple programming language that compiles to WebAssembly. It has a minimalisting syntax akin to CoffeeScript and gives you full control over wasm imports and exports.</li> <li><a href="/languages/python">Python</a></li> <li><a href="/languages/prolog">Prolog</a> - Prolog is a general-purpose logic programming language associated with artificial intelligence and computational linguistics. Prolog has its roots in first-order logic, a formal logic, and unlike many other programming languages, Prolog is intended primarily as a declarative programming language: the program logic is expressed in terms of relations, represented as facts and rules. A computation is initiated by running a query over these relations.</li> <li><a href="/languages/r">R</a> - R is a language and environment for statistical computing and graphics.</li> <li><a href="/languages/rebol">Rebol</a> - Homoiconic ("data is code" and vice-versa) dynamic programming language and data-format (representing data and metadata) language.</li> <li><a href="/languages/rego">Rego</a> - Open Policy Agent (OPA) is an open source, general-purpose policy engine that unifies policy enforcement across the stack. Rego is a high-level declarative policy language purpose-built for expressing policies over complex hierarchical data structures.</li> <li><a href="/languages/ring">Ring</a> - Ring is a Simple, Small, and Flexible practical general-purpose multi-paradigm language. The supported programming paradigms are Imperative, Procedural, Object-Oriented, Functional, Metaprogramming, Declarative programming using nested structures, and Natural programming.</li> <li><a href="/languages/roc">Roc</a> - A fast, friendly, functional language. Compiles to machine code or WASM.</li> <li><a href="/languages/ruby">Ruby</a> - Ruby is an open source interpreted high-level programming language for general-purpose programming. Created by Matz. Ruby has a design philosophy that emphasizes code readability, notably using as few sigils (special chars<code>:.{}%[]&amp;=&gt;;</code>) as possible.</li> <li><a href="/languages/rust">Rust</a> - Rust is a systems programming language sponsored by Mozilla Research, which describes it as a "safe, concurrent, practical language,"supporting functional and imperative-procedural paradigms. Rust is syntactically similar to C++, but its designers intend it to provide better memory safety while maintaining performance.</li> <li><a href="/languages/scheme">Scheme</a> - Scheme is a programming language that supports multiple paradigms, including functional programming and imperative programming, and is one of the two main dialects of Lisp. Unlike Common Lisp, the other main dialect, Scheme follows a minimalist design philosophy specifying a small standard core with powerful tools for language extension.</li> <li>Speedy.js - Speedy.js is a compiler for a well considered, performance pitfalls free subset of JavaScript targeting WebAssembly. Because WebAssembly is statically-typed, the project uses TypeScript as type-checker and to resolve the types of the program symbols. ~<a href="https://github.com/MichaReiser/speedy.js">Source</a>~ <code>Unmaintained</code></li> <li><a href="/languages/swift">Swift</a> - Swift is a general-purpose, multi-paradigm, compiled programming language developed by Apple Inc. for iOS, macOS, watchOS, tvOS, Linux, and z/OS.</li> <li><a href="/languages/tcl">Tcl</a> - Tcl (Tool Command Language) is a very powerful but easy to learn dynamic programming language, mature but evolving, highly extensible and suitable for a very wide range of uses.</li> <li><a href="/languages/wasm/theta">Theta</a> - Theta is a modern, general purpose, functional programming language with a strong type system and expressive syntax. It features a modular design and supports pattern matching, function overloading, and other goodies.</li> <li><a href="https://github.com/01alchemist/TurboScript">TurboScript</a> - TurboScript is an experimental programming language for parallel programming for web which compiles to JavaScript (asm.js) and WebAssembly (targeting post-MVP). The syntax is similar to TypeScript and the compiler is open source and written in TypeScript. TurboScript has zero dependencies. <code>Unmaintained</code> since 2018</li> <li><a href="/languages/v">V</a> - V is a statically typed compiled programming language designed for building maintainable software.</li> <li><a href="/languages/virgil">Virgil</a> - A fast and lightweight safe, garbage-collected systems programming language. Its compiler produces optimized, standalone native executables, WebAssembly modules, or JARs for the JVM.</li> <li><a href="/languages/wasm/wa">Wa</a> - Wa is a general-purpose programming language designed for developing robustness and maintainability WebAssembly software. Instead of requiring complex toolchains to set up, you can simply go install it - or run it in a browser.</li> <li><a href="https://github.com/tmcw/wah">Wah</a> - Wah is a slightly higher level language that is a superset of WebAssembly. It aims to make WebAssembly's text format slightly more friendly to humans, without introducing new syntax or datatypes. <code>Unmaintained</code></li> <li><a href="https://github.com/ballercat/walt">WAlt</a> - WAlt is an alternative syntax for WebAssembly text format. It's an experiment for using JavaScript syntax to write to as 'close to the metal' as possible. It's JavaScript with rules. .walt files compile directly to WebAssembly binary format. <code>Unmaintained</code></li> <li><a href="https://github.com/kanaka/wam">Wam</a> - WebAssembly Macro language: Wam syntax is a near superset of wast syntax that is more convenient for human developers to write directly. <code>Unmaintained</code></li> <li><a href="/languages/wasm/wase">Wase</a> - WASE: WebAssembly made easy. Wase is a language, which tries to make WASM easy to write. The language maps closely to WebAssembly, and compiles directly to Wasm bytecode. Has strong typing with type inference.</li> <li><a href="/languages/wonkey">Wonkey</a> - Wonkey is an easy to learn, object-oriented, modern and cross-platform programming language for creating cross-platform video games, highly inspired by the "BlitzBasic" range of languages.</li> <li><a href="/languages/c">xcc</a> - Toy C compiler for x86-64 and wasm - <a href="https://github.com/tyfkda/xcc">Source</a>, <a href="https://tyfkda.github.io/xcc/">Online demo</a>.</li> <li><a href="/languages/zig">Zig</a> - Zig is a general-purpose programming language designed for robustness, optimality, and maintainability. <a href="https://ziglang.org/documentation/master/#WebAssembly">Documentation on WebAssembly</a></li> </ul> https://tedneward.github.io/Research/platforms/web/analytics/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/web/analytics/index.html <h2>Commercial/non-OSS tools</h2> <p><a href="https://analytics.google.com/">Google Analytics</a></p> <h2>Open-source Alternatives</h2> <p><a href="https://www.goatcounter.com/">GoatCounter</a>: <a href="https://github.com/arp242/goatcounter">Source</a></p> <p><a href="https://matomo.org/">Matomo</a>|<a href="https://github.com/matomo-org/matomo">Source</a></p> <p><a href="https://plausible.io/">Plausible</a>|<a href="https://github.com/plausible/analytics">Source</a></p> <p><a href="https://umami.is">Umami</a>|<a href="https://github.com/mikecao/umami">Source</a></p> https://tedneward.github.io/Research/platforms/wasm/tools/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wasm/tools/index.html <h2>Inspecting</h2> <ul> <li><a href="https://wasdk.github.io/wasmcodeexplorer/">WebAssembly Code Explorer</a>: A simple binary explorer with neat binary code highlighting.</li> <li><a href="https://evmar.github.io/weave">Weave</a>: Another simple binary explorer.</li> <li><a href="https://github.com/WebAssembly/binaryen">wasm-opt</a> (part of Binaryen): <ul> <li>Color output of s-expression format: <code>wasm-opt --print test.wasm</code></li> <li>Plot the callgraph using <code>Graphviz</code>: <code>wasm-opt --print-call-graph test.wasm | dot -Tpng -o callgraph.png</code></li> <li>Dump DWARF debug info sections: <code>wasm-opt --dwarfdump test.wasm</code></li> <li>Print function metrics: <code>wasm-opt --func-metrics test.wasm</code></li> </ul> </li> <li> <p><a href="https://github.com/WebAssembly/wabt">wasm-decompile</a> (part of WABT) : <code>wasm-decompile</code> decompiles a wasm binary into readable code. It generates output that tries to look like a "very average programming language" while still staying close to the wasm it represents.</p> </li> <li><a href="https://wwwg.github.io/web-wasmdec/">wasmdec</a> | <a href="https://github.com/wwwg/wasmdec">Source</a>: Converts WebAssembly binaries to C. Similar to <code>wasm2c</code>.</li> <li><a href="https://github.com/WebAssembly/wasp">wasp</a>: Generate callgraphs, CFG and DFG graphs for wasm functions.</li> <li><a href="https://github.com/WebAssembly/wabt">wasm-objdump</a> (part of WABT): Print low-level details about a <code>.wasm</code> binary and each of its sections.</li> <li><a href="https://github.com/fitzgen/wasm-nm">wasm-nm</a>: List the imported, exported, and private function symbols defined within a <code>.wasm</code> binary.</li> </ul> <h2>Static analysis</h2> <ul> <li><strong>Twiggy</strong> | <a href="https://github.com/rustwasm/twiggy">repo</a>: Code size profiler, analyzes a binary's call graph.</li> <li><strong>Manticore</strong> | <a href="https://github.com/trailofbits/manticore">repo</a>, <a href="https://blog.trailofbits.com/2020/01/31/symbolically-executing-webassembly-in-manticore/">article</a>: Symbolic execution of WebAssembly binaries.</li> <li><strong>Octopus</strong> | <a href="https://github.com/pventuzelo/octopus">repo</a>: Security analysis framework for WebAssembly modules and Smart Contracts.</li> <li><strong>Wassail</strong> | <a href="https://github.com/acieroid/wassail">repo</a>: A toolkit to perform both lightweight and heavyweight static analysis of WebAssembly modules.</li> <li><strong>wasm-opcodecnt</strong> | part of <a href="https://github.com/WebAssembly/wabt"><code>WABT</code></a>: Count wasm opcode usage statistics.</li> </ul> <h2>Manipulating (optimization, transformation, instrumentation)</h2> <ul> <li><strong>wasm-opt</strong> | part of <a href="https://github.com/WebAssembly/binaryen"><code>Binaryen</code></a>:</li> <li>Transform binary for asynchronous execution (read more in <a href="https://kripken.github.io/blog/wasm/2019/07/16/asyncify.html">this article</a>): <code>wasm-opt test.wasm --asyncify -O3 -o asyncified.wasm</code></li> <li>Instrument binary for dynamic execution tracing: <code>wasm-opt test.wasm --instrument-memory --instrument-locals --log-execution -o instrumetred.wasm</code></li> <li> <p><strong>wizer</strong> | <a href="https://github.com/bytecodealliance/wizer">repo</a>: Don't wait for your Wasm module to initialize itself, pre-initialize it! Wizer instantiates your WebAssembly module, executes its initialization function, and then snapshots the initialized state out into a new WebAssembly module.</p> </li> <li><strong>wasm2c</strong> | part of <a href="https://github.com/WebAssembly/wabt/blob/main/wasm2c/README.md"><code>WABT</code></a>: Takes a WebAssembly module and produces an equivalent (and runnable) C source and header.</li> <li><strong>wasm-snip</strong> | <a href="https://github.com/rustwasm/wasm-snip">repo</a>: Replaces a WebAssembly function's body with an <code>unreachable</code>.</li> <li><strong>wasmonkey</strong> | <a href="https://github.com/jedisct1/wasmonkey">repo</a>: Magically turns exported WASM functions into imported functions.</li> <li><strong>wasm-meter</strong> | <a href="https://www.npmjs.org/package/wasm-metering">npm</a>, <a href="https://github.com/ewasm/wasm-metering">repo</a>: Injects metering into webassembly binaries. This counts computation time for a given program in units of <code>gas</code> (and allows limiting it).</li> <li><strong>CROW</strong> | <a href="https://github.com/KTH/slumps/tree/master/crow">repo</a>: The Wasm superdiversifier. It takes C source code or LLVM bitcode as input and generates several functionally equivalent, but diverse Wasm binaries.</li> </ul> <h2>Dynamic analysis (tracing, profiling)</h2> <ul> <li><strong>wasm3-strace</strong> | <a href="https://wapm.io/package/vshymanskyy/wasm3">wapm</a>, <a href="https://github.com/wasm3/wasm3">repo</a>: Structured, seamless tracing of arbitrary WebAssembly/WASI execution.</li> <li><strong>Wasabi</strong> | <a href="http://wasabi.software-lab.org/">home</a>, <a href="https://github.com/danleh/wasabi">repo</a>: "WebAssembly analysis using binary instrumentation", a dynamic analysis framework.</li> <li><strong>wasmsign2</strong> | <a href="https://github.com/wasm-signatures/wasmsign2">repo</a>: A tool to add and verify digital signatures to/from WASM binaries.</li> <li><strong>swam</strong> | <a href="https://github.com/satabin/swam/">repo</a>: A WASM interpreter with advanced tracing capabilities.</li> </ul> <h2>Source-level debugging</h2> <ul> <li><strong>wasminspect</strong> | <a href="https://github.com/kateinoigakukun/wasminspect">repo</a>: An interactive and self-contained debugger for WebAssembly/WASI.</li> </ul> <h2>Tool development</h2> <ul> <li><strong>Tool Conventions</strong> | <a href="https://github.com/WebAssembly/tool-conventions">docs</a>: Documents describing conventions useful for coordinating interoperability between wasm-related tools.</li> <li><strong>wasm2json, json2wasm</strong> | <a href="https://www.npmjs.com/package/wasm-json-toolkit">npm</a>, <a href="https://github.com/ewasm/wasm-json-toolkit">repo</a>: A small toolkit for converting wasm binaries into json and back. Incredibly helpful for experimenting and creating your own transformations.</li> <li><strong>walrus</strong> | <a href="https://github.com/rustwasm/walrus">repo</a>: Rust library for performing WebAssembly transformations in a robust and ergonomic fashion.</li> <li><strong>wasp</strong> | <a href="https://github.com/WebAssembly/wasp">repo</a>: C++ library designed to make it easy to work with WebAssembly modules.</li> <li><strong>Binaryen.js</strong> | <a href="https://github.com/AssemblyScript/binaryen.js">repo</a>: A port of Binaryen to the Web, allowing you to generate WebAssembly using JavaScript.</li> <li><strong>wasm-tools</strong> | <a href="https://github.com/bytecodealliance/wasm-tools">repo</a>: Rust tooling for low-level manipulation of WebAssembly modules. <p><code>wasm-smith</code> test case generator is of particular interest.<br> <code>wasm-shrink</code> shrinks a Wasm file while preserving an interesting property (such as triggering a bug).</p> </li> <li> <p><strong>wasm-debug</strong> | <a href="https://github.com/wasmerio/wasm-debug">repo</a>, <a href="https://crates.io/crates/wasm-debug">crate</a>: A runtime-independent Rust library that provides functionality to read, transform, and write DWARF section.</p> </li> <li><strong>witx-codegen</strong> | <a href="https://github.com/jedisct1/witx-codegen">repo</a>: A code generator to access WebAssembly standard APIs from different programming languages. Can also generate documentation.</li> </ul> <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://kripken.github.io/blog/binaryen/2019/06/11/fuzz-reduce-productivity.html">Fuzzers &amp; Reducers as Productivity Tools</a></li> <li>wasm2c: <a href="https://kripken.github.io/blog/wasm/2020/07/27/wasmboxc.html">wasmboxc</a>, <a href="https://petersalomonsen.com/articles/wasm2c/wasm2c.html">wasm2c</a></li> </ul> <h4>Debugging</h4> <ul> <li><a href="https://developers.google.com/web/updates/2020/12/webassembly">https://developers.google.com/web/updates/2020/12/webassembly</a></li> <li><a href="https://hacks.mozilla.org/2019/09/debugging-webassembly-outside-of-the-browser/">https://hacks.mozilla.org/2019/09/debugging-webassembly-outside-of-the-browser/</a></li> <li><a href="https://v8.dev/blog/wasm-decompile">wasm-decompile</a></li> </ul> <h3>Reference</h3> <ul> <li><strong>WebAssembly Opcode Table</strong> | <a href="https://pengowray.github.io/wasm-ops/">docs</a></li> </ul> https://tedneward.github.io/Research/platforms/wasm/wit/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wasm/wit/index.html <p><a href="https://component-model.bytecodealliance.org/design/wit.html">Bytecodealliance Specification</a> (Proposal?)</p> <h1>The <code>wit</code> format</h1> <p>The Wasm Interface Type (WIT) format is an <a href="https://en.wikipedia.org/wiki/Interface_description_language">IDL</a> to provide tooling for the <a href="https://github.com/webassembly/component-model">WebAssembly Component Model</a> in two primary ways:</p> <ul> <li> <p>WIT is a developer-friendly format to describe the imports and exports to a component. It is easy to read and write and provides the foundational basis for producing components from guest languages as well as consuming components in host languages.</p> </li> <li> <p>WIT packages are the basis of sharing types and definitions in an ecosystem of components. Authors can import types from other WIT packages when generating a component, publish a WIT package representing a host embedding, or collaborate on a WIT definition of a shared set of APIs between platforms.</p> </li> </ul> <p>A WIT package is a collection of WIT <a href="#wit-interfaces"><code>interface</code>s</a> and <a href="#wit-worlds"><code>world</code>s</a> defined in files in the same directory that all use the file extension <code>wit</code>, for example <code>foo.wit</code>. Files are encoded as valid utf-8 bytes. Types can be imported between interfaces within a package using unqualified names and additionally from other packages through namespace-and-package-qualified names.</p> <p>This document will go through the purpose of the syntactic constructs of a WIT document, a pseudo-formal <a href="#lexical-structure">grammar specification</a>, and additionally a specification of the <a href="#package-format">package format</a> of a WIT package suitable for distribution.</p> <h2>Package Names</h2> <p>All WIT packages are assigned a <em>package name</em>. Package names look like <code>foo:bar@1.0.0</code> and have three fields:</p> <ul> <li> <p>A <em>namespace field</em>, for example <code>foo</code> in <code>foo:bar</code>. This namespace is intended to disambiguate between registries, top-level organizations, etc. For example WASI interfaces use the <code>wasi</code> namespace.</p> </li> <li> <p>A <em>package field</em>, for example <code>clocks</code> in <code>wasi:clocks</code>. A "package" groups together a set of interfaces and worlds that would otherwise be named with a common prefix.</p> </li> <li> <p>An optional <em>version field</em>, specified as <a href="https://semver.org/">full semver</a>.</p> </li> </ul> <p>🪺 With "nested namespaces and packages", package names are generalized to look like <code>foo:bar:baz/quux</code>, where <code>bar</code> is a nested namespace of <code>foo</code> and <code>quux</code> is a nested package of <code>baz</code>. See the <a href="#package-declaration">package declaration</a> section for more details.</p> <p>Package names are specified at the top of a WIT file via a <code>package</code> declaration:</p> <pre><code class="language-wit">package wasi:clocks; </code></pre> <p>or</p> <pre><code class="language-wit">package wasi:clocks@1.2.0; </code></pre> <p>WIT packages can be defined in a collection of files. At least one of these files must specify a package name. Multiple files can specify the <code>package</code>, though they must all agree on what the package name is.</p> <p>Additionally, many packages can be declared consecutively in one or more files, if the following nested package notation is used:</p> <pre><code class="language-wit">package local:a { interface foo {} } package local:b { interface bar {} } </code></pre> <p>It is worth noting that defining nested packages does not remove the need for the "root" package declaration above. These nested package definitions simply provide the contents of other packages inline so that they don't have to be otherwise resolved via the filesystem or a registry.</p> <p>Package names are used to generate the <a href="Explainer.md#import-and-export-definitions">names of imports and exports</a> in the Component Model's representation of <a href="#wit-interfaces"><code>interface</code>s</a> and <a href="#wit-worlds"><code>world</code>s</a> as described <a href="#package-format">below</a>.</p> <h2>WIT Interfaces</h2> <p>The concept of an "interface" is central in WIT as a collection of <a href="#wit-functions">functions</a> and <a href="#wit-types">types</a>. An interface can be thought of as an instance in the WebAssembly Component Model, for example a unit of functionality imported from the host or implemented by a component for consumption on a host. All functions and types belong to an interface.</p> <p>An example of an interface is:</p> <pre><code class="language-wit">package local:demo; interface host { log: func(msg: string); } </code></pre> <p>represents an interface called <code>host</code> which provides one function, <code>log</code>, which takes a single <code>string</code> argument. If this were imported into a component then it would correspond to:</p> <pre><code class="language-wat">(component (import "local:demo/host" (instance $host (export "log" (func (param "msg" string))) )) ;; ... ) </code></pre> <p>An <code>interface</code> can contain <a href="#wit-packages-and-use"><code>use</code></a> statements, <a href="#wit-types">type</a> definitions, and <a href="#wit-functions">function</a> definitions. For example:</p> <pre><code class="language-wit">package wasi:filesystem; interface types { use wasi:clocks/wall-clock.{datetime}; record stat { ino: u64, size: u64, mtime: datetime, // ... } stat-file: func(path: string) -&gt; result&lt;stat&gt;; } </code></pre> <p>More information about <a href="#wit-packages-and-use"><code>use</code></a> and <a href="#wit-types">types</a> are described below, but this is an example of a collection of items within an <code>interface</code>. All items defined in an <code>interface</code>, including <a href="#wit-packages-and-use"><code>use</code></a> items, are considered as exports from the interface. This means that types can further be used from the interface by other interfaces. An interface has a single namespace which means that none of the defined names can collide.</p> <p>A WIT package can contain any number of interfaces listed at the top-level and in any order. The WIT validator will ensure that all references between interfaces are well-formed and acyclic.</p> <h2>WIT Worlds</h2> <p>WIT packages can contain <code>world</code> definitions at the top-level in addition to <a href="#wit-interfaces"><code>interface</code></a> definitions. A world is a complete description of both imports and exports of a component. A world can be thought of as an equivalent of a <code>component</code> type in the component model. For example this world:</p> <pre><code class="language-wit">package local:demo; world my-world { import host: interface { log: func(param: string); } export run: func(); } </code></pre> <p>can be thought of as this component type:</p> <pre><code class="language-wat">(type $my-world (component (import "host" (instance (export "log" (func (param "param" string))) )) (export "run" (func)) )) </code></pre> <p>Worlds describe a concrete component and are the basis of bindings generation. A guest language will use a <code>world</code> to determine what functions are imported, what they're named, and what functions are exported, in addition to their names.</p> <p>Worlds can contain any number of imports and exports, and can be either a function or an interface.</p> <pre><code class="language-wit">package local:demo; world command { import wasi:filesystem/filesystem; import wasi:random/random; import wasi:clocks/monotonic-clock; // ... export main: func(args: list&lt;string&gt;); } </code></pre> <p>More information about the <code>wasi:random/random</code> syntax is available below in the description of <a href="#wit-packages-and-use"><code>use</code></a>.</p> <p>An imported or exported interface corresponds to an imported or exported instance in the component model. Functions are equivalent to bare component functions. Additionally interfaces can be defined inline with an explicit <a href="Explainer.md#import-and-export-definitions">plain name</a> that avoids the need to have an out-of-line definition.</p> <pre><code class="language-wit">package local:demo; interface out-of-line { the-function: func(); } world your-world { import out-of-line; // ... is roughly equivalent to ... import out-of-line: interface { the-function: func(); } } </code></pre> <p>The plain name of an <code>import</code> or <code>export</code> statement is used as the plain name of the final component <code>import</code> or <code>export</code> definition.</p> <p>In the component model imports to a component either use a plain or interface name, and in WIT this is reflected in the syntax:</p> <pre><code class="language-wit">package local:demo; interface my-interface { // .. } world command { // generates an import of the name `local:demo/my-interface` import my-interface; // generates an import of the name `wasi:filesystem/types` import wasi:filesystem/types; // generates an import of the plain name `foo` import foo: func(); // generates an import of the plain name `bar` import bar: interface { // ... } } </code></pre> <p>Each name must be case-insensitively unique in the scope in which it is declared. In the case of worlds, all imported names are in the same scope, but separate from all the export names, and thus the same name can <em>not</em> be imported twice, but <em>can</em> be both imported and exported.</p> <h3>Union of Worlds with <code>include</code></h3> <p>A World can be created by taking the union of two or more worlds. This operation allows world builders to form larger worlds from smaller worlds.</p> <p>Below is a simple example of a world that includes two other worlds.</p> <pre><code class="language-wit">package local:demo; // definitions of a, b, c, foo, bar, baz are omitted world my-world-a { import a; import b; export c; } world my-world-b { import foo; import bar; export baz; } world union-my-world { include my-world-a; include my-world-b; } </code></pre> <p>The <code>include</code> statement is used to include the imports and exports of another World to the current World. It says that the new World should be able to run all components that target the included worlds and more.</p> <p>The <code>union-my-world</code> World defined above is equivalent to the following World:</p> <pre><code class="language-wit">world union-my-world { import a; import b; export c; import foo; import bar; export baz; } </code></pre> <h3>De-duplication of interfaces</h3> <p>If two worlds share an imported or exported <a href="Explainer.md#import-and-export-definitions">interface name</a>, then the union of the two worlds will only contain one copy of that imported or exported name. For example, the following two worlds <code>union-my-world-a</code> and <code>union-my-world-b</code> are equivalent:</p> <pre><code class="language-wit">package local:demo; world my-world-a { import a1; import b1; } world my-world-b { import a1; import b1; } world union-my-world-a { include my-world-a; include my-world-b; } world union-my-world-b { import a1; import b1; } </code></pre> <h3>Name Conflicts and <code>with</code></h3> <p>When two or more included Worlds have the same name for an import or export with a <em>plain</em> name, automatic de-duplication cannot be used (because the two same-named imports/exports might have different meanings in the different worlds) and thus the conflict has to be resolved manually using the <code>with</code> keyword.</p> <p>The following example shows how to resolve name conflicts where <code>union-my-world-a</code> and <code>union-my-world-b</code> are equivalent:</p> <pre><code class="language-wit">package local:demo; world world-one { import a: func(); } world world-two { import a: func(); } world union-my-world-a { include world-one; include world-two with { a as b } } world union-my-world-b { import a: func(); import b: func(); } </code></pre> <p><code>with</code> cannot be used to rename interface names, however, so the following world would be invalid:</p> <pre><code class="language-wit">package local:demo; interface a { foo: func(); } world world-using-a { import a; } world invalid-union-world { include my-using-a with { a as b } // invalid: 'a', which is short for 'local:demo/a', is an interface name } </code></pre> <h3>A Note on Subtyping</h3> <p>In the future, when <code>optional</code> export is supported, the world author may explicitly mark exports as optional to make a component targeting an included World a subtype of the union World.</p> <p>For now, we are not following the subtyping rules for the <code>include</code> statement. That is, the <code>include</code> statement does not imply any subtyping relationship between the included worlds and the union world.</p> <h2>WIT Packages and <code>use</code></h2> <p>A WIT package represents a unit of distribution that can be published to a registry, for example, and used by other WIT packages. WIT packages are a flat list of interfaces and worlds defined in <code>*.wit</code> files. The current thinking for a convention is that projects will have a <code>wit</code> folder where all <code>wit/*.wit</code> files within describe a single package.</p> <p>The purpose of the <code>use</code> statement is to enable sharing types between interfaces, even if they're defined outside of the current package in a dependency. The <code>use</code> statement can be used both within interfaces and worlds and at the top-level of a WIT file.</p> <h4>Interfaces, worlds, and <code>use</code></h4> <p>A <code>use</code> statement inside of an <code>interface</code> or <code>world</code> block can be used to<br> import types:</p> <pre><code class="language-wit">package local:demo; interface types { enum errno { /* ... */ } type size = u32; } interface my-host-functions { use types.{errno, size}; } </code></pre> <p>The <code>use</code> target, <code>types</code>, is resolved within the scope of the package to an<br> interface, in this case defined prior. Afterwards a list of types are provided<br> as what's going to be imported with the <code>use</code> statement. The interface <code>types</code><br> may textually come either after or before the <code>use</code> directive's interface.<br> Interfaces linked with <code>use</code> must be acyclic.</p> <p>Names imported via <code>use</code> can be renamed as they're imported as well:</p> <pre><code class="language-wit">package local:demo; interface my-host-functions { use types.{errno as my-errno}; } </code></pre> <p>This form of <code>use</code> is using a single identifier as the target of what's being<br> imported, in this case <code>types</code>. The name <code>types</code> is first looked up within the<br> scope of the current file, but it will additionally consult the package's<br> namespace as well. This means that the above syntax still works if the<br> interfaces are defined in sibling files:</p> <pre><code class="language-wit">// types.wit interface types { enum errno { /* ... */ } type size = u32; } // host.wit package local:demo; interface my-host-functions { use types.{errno, size}; } </code></pre> <p>Here the <code>types</code> interface is not defined in <code>host.wit</code> but lookup will find it<br> as it's defined in the same package, just instead in a different file. Since<br> files are not ordered, but type definitions in the Component Model are ordered<br> and acyclic, the WIT parser will perform an implicit topological sort of all<br> parsed WIT definitions to find an acyclic definition order (or produce an error<br> if there is none).</p> <p>When importing or exporting an <a href="#wit-interfaces">interface</a> in a <a href="#wit-worlds">world</a><br> the same syntax is used in <code>import</code> and <code>export</code> directives:</p> <pre><code class="language-wit">// a.wit package local:demo; world my-world { import host; export another-interface; } interface host { // ... } // b.wit interface another-interface { // ... } </code></pre> <p>When referring to an interface, a fully-qualified <a href="Explainer.md#import-and-export-definitions">interface name</a> can be used.<br> For example, in this WIT document:</p> <pre><code class="language-wit">package local:demo; world my-world { import wasi:clocks/monotonic-clock; } </code></pre> <p>The <code>monotonic-clock</code> interface of the <code>wasi:clocks</code> package is being imported.<br> This same syntax can be used in <code>use</code> as well:</p> <pre><code class="language-wit">package local:demo; interface my-interface { use wasi:http/types.{request, response}; } </code></pre> <h4>Top-level <code>use</code></h4> <p>If a package being referred to has a version number, then using the above syntax<br> so far it can get a bit repetitive to be referred to:</p> <pre><code class="language-wit">package local:demo; interface my-interface { use wasi:http/types@1.0.0.{request, response}; } world my-world { import wasi:http/handler@1.0.0; export wasi:http/handler@1.0.0; } </code></pre> <p>To reduce repetition and to possibly help avoid naming conflicts the <code>use</code><br> statement can additionally be used at the top-level of a file to rename<br> interfaces within the scope of the file itself. For example the above could be<br> rewritten as:</p> <pre><code class="language-wit">package local:demo; use wasi:http/types@1.0.0; use wasi:http/handler@1.0.0; interface my-interface { use types.{request, response}; } world my-world { import handler; export handler; } </code></pre> <p>The meaning of this and the previous world are the same, and <code>use</code> is purely a<br> developer convenience for providing smaller names if necessary.</p> <p>The interface referred to by a <code>use</code> is the name that is defined in the current<br> file's scope:</p> <pre><code class="language-wit">package local:demo; use wasi:http/types; // defines the name `types` use wasi:http/handler; // defines the name `handler` </code></pre> <p>Like with interface-level-<code>use</code> the <code>as</code> keyword can be used to rename the<br> inferred name:</p> <pre><code class="language-wit">package local:demo; use wasi:http/types as http-types; use wasi:http/handler as http-handler; </code></pre> <p>Note that these can all be combined to additionally import packages with<br> multiple versions and renaming as different WIT identifiers.</p> <pre><code class="language-wit">package local:demo; use wasi:http/types@1.0.0 as http-types1; use wasi:http/types@2.0.0 as http-types2; // ... </code></pre> <h3>Transitive imports and worlds</h3> <p>A <code>use</code> statement is not implemented by copying type information around but<br> instead retains that it's a reference to a type defined elsewhere. This<br> representation is plumbed all the way through to the final component, meaning<br> that <code>use</code>d types have an impact on the structure of the final generated<br> component.</p> <p>For example this document:</p> <pre><code class="language-wit">package local:demo; interface shared { record metadata { // ... } } world my-world { import host: interface { use shared.{metadata}; get: func() -&gt; metadata; } } </code></pre> <p>would generate this component:</p> <pre><code class="language-wat">(component (import "local:demo/shared" (instance $shared (type $metadata (record (; ... ;))) (export "metadata" (type (eq $metadata))) )) (alias export $shared "metadata" (type $metadata_from_shared)) (import "host" (instance $host (export $metadata_in_host "metadata" (type (eq $metadata_from_shared))) (export "get" (func (result $metadata_in_host))) )) ) </code></pre> <p>Here it can be seen that despite the <code>world</code> only listing <code>host</code> as an import<br> the component additionally imports a <code>local:demo/shared</code> interface. This is due<br> to the fact that the <code>use shared.{ ... }</code> implicitly requires that <code>shared</code> is<br> imported into the component as well.</p> <p>Note that the name <code>"local:demo/shared"</code> here is derived from the name of the<br> <code>interface</code> plus the package name <code>local:demo</code>.</p> <p>For <code>export</code>ed interfaces, any transitively <code>use</code>d interface is assumed to be an<br> import unless it's explicitly listed as an export. For example, here <code>w1</code> is<br> equivalent to <code>w2</code>:</p> <pre><code class="language-wit">interface a { resource r; } interface b { use a.{r}; foo: func() -&gt; r; } world w1 { export b; } world w2 { import a; export b; } </code></pre> <blockquote> <p><strong>Note</strong>: It's planned in the future to have "power user syntax" to configure<br> this on a more fine-grained basis for exports, for example being able to<br> configure that a <code>use</code>'d interface is a particular import or a particular<br> export.</p> </blockquote> <h2>WIT Functions</h2> <p>Functions are defined in an <a href="#wit-interfaces"><code>interface</code></a> or are listed as an<br> <code>import</code> or <code>export</code> from a <a href="#wit-worlds"><code>world</code></a>. Parameters to a function must all<br> be named and have case-insensitively unique names:</p> <pre><code class="language-wit">package local:demo; interface foo { a1: func(); a2: func(x: u32); a3: func(y: u64, z: f32); } </code></pre> <p>Functions can optionally return a type:</p> <pre><code class="language-wit">package local:demo; interface foo { a1: func() -&gt; u32; a2: func() -&gt; string; } </code></pre> <p>Multiple return values can be achieved via <code>tuple</code> or <code>record</code> type:</p> <pre><code class="language-wit">package local:demo; interface foo { record r { a: u32, b: f32 } a1: func() -&gt; r; a2: func() -&gt; tuple&lt;u32, f32&gt;; } </code></pre> <h2>WIT Types</h2> <p>Types in WIT files can only be defined in <a href="#wit-interfaces"><code>interface</code>s</a> at this<br> time. The types supported in WIT is the same set of types supported in the<br> component model itself:</p> <pre><code class="language-wit">package local:demo; interface foo { // "package of named fields" record r { a: u32, b: string, } // values of this type will be one of the specified cases variant human { baby, child(u32), // optional type payload adult, } // similar to `variant`, but no type payloads enum errno { too-big, too-small, too-fast, too-slow, } // a bitflags type flags permissions { read, write, exec, } // type aliases are allowed to primitive types and additionally here are some // examples of other types type t1 = u32; type t2 = tuple&lt;u32, u64&gt;; type t3 = string; type t4 = option&lt;u32&gt;; type t5 = result&lt;_, errno&gt;; // no "ok" type type t6 = result&lt;string&gt;; // no "err" type type t7 = result&lt;char, errno&gt;; // both types specified type t8 = result; // no "ok" or "err" type type t9 = list&lt;string&gt;; type t10 = t9; } </code></pre> <p>The <code>record</code>, <code>variant</code>, <code>enum</code>, and <code>flags</code> types must all have names<br> associated with them. The <code>list</code>, <code>option</code>, <code>result</code>, <code>tuple</code>, and primitive<br> types do not need a name and can be mentioned in any context. This restriction<br> is in place to assist with code generation in all languages to leverage<br> language-builtin types where possible while accommodating types that need to be<br> defined within each language as well.</p> <h2>WIT Identifiers</h2> <p>Identifiers in WIT can be defined with two different forms. The first is the<br> <a href="https://en.wikipedia.org/wiki/Letter_case#Kebab_case">kebab-case</a> <a href="Explainer.md#import-and-export-names"><code>label</code></a> production in the<br> Component Model text format.</p> <pre><code class="language-wit">foo: func(bar: u32); red-green-blue: func(r: u32, g: u32, b: u32); resource XML { ... } parse-XML-document: func(s: string) -&gt; XML; </code></pre> <p>This form can't lexically represent WIT <a href="#keywords">keywords</a>, so the second form is the<br> same syntax with the same restrictions as the first, but prefixed with '%':</p> <pre><code class="language-wit">%foo: func(%bar: u32); %red-green-blue: func(%r: u32, %g: u32, %b: u32); // This form also supports identifiers that would otherwise be keywords. %variant: func(%enum: s32); </code></pre> <h1>Filesystem structure</h1> <p>WIT supports multiple <code>package</code>s and the ability to define a single package<br> across many files, and this section is intended to set a number of conventions<br> for WIT-processing tooling to conform to.</p> <p>This won't go into the specific details of any one particular tool and you<br> should consult tooling-specific documentation for more detailed information<br> about exactly how to configure a WIT parser. This will, however use the Rust<br> guest <code>wit-bindgen</code> crate as an example to have a concrete example to link to,<br> but this is intended to be translatable to other examples and bindings<br> generators as well.</p> <h2>Specifying a "Root Package"</h2> <p>To start out when processing WIT a package needs to be conceptually considered<br> the "root package". This is used down below in <code>world</code> selection and the<br> conventional processing of WIT is intended to currently generally have a package<br> as the "default" for lookups. A root package is specified via a path on the<br> filesystem to either a file or directory. Lookup of dependencies and of this<br> package happen differently depending if it's a file or directory.</p> <h3>Root Package: A File</h3> <p>When the root package is a single file then it means that file contains all WIT<br> that is going to be parsed. No further file discovery on the filesystem will<br> happen and after the file is read then no more filesystem interaction will be<br> happening.</p> <pre><code class="language-rust">wit_bindgen::generate!("./my.wit"); </code></pre> <p>To be a valid WIT file the file being parsed must have a leading <code>package ...;</code><br> statement meaning that it's now the "root package". Dependencies of this package<br> must be specified inline with <code>package ... { ... }</code> blocks when using this<br> format.</p> <p>Some tooling may support the ability to load multiple "roots" which means that<br> the final root is used for <code>world</code> selection and the other roots are used to<br> load dependencies. This can be used when you don't necessarily have full control<br> over filesystem structure and need to load dependencies from a possibly<br> non-standard location.</p> <pre><code class="language-rust">// here `deps.wit` will be available when parsing `my.wit` for dependency // resolution. wit_bindgen::generate!({ path: ["./deps.wit", "./my.wit"], }); </code></pre> <p>Note that specifying a file is not the only option for organizing WIT bindings.<br> Below can be a more maintainable strategy with WIT files separate from each<br> other. A single file can be useful when tooling manages WIT for you, but<br> handwritten WIT may often prefer to use a directory.</p> <h3>Root Package: A Directory</h3> <p>When the root package is a directory then it means the filesystem structure of<br> that directory will be traversed to look for WIT to load. A directory not only<br> supports splitting a single package across multiple files on the filesystem but<br> it also enables having all dependencies located within the directory as well.</p> <pre><code class="language-rust">wit_bindgen::generate!("./wit"); </code></pre> <p>This example will parse the directory <code>./wit</code> and look for WIT files. The<br> parsing process first looks at all <code>*.wit</code> files inside the directory itself.<br> This collection of <code>*.wit</code> files will be combined together to form the "root<br> package". No other files will be considered for the "root package". For example<br> though you could have this filesystem structure.</p> <pre><code class="language-rust">wit/ types.wit world.wit my-interface.wit </code></pre> <p>Here <code>types.wit</code>, <code>world.wit</code>, and <code>my-interface.wit</code> would all be parsed<br> together as a single package.</p> <p>Dependencies in the directory format of the filesystem are specified in a <code>deps</code><br> folder within the root folder. Above for example dependencies would be specified<br> in <code>wit/deps</code>. Dependencies are specified in a flat format where each dependency<br> may itself be a file or a directory, but directories do not have recursive<br> <code>deps</code> folders. The name of files/folders used for organization within a<br> directory are not used during parsing and are purely meant for human-read<br> organization.</p> <p>For example we can extend our above <code>wit/</code> folder like so:</p> <pre><code class="language-rust">wit/ types.wit world.wit my-interface.wit deps/ my-dependency.wit wasi:clocks/ types.wit world.wit wasi:clocks@0.3.0-pre/ types.wit world.wit </code></pre> <p>The name <code>my-dependency</code> in <code>my-dependency.wit</code>, as well as <code>wasi:clocks</code> in<br> <code>wasi:clocks/</code>, is arbitrary. This distinguishes one dependency from another but<br> is only used for uniqueness on the filesystem.</p> <p>All dependencies in <code>deps</code> will be loaded and processed in topological order.<br> The <code>my-dependency.wit</code> file may, for example, depend on <code>wasi:clocks/</code>.<br> Additionally <code>my-dependency.wit</code> may have its own inline <code>package .. { ... }</code><br> blocks too which define packages available for dependency resolution. Any<br> package which is duplicated across dependencies must have the same contents.</p> <h2>Specifying a World</h2> <p>The primary unit of bindings generation for WIT tooling is a <code>world</code> which means<br> that various phases of the process must take a <code>world</code> input. For example when<br> generating guest bindings within a language you'll be specifying a <code>world</code> as<br> the unit of bindings generation. WIT tooling should follow these conventions for<br> selecting a world:</p> <ul> <li>Inputs to world selection are a "root package" (what was parsed from the WIT<br> path specified) as well as an optional world string.</li> <li>If the world string is not present, then the root package must have a single<br> world in it. If so that world is used for bindings generation.</li> <li>If the world string is a WIT identifier, then it specifies the name of a world<br> in the root package to use for bindings generation.</li> <li>If the world string is a WIT path, such as <code>a:b/c</code>, then that is a<br> fully-qualified path which can be used to select a world in the dependencies<br> for bindings generation as well.</li> </ul> <p>If the above heuristics all fail then bindings generation fails and a different<br> combination of arguments must be passed to select a world for bindings<br> generation.</p> <h1>Lexical structure</h1> <p>The <code>wit</code> format is a curly-braced-based format where whitespace is optional (but<br> recommended). A <code>wit</code> document is parsed as a unicode string, and when stored in<br> a file is expected to be encoded as utf-8.</p> <p>Additionally, wit files must not contain any bidirectional override scalar<br> values, control codes other than newline, carriage return, and horizontal tab,<br> or codepoints that Unicode officially deprecates or strongly discourages.</p> <p>The current structure of tokens are:</p> <pre><code class="language-ebnf">token ::= whitespace | operator | keyword | integer | identifier </code></pre> <p>Whitespace and comments are ignored when parsing structures defined elsewhere<br> here.</p> <h3>Whitespace</h3> <p>A <code>whitespace</code> token in <code>wit</code> is a space, a newline, a carriage return, a<br> tab character, or a comment:</p> <pre><code class="language-ebnf">whitespace ::= ' ' | '\n' | '\r' | '\t' | comment </code></pre> <h3>Comments</h3> <p>A <code>comment</code> token in <code>wit</code> is either a line comment preceded with <code>//</code> which<br> ends at the next newline (<code>\n</code>) character or it's a block comment which starts<br> with <code>/*</code> and ends with <code>*/</code>. Note that block comments are allowed to be nested<br> and their delimiters must be balanced</p> <pre><code class="language-ebnf">comment ::= '//' character-that-isnt-a-newline* | '/*' any-unicode-character* '*/' </code></pre> <h3>Operators</h3> <p>There are some common operators in the lexical structure of <code>wit</code> used for<br> various constructs. Note that delimiters such as <code>{</code> and <code>(</code> must all be<br> balanced.</p> <pre><code class="language-ebnf">operator ::= '=' | ',' | ':' | ';' | '(' | ')' | '{' | '}' | '&lt;' | '&gt;' | '*' | '-&gt;' | '/' | '.' | '@' </code></pre> <h3>Keywords</h3> <p>Certain identifiers are reserved for use in WIT documents and cannot be used<br> bare as an identifier. These are used to help parse the format, and the list of<br> keywords is still in flux at this time but the current set is:</p> <pre><code class="language-ebnf">keyword ::= 'as' | 'async' | 'bool' | 'borrow' | 'char' | 'constructor' | 'enum' | 'export' | 'f32' | 'f64' | 'flags' | 'from' | 'func' | 'future' | 'import' | 'include' | 'interface' | 'list' | 'option' | 'own' | 'package' | 'record' | 'resource' | 'result' | 's16' | 's32' | 's64' | 's8' | 'static' | 'stream' | 'string' | 'tuple' | 'type' | 'u16' | 'u32' | 'u64' | 'u8' | 'use' | 'variant' | 'with' | 'world' </code></pre> <h3>Integers</h3> <p>Integers are currently only used for package versions and are a contiguous<br> sequence of digits:</p> <pre><code class="language-ebnf">integer ::= [0-9]+ </code></pre> <h2>Top-level items</h2> <p>A <code>wit</code> document is a sequence of items specified at the top level. These items<br> come one after another and it's recommended to separate them with newlines for<br> readability but this isn't required.</p> <p>Concretely, the structure of a <code>wit</code> file is:</p> <pre><code class="language-ebnf">wit-file ::= (package-decl ';')? (package-items | nested-package-definition)* nested-package-definition ::= package-decl '{' package-items* '}' package-items ::= toplevel-use-item | interface-item | world-item </code></pre> <p>Essentially, these top level items are <a href="#wit-worlds">worlds</a>, <a href="#wit-interfaces">interfaces</a>, <a href="#wit-packages-and-use">use statements</a> and other package definitions.</p> <h3>Feature Gates</h3> <p>Various WIT items can be "gated", to reflect the fact that the item is part of<br> an unstable feature, that the item was added as part of a minor version<br> update and shouldn't be used when targeting an earlier minor version, or that a<br> feature has been deprecated and should no longer be used.</p> <p>For example, the following interface has 4 items, 3 of which are gated:</p> <pre><code class="language-wit">interface foo { a: func(); @since(version = 0.2.1) b: func(); @since(version = 0.2.2) c: func(); @unstable(feature = fancier-foo) d: func(); @since(version = 0.2.0) @deprecated(version = 0.2.2) e: func(); } </code></pre> <p>The <code>@since</code> gate indicates that <code>b</code> and <code>c</code> were added as part of the <code>0.2.1</code><br> and <code>0.2.2</code> releases, resp. Thus, when building a component targeting, e.g.,<br> <code>0.2.1</code>, <code>b</code> can be used, but <code>c</code> cannot. An important expectation set by the<br> <code>@since</code> gate is that, once applied to an item, the item is not modified<br> incompatibly going forward (according to general semantic versioning rules).</p> <p>In contrast, the <code>@unstable</code> gate on <code>d</code> indicates that <code>d</code> is part of the<br> <code>fancier-foo</code> feature that is still under active development and thus <code>d</code> may<br> change type or be removed at any time. An important expectation set by the<br> <code>@unstable</code> gate is that toolchains will not expose <code>@unstable</code> features by<br> default unless explicitly opted-into by the developer.</p> <p>Finally, the <code>@deprecated</code> gate on <code>e</code> indicates that <code>e</code> should no longer be<br> used starting version <code>0.2.2</code>. Both toolchains and host runtimes may warn users<br> if they detect an <code>@deprecated</code> API is being used. A <code>@deprecated</code> gate is<br> required to always be paired up with either a <code>@since</code> or <code>@deprecated</code> gate.</p> <p>Together, these gates support a development flow in which new features start<br> with an <code>@unstable</code> gate while the details are still being hashed out. Then,<br> once the feature is stable (and, in a WASI context, voted upon), the<br> <code>@unstable</code> gate is switched to a <code>@since</code> gate.</p> <h4>Feature gate syntax</h4> <p>The grammar that governs feature gate syntax is:</p> <pre><code class="language-wit">gate ::= gate-item* gate-item ::= unstable-gate | since-gate | deprecated-gate unstable-gate ::= '@unstable' '(' feature-field ')' since-gate ::= '@since' '(' version-field ')' deprecated-gate ::= '@deprecated' '(' version-field ')' feature-field ::= 'feature' '=' id version-field ::= 'version' '=' &lt;valid semver&gt; </code></pre> <h4>Rules for feature gate usage</h4> <p>As part of WIT validation, any item that refers to another gated item must also<br> be compatibly gated. For example, this is an error:</p> <pre><code class="language-wit">interface i { @since(version = 1.0.1) type t1 = u32; type t2 = t1; // error } </code></pre> <p>Additionally, if an item is <em>contained</em> by a gated item, it must also be<br> compatibly gated. For example, this is an error:</p> <pre><code class="language-wit">@since(version = 1.0.2) interface i { foo: func(); // error: no gate @since(version = 1.0.1) bar: func(); // also error: weaker gate } </code></pre> <p>The following rules apply to the use of feature gates:</p> <ul> <li>Either <code>@since</code> <em>or</em> <code>@unstable</code> should be used, but not both (exclusive or).</li> <li>If a package contains a feature gate, it's version must be specified (i.e. <code>namespace:package@x.y.z</code>)</li> </ul> <h4>Scenario: Stabilization of a new feature</h4> <p>This section lays out the basic flow and expected usage of feature gate machinery<br> when stabilizing new features and deprecating old ones.</p> <p>Assume the following WIT package as the initial interface:</p> <pre><code class="language-wit">package examples:fgates-calc@0.1.0; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @since(version = 0.1.0) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <p><strong>First, add new items under an <code>@unstable</code> annotation with a <code>feature</code> specified:</strong></p> <pre><code class="language-wit">package examples:fgates-calc@0.1.1; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @since(version = 0.1.0) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; /// By convention, feature flags should be prefixed with package name to reduce chance of collisions /// /// see: https://github.com/WebAssembly/WASI/blob/main/Contributing.md#filing-changes-to-existing-phase-3-proposals @unstable(feature = fgates-calc-minus) sub: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <p>At this point, consumers of the WIT can enable feature <code>fgates-calc-minus</code> through their relevant tooling and get access to the <code>sub</code> function.</p> <p>Note that, at least until subtyping is relaxed in the Component Model, if we had to <em>add</em> a new case to <code>calc-error</code>, this would be a <em>breaking change</em> and require either a new major version or adding a second, distinct <code>variant</code> definition used by new functions.</p> <p><strong>Second, when the feature is ready to be stabilized, switch to a <code>@since</code> annotation:</strong></p> <pre><code class="language-wit">package examples:fgates-calc@0.1.2; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @since(version = 0.1.0) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; @since(version = 0.1.2) sub: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <h4>Scenario: Deprecation of an existing stable feature</h4> <p>This section lays out the basic flow and expected usage of feature gate machinery when stabilizing a new feature.</p> <p>Assume the following WIT package as the initial interface:</p> <pre><code class="language-wit">package examples:fgates-deprecation@0.1.1; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @since(version = 0.1.0) add-one: func(x: s32) -&gt; result&lt;s32, calc-error&gt;; @since(version = 0.1.1) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <p><strong>First: Add the <code>@deprecated</code> annotation to the relevant item in a new version</strong></p> <pre><code class="language-wit">package examples:fgates-deprecation@0.1.2; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @deprecated(version = 0.1.2) add-one: func(x: s32) -&gt; result&lt;s32, calc-error&gt;; @since(version = 0.1.1) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <p>At this point, tooling consuming this WIT will be able to appropriately alert users to the now-deprecated <code>add-one</code> function.</p> <p><strong>Second: completely remove the deprecated item in some future SemVer-compliant major version</strong></p> <pre><code class="language-wit">package examples:fgates-deprecation@0.2.0; @since(version = 0.1.0) interface calc { @since(version = 0.1.0) variant calc-error { integer-overflow, integer-underflow, unexpected, } @since(version = 0.1.1) add: func(x: s32, y: s32) -&gt; result&lt;s32, calc-error&gt;; } </code></pre> <p>In this new "major" version (this is considered a major version under SemVer 0.X rules) -- the <code>add-one</code> function can be fully removed.</p> <h2>Package declaration</h2> <p>WIT files optionally start with a package declaration which defines the name of<br> the package.</p> <pre><code class="language-ebnf">package-decl ::= 'package' ( id ':' )+ id ( '/' id )* ('@' valid-semver)? </code></pre> <p>The production <code>valid-semver</code> is as defined by<br> <a href="https://semver.org/">Semantic Versioning 2.0</a> and optional.</p> <h2>Item: <code>toplevel-use</code></h2> <p>A <code>use</code> statement at the top-level of a file can be used to bring interfaces<br> into the scope of the current file and/or rename interfaces locally for<br> convenience:</p> <pre><code class="language-ebnf">toplevel-use-item ::= 'use' use-path ('as' id)? ';' use-path ::= id | id ':' id '/' id ('@' valid-semver)? | ( id ':' )+ id ( '/' id )+ ('@' valid-semver)? 🪺 </code></pre> <p>Here <code>use-path</code> is an <a href="Explainer.md#import-and-export-definitions">interface name</a>. The bare form <code>id</code><br> refers to interfaces defined within the current package, and the full form<br> refers to interfaces in package dependencies.</p> <p>The <code>as</code> syntax can be optionally used to specify a name that should be assigned<br> to the interface. Otherwise the name is inferred from <code>use-path</code>.</p> <p>As a future extension, WIT, components and component registries may allow<br> nesting both namespaces and packages, which would then generalize the syntax of<br> <code>use-path</code> as suggested by the 🪺 suffixed rule.</p> <h2>Item: <code>world</code></h2> <p>Worlds define a <a href="Explainer.md#type-definitions"><code>componenttype</code></a> as a collection of imports and exports, all<br> of which can be gated.</p> <p>Concretely, the structure of a world is:</p> <pre><code class="language-ebnf">world-item ::= gate 'world' id '{' world-items* '}' world-items ::= gate world-definition world-definition ::= export-item | import-item | use-item | typedef-item | include-item export-item ::= 'export' id ':' extern-type | 'export' use-path ';' import-item ::= 'import' id ':' extern-type | 'import' use-path ';' extern-type ::= func-type ';' | 'interface' '{' interface-items* '}' </code></pre> <p>Note that worlds can import types and define their own types to be exported<br> from the root of a component and used within functions imported and exported.<br> The <code>interface</code> item here additionally defines the grammar for IDs used to refer<br> to <code>interface</code> items.</p> <h2>Item: <code>include</code></h2> <p>A <code>include</code> statement enables the union of the current world with another world. The structure of an <code>include</code> statement is:</p> <pre><code class="language-wit">include wasi:io/my-world-1 with { a as a1, b as b1 }; include my-world-2; </code></pre> <pre><code class="language-ebnf">include-item ::= 'include' use-path ';' | 'include' use-path 'with' '{' include-names-list '}' include-names-list ::= include-names-item | include-names-list ',' include-names-item include-names-item ::= id 'as' id </code></pre> <h2>Item: <code>interface</code></h2> <p>Interfaces can be defined in a <code>wit</code> file. Interfaces have a name and a<br> sequence of items and functions, all of which can be gated.</p> <p>Specifically interfaces have the structure:</p> <blockquote> <p><strong>Note</strong>: The symbol <code>ε</code>, also known as Epsilon, denotes an empty string.</p> </blockquote> <pre><code class="language-ebnf">interface-item ::= gate 'interface' id '{' interface-items* '}' interface-items ::= gate interface-definition interface-definition ::= typedef-item | use-item | func-item typedef-item ::= resource-item | variant-items | record-item | flags-items | enum-items | type-item func-item ::= id ':' func-type ';' func-type ::= 'async'? 'func' param-list result-list param-list ::= '(' named-type-list ')' result-list ::= ϵ | '-&gt;' ty named-type-list ::= ϵ | named-type ( ',' named-type )* named-type ::= id ':' ty </code></pre> <p>The optional <code>async</code> hint in a WIT function type indicates that the callee<br> is expected to block and thus the caller should emit whatever asynchronous<br> language bindings are appropriate (e.g., in JS, Python, C# or Rust, an <code>async</code><br> WIT function would emit an <code>async</code> JS/Python/C#/Rust function). Because <code>async</code><br> is just a hint and not enforced by the runtime, it is technically possible for<br> a non-<code>async</code> callee to block. In that case, though, it is the <em>callee's</em> fault<br> for any resultant loss of concurrency, not the caller's. Thus, <code>async</code> is<br> primarily intended to document expectations in a way that can be taken<br> advantage of by bindings generators. (For more details, see the <a href="Async.md#sync-and-async-functions">async<br> explainer</a>.)</p> <h2>Item: <code>use</code></h2> <p>A <code>use</code> statement enables importing type or resource definitions from other<br> wit packages or interfaces. The structure of a use statement is:</p> <pre><code class="language-wit">use an-interface.{a, list, of, names} use my:dependency/the-interface.{more, names as foo} </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">use-item ::= 'use' use-path '.' '{' use-names-list '}' ';' use-names-list ::= use-names-item | use-names-item ',' use-names-list? use-names-item ::= id | id 'as' id </code></pre> <p>Note: Here <code>use-names-list?</code> means at least one <code>use-name-list</code> term.</p> <h2>Items: type</h2> <p>There are a number of methods of defining types in a <code>wit</code> package, and all of<br> the types that can be defined in <code>wit</code> are intended to map directly to types in<br> the <a href="https://github.com/WebAssembly/component-model">component model</a>.</p> <h3>Item: <code>type</code> (alias)</h3> <p>A <code>type</code> statement declares a new named type in the <code>wit</code> document. This name can<br> be later referred to when defining items using this type. This construct is<br> similar to a type alias in other languages</p> <pre><code class="language-wit">type my-awesome-u32 = u32; type my-complicated-tuple = tuple&lt;u32, s32, string&gt;; </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">type-item ::= 'type' id '=' ty ';' </code></pre> <h3>Item: <code>record</code> (bag of named fields)</h3> <p>A <code>record</code> statement declares a new named structure with named fields. Records<br> are similar to a <code>struct</code> in many languages. Instances of a <code>record</code> always have<br> their fields defined.</p> <pre><code class="language-wit">record pair { x: u32, y: u32, } record person { name: string, age: u32, has-lego-action-figure: bool, } </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">record-item ::= 'record' id '{' record-fields '}' record-fields ::= record-field | record-field ',' record-fields? record-field ::= id ':' ty </code></pre> <h3>Item: <code>flags</code> (bag-of-bools)</h3> <p>A <code>flags</code> represents a bitset structure with a name for each bit. The <code>flags</code><br> type is represented as a bit flags representation in<br> the canonical ABI.</p> <pre><code class="language-wit">flags properties { lego, marvel-superhero, supervillan, } </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">flags-items ::= 'flags' id '{' flags-fields '}' flags-fields ::= id | id ',' flags-fields? </code></pre> <h3>Item: <code>variant</code> (one of a set of types)</h3> <p>A <code>variant</code> statement defines a new type where instances of the type match<br> exactly one of the variants listed for the type. This is similar to a "sum" type<br> in algebraic datatypes (or an <code>enum</code> in Rust if you're familiar with it).<br> Variants can be thought of as tagged unions as well.</p> <p>Each case of a variant can have an optional type associated with it which is<br> present when values have that particular case's tag.</p> <p>All <code>variant</code> type must have at least one case specified.</p> <pre><code class="language-wit">variant filter { all, none, some(list&lt;string&gt;), } </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">variant-items ::= 'variant' id '{' variant-cases '}' variant-cases ::= variant-case | variant-case ',' variant-cases? variant-case ::= id | id '(' ty ')' </code></pre> <h3>Item: <code>enum</code> (variant but with no payload)</h3> <p>An <code>enum</code> statement defines a new type which is semantically equivalent to a<br> <code>variant</code> where none of the cases have a payload type. This is special-cased,<br> however, to possibly have a different representation in the language ABIs or<br> have different bindings generated in for languages.</p> <pre><code class="language-wit">enum color { red, green, blue, yellow, other, } </code></pre> <p>Specifically the structure of this is:</p> <pre><code class="language-ebnf">enum-items ::= 'enum' id '{' enum-cases '}' enum-cases ::= id | id ',' enum-cases? </code></pre> <h3>Item: <code>resource</code></h3> <p>A <code>resource</code> statement defines a new abstract type for a <em>resource</em>, which is<br> an entity with a lifetime that can only be passed around indirectly via <a href="#handles">handle<br> values</a>. Resource types are used in interfaces to describe things<br> that can't or shouldn't be copied by value.</p> <p>For example, the following Wit defines a resource type and a function that<br> takes and returns a handle to a <code>blob</code>:</p> <pre><code class="language-wit">resource blob; transform: func(blob) -&gt; blob; </code></pre> <p>As syntactic sugar, resource statements can also declare any number of<br> <em>methods</em>, which are functions that implicitly take a <code>self</code> parameter that is<br> a handle. A resource statement can also contain any number of <em>static<br> functions</em>, which do not have an implicit <code>self</code> parameter but are meant to be<br> lexically nested in the scope of the resource type. Lastly, a resource<br> statement can contain at most one <em>constructor</em> function, which is syntactic<br> sugar for a function returning a handle of the containing resource type.</p> <p>For example, the following resource definition:</p> <pre><code class="language-wit">resource blob { constructor(init: list&lt;u8&gt;); write: func(bytes: list&lt;u8&gt;); read: func(n: u32) -&gt; list&lt;u8&gt;; merge: static func(lhs: borrow&lt;blob&gt;, rhs: borrow&lt;blob&gt;) -&gt; blob; } </code></pre> <p>desugars into:</p> <pre><code class="language-wit">resource blob; %[constructor]blob: func(init: list&lt;u8&gt;) -&gt; blob; %[method]blob.write: func(self: borrow&lt;blob&gt;, bytes: list&lt;u8&gt;); %[method]blob.read: func(self: borrow&lt;blob&gt;, n: u32) -&gt; list&lt;u8&gt;; %[static]blob.merge: func(lhs: borrow&lt;blob&gt;, rhs: borrow&lt;blob&gt;) -&gt; blob; </code></pre> <p>These <code>%</code>-prefixed <a href="Explainer.md"><code>name</code>s</a> embed the resource type name so that<br> bindings generators can generate idiomatic syntax for the target language or<br> (for languages like C) fall back to an appropriately-prefixed free function<br> name.</p> <p>When a resource type name is used directly (e.g. when <code>blob</code> is used as the<br> return value of the constructor above), it stands for an "owning" handle<br> that will call the resource's destructor when dropped. When a resource<br> type name is wrapped with <code>borrow&lt;...&gt;</code>, it stands for a "borrowed" handle<br> that will <em>not</em> call the destructor when dropped. As shown above, methods<br> always desugar to a borrowed self parameter whereas constructors always<br> desugar to an owned return value.</p> <p>Specifically, the syntax for a <code>resource</code> definition is:</p> <pre><code class="language-ebnf">resource-item ::= 'resource' id ';' | 'resource' id '{' resource-method* '}' resource-method ::= func-item | id ':' 'static' func-type ';' | 'constructor' param-list ';' </code></pre> <p>The optional <code>async</code> hint on <code>static</code> functions has the same meaning as<br> in a non-<code>static</code> <code>func-item</code>.</p> <p>The syntax for handle types is presented <a href="#handles">below</a>.</p> <h2>Types</h2> <p>As mentioned previously the intention of <code>wit</code> is to allow defining types<br> corresponding to the interface types specification. Many of the top-level items<br> above are introducing new named types but "anonymous" types are also supported,<br> such as built-ins. For example:</p> <pre><code class="language-wit">type number = u32; type fallible-function-result = result&lt;u32, string&gt;; type headers = list&lt;string&gt;; </code></pre> <p>Specifically the following types are available:</p> <pre><code class="language-ebnf">ty ::= 'u8' | 'u16' | 'u32' | 'u64' | 's8' | 's16' | 's32' | 's64' | 'f32' | 'f64' | 'char' | 'bool' | 'string' | tuple | list | option | result | handle | future | stream | id tuple ::= 'tuple' '&lt;' tuple-list '&gt;' tuple-list ::= ty | ty ',' tuple-list? list ::= 'list' '&lt;' ty '&gt;' | 'list' '&lt;' ty ',' uint '&gt;' 🔧 uint ::= [1-9][0-9]* option ::= 'option' '&lt;' ty '&gt;' result ::= 'result' '&lt;' ty ',' ty '&gt;' | 'result' '&lt;' '_' ',' ty '&gt;' | 'result' '&lt;' ty '&gt;' | 'result' future ::= 'future' '&lt;' ty '&gt;' | 'future' stream ::= 'stream' '&lt;' ty '&gt;' | 'stream' </code></pre> <p>The <code>tuple</code> type is semantically equivalent to a <code>record</code> with numerical fields,<br> but it frequently can have language-specific meaning so it's provided as a<br> first-class type.</p> <p>🔧 A <code>list</code> with a fixed length provides the low-level memory representation of a<br> homogeneous <code>tuple</code> of the same length, but with the dynamic indexing of a<br> list. E.g., the following two functions have the same low-level (Core<br> WebAssembly) representation, but will naturally produce different source-level<br> bindings:</p> <pre><code class="language-wit">get-ipv4-address1: func() -&gt; list&lt;u8, 4&gt;; get-ipv4-address2: func() -&gt; tuple&lt;u8, u8, u8, u8&gt;; </code></pre> <p>The <code>option</code> and <code>result</code> types are semantically equivalent to the variants:</p> <pre><code class="language-wit">variant option { none, some(ty), } variant result { ok(ok-ty), err(err-ty), } </code></pre> <p>These types are so frequently used and frequently have language-specific<br> meanings though so they're also provided as first-class types.</p> <p>The <code>future</code> and <code>stream</code> types are described as part of the <a href="Async.md#streams-and-futures">async<br> explainer</a>.</p> <p>Finally the last case of a <code>ty</code> is simply an <code>id</code> which is intended to refer to<br> another type or resource defined in the document. Note that definitions can come<br> through a <code>use</code> statement or they can be defined locally.</p> <h2>Handles</h2> <p>There are two types of handles in Wit: "owned" handles and "borrowed" handles.<br> Owned handles represent the passing of unique ownership of a resource between<br> two components. When the owner of an owned handle drops that handle, the<br> resource is destroyed. In contrast, a borrowed handle represents a temporary<br> loan of a handle from the caller to the callee for the duration of the call.</p> <p>The syntax for handles is:</p> <pre><code class="language-ebnf">handle ::= id | 'borrow' '&lt;' id '&gt;' </code></pre> <p>The <code>id</code> case denotes an owned handle, where <code>id</code> is the name of a preceding<br> <code>resource</code> item. Thus, the "default" way that resources are passed between<br> components is via transfer of unique ownership.</p> <p>The resource method syntax defined above is syntactic sugar that expands into<br> separate function items that take a first parameter named <code>self</code> of type<br> <code>borrow</code>. For example, the compound definition:</p> <pre><code class="language-wit">resource file { read: func(n: u32) -&gt; list&lt;u8&gt;; } </code></pre> <p>is expanded into:</p> <pre><code class="language-wit">resource file %[method]file.read: func(self: borrow&lt;file&gt;, n: u32) -&gt; list&lt;u8&gt;; </code></pre> <p>where <code>%[method]file.read</code> is the desugared name of a method according to the<br> Component Model's definition of <a href="Explainer.md"><code>name</code></a>.</p> <h2>Name resolution</h2> <p>A <code>wit</code> document is resolved after parsing to ensure that all names resolve<br> correctly. For example this is not a valid <code>wit</code> document:</p> <pre><code class="language-wit">type foo = bar; // ERROR: name `bar` not defined </code></pre> <p>Type references primarily happen through the <code>id</code> production of <code>ty</code>.</p> <p>Additionally names in a <code>wit</code> document can only be defined once:</p> <pre><code class="language-wit">type foo = u32; type foo = u64; // ERROR: name `foo` already defined </code></pre> <p>Names do not need to be defined before they're used (unlike in C or C++),<br> it's ok to define a type after it's used:</p> <pre><code class="language-wit">type foo = bar; record bar { age: u32, } </code></pre> <p>Types, however, cannot be recursive:</p> <pre><code class="language-wit">type foo = foo; // ERROR: cannot refer to itself record bar1 { a: bar2, } record bar2 { a: bar1, // ERROR: record cannot refer to itself } </code></pre> <h1>Package Format</h1> <p>Each top-level WIT definition can be compiled into a single canonical<br> Component Model <a href="Explainer.md#type-definitions">type definition</a> that<br> captures the result of performing the type resolution described above. These<br> Component Model types can then be exported by a component along with other<br> sorts of exports, allowing a single component to package both runtime<br> functionality and development-time WIT interfaces. Thus, WIT does not need its<br> own separate package format; WIT can be packaged as a component binary.</p> <p>Using component binaries to package WIT in this manner has several advantages:<br> * We get to reuse the <a href="Binary.md">binary format</a> of components, especially the<br> tricky type bits.<br> * Downstream tooling does not need to replicate the resolution logic nor the<br> resolution environment (directories, registries, paths, arguments, etc) of<br> the WIT package producer; it can reuse the simpler compiled result.<br> * Many aspects of the WIT syntax can evolve over time without breaking<br> downstream tooling, similar to what has happened with the Core WebAssembly<br> WAT text format over time.<br> * When components are published in registries and assigned names (see the<br> discussion of naming in <a href="Explainer.md#import-and-export-definitions">Import and Export Definitions</a>),<br> WIT interfaces and worlds can be published with the same tooling and named<br> using the same <code>namespace:package/export</code> naming scheme.<br> * A single package can both contain an implementation and a collection of<br> <code>interface</code> and <code>world</code> definitions that are imported by that implementation<br> (e.g., an engine component can define and exports its own plugin <code>world</code>).</p> <p>As a first example, the following WIT:</p> <pre><code class="language-wit">package local:demo; interface types { resource file { read: func(off: u32, n: u32) -&gt; list&lt;u8&gt;; write: func(off: u32, bytes: list&lt;u8&gt;); } } interface namespace { use types.{file}; open: func(name: string) -&gt; file; } </code></pre> <p>can be packaged into a component as:</p> <pre><code class="language-wat">(component (type (export "types") (component (export "local:demo/types" (instance (export $file "file" (type (sub resource))) (export "[method]file.read" (func (param "self" (borrow $file)) (param "off" u32) (param "n" u32) (result (list u8)) )) (export "[method]file.write" (func (param "self" (borrow $file)) (param "bytes" (list u8)) )) )) )) (type (export "namespace") (component (import "local:demo/types" (instance $types (export "file" (type (sub resource))) )) (alias export $types "file" (type $file)) (export "local:demo/namespace" (instance (export "open" (func (param "name" string) (result (own $file)))) )) )) ) </code></pre> <p>This example illustrates the basic structure of interfaces:<br> * Each top-level WIT definition (in this example: <code>types</code> and <code>namespace</code>)<br> turns into a type export of the same kebab-name.<br> * Each WIT interface is mapped to a component-type that exports an<br> instance with a fully-qualified <a href="Explainer.md#import-and-export-definitions">interface name</a>(in this example:<br> <code>local:demo/types</code> and <code>local:demo/namespace</code>). Note that this nested<br> scheme allows a single component to both define and implement a WIT interface<br> without name conflict.<br> * The wrapping component-type has an <code>import</code> for every <code>use</code> in the interface,<br> bringing any <code>use</code>d types into scope so that they can be aliased when<br> building the instance-type. The component-type can be thought of as<br> "parameterizing" the interface's compiled instance type (∀T.{instance type}).<br> Note that there is <em>always</em> an outer wrapping component-type, even when the<br> interface contains no <code>use</code>s.</p> <p>One useful consequence of this encoding scheme is that each top-level<br> definition is self-contained and valid (according to Component Model validation<br> rules) independent of each other definition. This allows packages to be<br> trivially split or unioned (assuming the result doesn't have to be a valid<br> package, but rather just a raw list of non-exported type definitions).</p> <p>Another expectation is that, when a component containing WIT definitions is<br> published to a registry, the registry validates that the fully-qualified WIT<br> interface names inside the component are consistent with the registry-assigned<br> package name. For example, the above component would only be valid if published<br> with package name <code>local:demo</code>; any other package name would be inconsistent<br> with the internal <code>local:demo/types</code> and <code>local:demo/namespace</code> exported<br> interface names.</p> <p>Inter-package references are structurally no different than intra-package<br> references other than the referenced WIT definition is not present in<br> the component. For example, the following WIT:</p> <pre><code class="language-wit">package local:demo interface foo { use wasi:http/types.{request}; frob: func(r: request) -&gt; request; } </code></pre> <p>is encoded as:</p> <pre><code class="language-wat">(component (type (export "foo") (component (import "wasi:http/types" (instance $types (export "request" (type (sub resource))) )) (alias export $types "request" (type $request)) (export "local:demo/foo" (instance (export "frob" (func (param "r" (own $request)) (result (own $request)))) )) )) ) </code></pre> <p>Worlds are encoded similarly to interfaces, but replace the inner exported<br> instance with an inner exported <em>component</em>. For example, this WIT:</p> <pre><code class="language-wit">package local:demo; world the-world { export test: func(); export run: func(); } </code></pre> <p>is encoded as:</p> <pre><code class="language-wat">(component (type (export "the-world") (component (export "local:demo/the-world" (component (export "test" (func)) (export "run" (func)) )) )) ) </code></pre> <p>In the current version of WIT, the outer wrapping component-type will only ever<br> contain a single <code>export</code> and thus only serves to separate the kebab-name<br> export from the inner exported interface name and to provide consistency with<br> the encoding of <code>interface</code> shown above.</p> <p>When a world imports or exports an interface, to produce a valid<br> component-type, the interface's compiled instance-type ends up getting copied<br> into the component-type. For example, the following WIT:</p> <pre><code class="language-wit">package local:demo; world the-world { import console; } interface console { log: func(arg: string); } </code></pre> <p>is encoded as:</p> <pre><code class="language-wat">(component (type (export "the-world") (component (export "local:demo/the-world" (component (import "local:demo/console" (instance (export "log" (func (param "arg" string))) )) )) )) (type (export "console") (component (export "local:demo/console" (instance (export "log" (func (param "arg" string))) )) )) ) </code></pre> <p>This duplication is useful in the case of cross-package references or split<br> packages, allowing a compiled <code>world</code> definition to be fully self-contained and<br> able to be used to compile a component without additional type information.</p> <p>Putting this all together, the following WIT definitions:</p> <pre><code class="language-wit">// wasi-http repo // wit/types.wit interface types { resource request { ... } resource response { ... } } // wit/handler.wit interface handler { use types.{request, response}; handle: func(r: request) -&gt; response; } // wit/proxy.wit package wasi:http; world proxy { import wasi:logging/logger; import handler; export handler; } </code></pre> <p>are encoded as:</p> <pre><code class="language-wat">(component (type (export "types") (component (export "wasi:http/types" (instance (export "request" (type (sub resource))) (export "response" (type (sub resource))) ... )) )) (type (export "handler") (component (import "wasi:http/types" (instance $http-types (export "request" (type (sub resource))) (export "response" (type (sub resource))) )) (alias export $http-types "request" (type $request)) (alias export $http-types "response" (type $response)) (export "wasi:http/handler" (instance (export "handle" (func (param "r" (own $request)) (result (own $response)))) )) )) (type (export "proxy") (component (export "wasi:http/proxy" (component (import "wasi:logging/logger" (instance ... )) (import "wasi:http/types" (instance $http-types (export "request" (type (sub resource))) (export "response" (type (sub resource))) ... )) (alias export $http-types "request" (type $request)) (alias export $http-types "response" (type $response)) (import "wasi:http/handler" (instance (export "handle" (func (param "r" (own $request)) (result (own $response)))) )) (export "wasi:http/handler" (instance (export "handle" (func (param "r" (own $request)) (result (own $response)))) )) )) )) ) </code></pre> <p>This examples shows how, in the context of concrete world (<code>wasi:http/proxy</code>),<br> standalone interface definitions (such <code>wasi:http/handler</code>) are no longer in a<br> "parameterized" form: there is no outer wrapping component-type and instead all<br> <code>use</code>s are replaced by direct aliases to preceding type imports as determined<br> by the WIT resolution process.</p> <p>Unlike most other WIT constructs, the <code>@since</code> and <code>@unstable</code> gates are not<br> represented in the component binary. Instead, they are considered "macro"<br> constructs that take the place of maintaining two copies of a single WIT<br> document. In particular, when encoding a collection of WIT documents into a<br> binary, the target version and set of explicitly-enabled feature names<br> determine whether individual gated features are included in the encoded type or<br> not.</p> <p>For example, the following WIT document:</p> <pre><code class="language-wit">package ns:p@1.1.0; interface i { f: func(); @since(version = 1.1.0) g: func(); } </code></pre> <p>is encoded as the following component when the target version is <code>1.0.0</code>:</p> <pre><code class="language-wat">(component (type (export "i") (component (export "ns:p/i@1.0.0" (instance (export "f" (func)) )) )) ) </code></pre> <p>If the target version was instead <code>1.1.0</code>, the same WIT document would be<br> encoded as:</p> <pre><code class="language-wat">(component (type (export "i") (component (export "ns:p/i@1.1.0" (instance (export "f" (func)) (export "g" (func)) )) )) ) </code></pre> <p>Thus, <code>@since</code> and <code>@unstable</code> gates are not part of the runtime semantics of<br> components, just part of the source-level tooling for producing components.</p> https://tedneward.github.io/Research/platforms/wasmcloud/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wasmcloud/index.html <p><a href="https://wasmcloud.com/">Website</a> | <a href="https://github.com/wasmcloud">Source</a></p> https://tedneward.github.io/Research/platforms/vanillaos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/vanillaos/index.html <p><a href="https://vanillaos.org/">Website</a></p> <p><a href="https://news.itsfoss.com/vanilla-os-release/">Blog Post announcing stable release</a></p> https://tedneward.github.io/Research/platforms/visly/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/visly/index.html <p>Looks to have been incorporated into Figma.</p> <p><a href="https://www.figma.com/">https://www.figma.com/</a></p> https://tedneward.github.io/Research/platforms/wappler/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wappler/index.html <p><a href="https://wappler.io/">Website</a></p> https://tedneward.github.io/Research/platforms/unqork/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/unqork/index.html <p><a href="https://www.unqork.com/">Website</a> Looks to be fully commercial</p> https://tedneward.github.io/Research/platforms/wasm/components/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/wasm/components/index.html <p><a href="https://component-model.bytecodealliance.org/introduction.html">Website</a></p> <p><a href="https://radu-matei.com/blog/intro-wasm-components/">An introduction to WebAssembly components</a></p> <h2>Component Model Concepts</h2> <ul> <li> <p>A WebAssembly Component is the next evolution of core WebAssembly binaries.</p> <ul> <li>WebAssembly components are nestable -- they may contain one or more core modules and/or sub-components composed together.</li> <li>Logically, components are containers for modules - or other components - which express their interfaces and dependencies via WIT.</li> <li>Conceptually, components are self-describing units of code that interact only through interfaces instead of shared memory.</li> <li>Physically, a component is a specially-formatted WebAssembly file. Internally, the component could include multiple traditional ("core") WebAssembly modules, and sub-components, composed via their imports and exports.</li> </ul> <p>The external interface of a component - its imports and exports - corresponds to a world. The component, however, internally defines how that world is implemented.</p> </li> <li> <p>The Component Model extends core WebAssembly by introducing higher level types and interface-driven development</p> <ul> <li>WebAssembly Interface Types (WIT) is the IDL (Interface Definition Language) used to formally define functionality for WebAssembly modules.</li> <li>With WIT, WebAssembly components gain the ability to conform an language-agnostic and encode that support, so any WebAssembly component binary can be interrogated and executed.</li> <li>An Interface describes the types and functions used for a specific, focused bit of functionality. They are defined using WIT. <ul> <li>An interface describes a single-focus, composable contract, through which components can interact with each other and with hosts. Interfaces describe the types and functions used to carry out that interaction. For example:</li> <li>A "receive HTTP requests" interface might have only a single "handle request" function, but contain types representing incoming requests, outgoing responses, HTTP methods and headers, and so on.</li> <li>A "wall clock" interface might have two functions, one to get the current time and one to get the granularity of the timer. It would also include a type to represent an instant in time.</li> </ul> </li> <li> <p>A World assembles interfaces to express what features a component offers, and what features it depends on. A WIT world is a higher-level contract that describes a component's capabilities and needs.</p> <p>On one hand, a world describes the shape of a component - it says which interfaces the component exposes for other code to call (its exports), and which interfaces the component depends on (its imports). A world only defines the surface of a component, not the internal behaviour. If you're an application or library developer creating a component, you'll specify the world your component targets. This world describes the functionality your component exposes and declares the functionality your component depends on in order to be able to run. Your component may target a custom world definition you have created with a unique set of imports and exports tailored just for your use case, or it may target an existing world definition that someone else has already specified.</p> <p>On the other hand though, a world defines a hosting environment for components (i.e., an environment in which a component can be instantiated and its functionality can be invoked). An environment supports a world by providing implementations for all of the imports and by optionally invoking one or more of the exports.</p> <p>For example, WASI (the WebAssembly System Interface) defines a "command line" world which imports interfaces that command line programs typically expect to have available to them such as file I/O, random number generation, clocks and so on. This world has a single export for running the command line tool. Components targeting this world must provide an implementation for this single export, and they may optionally call any of the imports. On the other hand, environments supporting this world must provide implementations for all of the imports and may invoke the single export.</p> <p>A world is composed of interfaces, but each interface is directional - it indicates whether the interface is available for outside code to call (an "export"), or whether outside code must fulfill the interface for the component to call (an "import"). These interfaces strictly bound the component. A component cannot interact with anything outside itself except by having its exports called, or by it calling its imports. This provides very strong sandboxing; for example, if a component does not have an import for a secret store, then it cannot access that secret store, even if the store is running in the same process.</p> <p>For a component to run, its imports must be fulfilled, by a host or by other components. Connecting up some or all of a component's imports to other components' matching exports is called composition.</p> <p>Example Worlds</p> <ul> <li>A (trivial) "HTTP proxy" world would export a "handle HTTP requests" interface, and import a "send HTTP requests" interface. A host, or another component, would call the exported "handle" interface, passing an HTTP request; the component would forward it on via the imported "send" interface. To be a useful proxy, the component may also need to import interfaces such as I/O and clock time - without those imports the component could not perform, for example, on-disk caching.</li> <li>A "regex parser" world would export a "parse regex" function, and would import nothing. This declares not only that the component implementing this world can parse regular expressions, but also that it calls no other APIs. A user of such a parser could know, without looking at the implementation, that is does not access the file system, or send the user's regexes to a network service.</li> </ul> </li> <li> <p>A Package is a set of WIT files containing a related set of interfaces and worlds.</p> <p>A WIT package is a set of one or more WIT (Wasm Interface Type) files containing a related set of interfaces and worlds. WIT is an IDL (interface definition language) for the Component Model. Packages provide a way for worlds and interfaces to refer to each other, and thus for an ecosystem of components to share common definitions.</p> <p>A WIT package is not a world. It's a way of grouping related interfaces and worlds together for ease of discovery and reference, more like a namespace.</p> <p>The WebAssembly System Interface (WASI) defines a number of packages, including one named <code>wasi:clocks</code>. Our HTTP proxy world could import the wall-clock interface from the <code>wasi:clocks</code> package, rather than having to define a custom clock interface.</p> </li> </ul> </li> <li> <p>The Component Model introduces the idea of a "platform" to core WebAssembly -- enabling the structured, standardized use of "host" functionality for WebAssembly "guest"s.</p> <ul> <li>The WebAssembly System Interface (WASI) defines in WIT a family of interfaces for common system-level functions.</li> <li>WASI defines common execution environments such as the command line (wasi:cli) or a HTTP server (wasi:http).</li> </ul> </li> <li> <p>The Component Model makes core WebAssembly composable -- components that provide functionality and those that use them can be composed together into one resulting component</p> </li> </ul> <h2>Component Model Concepts</h2> <ul> <li> <p>A WebAssembly Component is the next evolution of core WebAssembly binaries.</p> <ul> <li>WebAssembly components are nestable -- they may contain one or more core modules and/or sub-components composed together.</li> <li>Logically, components are containers for modules - or other components - which express their interfaces and dependencies via WIT.</li> <li>Conceptually, components are self-describing units of code that interact only through interfaces instead of shared memory.</li> <li>Physically, a component is a specially-formatted WebAssembly file. Internally, the component could include multiple traditional ("core") WebAssembly modules, and sub-components, composed via their imports and exports.</li> </ul> <p>The external interface of a component - its imports and exports - corresponds to a world. The component, however, internally defines how that world is implemented.</p> </li> <li> <p>The Component Model extends core WebAssembly by introducing higher level types and interface-driven development</p> <ul> <li>WebAssembly Interface Types (WIT) is the IDL (Interface Definition Language) used to formally define functionality for WebAssembly modules.</li> <li>With WIT, WebAssembly components gain the ability to conform an language-agnostic and encode that support, so any WebAssembly component binary can be interrogated and executed.</li> <li>An Interface describes the types and functions used for a specific, focused bit of functionality. They are defined using WIT. <ul> <li>An interface describes a single-focus, composable contract, through which components can interact with each other and with hosts. Interfaces describe the types and functions used to carry out that interaction. For example:</li> <li>A "receive HTTP requests" interface might have only a single "handle request" function, but contain types representing incoming requests, outgoing responses, HTTP methods and headers, and so on.</li> <li>A "wall clock" interface might have two functions, one to get the current time and one to get the granularity of the timer. It would also include a type to represent an instant in time.</li> </ul> </li> <li> <p>A World assembles interfaces to express what features a component offers, and what features it depends on. A WIT world is a higher-level contract that describes a component's capabilities and needs.</p> <p>On one hand, a world describes the shape of a component - it says which interfaces the component exposes for other code to call (its exports), and which interfaces the component depends on (its imports). A world only defines the surface of a component, not the internal behaviour. If you're an application or library developer creating a component, you'll specify the world your component targets. This world describes the functionality your component exposes and declares the functionality your component depends on in order to be able to run. Your component may target a custom world definition you have created with a unique set of imports and exports tailored just for your use case, or it may target an existing world definition that someone else has already specified.</p> <p>On the other hand though, a world defines a hosting environment for components (i.e., an environment in which a component can be instantiated and its functionality can be invoked). An environment supports a world by providing implementations for all of the imports and by optionally invoking one or more of the exports.</p> <p>For example, WASI (the WebAssembly System Interface) defines a "command line" world which imports interfaces that command line programs typically expect to have available to them such as file I/O, random number generation, clocks and so on. This world has a single export for running the command line tool. Components targeting this world must provide an implementation for this single export, and they may optionally call any of the imports. On the other hand, environments supporting this world must provide implementations for all of the imports and may invoke the single export.</p> <p>A world is composed of interfaces, but each interface is directional - it indicates whether the interface is available for outside code to call (an "export"), or whether outside code must fulfill the interface for the component to call (an "import"). These interfaces strictly bound the component. A component cannot interact with anything outside itself except by having its exports called, or by it calling its imports. This provides very strong sandboxing; for example, if a component does not have an import for a secret store, then it cannot access that secret store, even if the store is running in the same process.</p> <p>For a component to run, its imports must be fulfilled, by a host or by other components. Connecting up some or all of a component's imports to other components' matching exports is called composition.</p> <p>Example Worlds</p> <ul> <li>A (trivial) "HTTP proxy" world would export a "handle HTTP requests" interface, and import a "send HTTP requests" interface. A host, or another component, would call the exported "handle" interface, passing an HTTP request; the component would forward it on via the imported "send" interface. To be a useful proxy, the component may also need to import interfaces such as I/O and clock time - without those imports the component could not perform, for example, on-disk caching.</li> <li>A "regex parser" world would export a "parse regex" function, and would import nothing. This declares not only that the component implementing this world can parse regular expressions, but also that it calls no other APIs. A user of such a parser could know, without looking at the implementation, that is does not access the file system, or send the user's regexes to a network service.</li> </ul> </li> <li> <p>A Package is a set of WIT files containing a related set of interfaces and worlds.</p> <p>A WIT package is a set of one or more WIT (Wasm Interface Type) files containing a related set of interfaces and worlds. WIT is an IDL (interface definition language) for the Component Model. Packages provide a way for worlds and interfaces to refer to each other, and thus for an ecosystem of components to share common definitions.</p> <p>A WIT package is not a world. It's a way of grouping related interfaces and worlds together for ease of discovery and reference, more like a namespace.</p> <p>The WebAssembly System Interface (WASI) defines a number of packages, including one named <code>wasi:clocks</code>. Our HTTP proxy world could import the wall-clock interface from the <code>wasi:clocks</code> package, rather than having to define a custom clock interface.</p> </li> </ul> </li> <li> <p>The Component Model introduces the idea of a "platform" to core WebAssembly -- enabling the structured, standardized use of "host" functionality for WebAssembly "guest"s.</p> <ul> <li>The WebAssembly System Interface (WASI) defines in WIT a family of interfaces for common system-level functions.</li> <li>WASI defines common execution environments such as the command line (wasi:cli) or a HTTP server (wasi:http).</li> </ul> </li> <li> <p>The Component Model makes core WebAssembly composable -- components that provide functionality and those that use them can be composed together into one resulting component</p> </li> </ul> <hr> https://tedneward.github.io/Research/platforms/macos/setup/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/macos/setup/index.html <p><a href="https://wilsonmar.github.io/ansible-mac-osx-setup/">Ansible macOS setup</a></p> <p><a href="https://www.freecodecamp.org/news/set-up-your-macos-development-environment-using-thoughtbots-laptop-script-e6bf9b2e03dd/">Thoughtbot's Setup script</a></p> <p><a href="https://www.xda-developers.com/how-install-macos-unsupported-mac/">"How to install macOS Sonoma on unsupported Macs"</a></p> <p><a href="https://www.xda-developers.com/15-settings-you-didnt-know-you-need-to-change-on-mac/">"15 Settings you didn't know you need to change on Mac"</a>:</p> <ul> <li>Create text replacements</li> <li>Generate unique email addresses (iCloud subscribers)</li> <li>Change Mac gestures</li> <li>Disable Private Relay</li> <li>Tweak Spotlight search</li> <li>Use Apple Watch to unlock applications on Mac</li> <li>Launch Focus when you open an app</li> <li>Enable and tweak iPhone notifications</li> <li>Enable notifications when mirroring on Mac</li> <li>Set the picture folder as wallpaper</li> <li>Show a message on the lock screen</li> <li>Tweak hot corners</li> <li>Use storage recommendations</li> <li>Prevent automatic sleeping on power adapter</li> <li>Enable Low Data mode for Wi-Fi</li> </ul> https://tedneward.github.io/Research/platforms/mdk/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mdk/index.html <p><a href="https://www.gnu.org/software/mdk/mdk.html">Website</a> | <a href="https://savannah.gnu.org/git/?group=mdk">Source</a></p> https://tedneward.github.io/Research/platforms/minimicro/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/minimicro/index.html <p><a href="https://miniscript.org/MiniMicro/index.html">Website</a> | <a href="https://github.com/JoeStrout/MiniMicro2">Source</a></p> https://tedneward.github.io/Research/platforms/mobile/grapheneos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/grapheneos/index.html <p><a href="https://grapheneos.org/">Website</a></p> https://tedneward.github.io/Research/platforms/monaca/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/monaca/index.html <p><a href="https://monaca.io/">Website</a></p> <p>Commercial--no source. Cloud-hosted IDEs.</p> <p>Supports both <a href="/presentation/onsenui">Onsen UI</a> and <a href="/presentation/cordova">Cordova</a> projects?</p> https://tedneward.github.io/Research/platforms/nextspace/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/nextspace/index.html <p><a href="https://github.com/trunkmaster/nextspace">Source</a> | <a href="https://trunkmaster.github.io/">Blog</a></p> <p><img src="https://github.com/trunkmaster/nextspace/raw/master/Documentation/NEXTSPACE_Screenshot.png" alt="NEXTSPACE Screenshot"></p> https://tedneward.github.io/Research/platforms/ockam/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/ockam/index.html <p><a href="https://www.ockam.io/">Website</a> | <a href="https://github.com/build-trust/ockam">Source</a></p> https://tedneward.github.io/Research/platforms/openxava/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/openxava/index.html <p><a href="https://www.openxava.org/">Website</a> | <a href="https://www.openxava.org/">Source</a> | <a href="https://www.openxava.org/doc/">Docs</a></p> <p>Open Source Low-Code Platform for Rapid Development of Enterprise Web Applications.</p> <p>Write just the domain classes in plain Java. Get a web application ready for production.</p> <h4>High productivity</h4> <p>You only write the code for your data structure and business logic. You do not write HTML, JavaScript, CSS, SQL, etc. The user interface and the database logic are automatically provided.</p> <h4>Short learning curve</h4> <p>Learning how to write simple Java classes is enough to write complete applications. The OpenXava bundle is prepared for a rapid startup.</p> <h4>Full-featured applications</h4> <p>AJAX user interface with no page reloading. List mode with paging, ordering, filtering, adding / removing / moving columns, PDF reports, export to Excel, cards format, charts, etc. Detail mode with tabs, frames, dialogs, editors for references and collections, responsive layout, etc.</p> <h4>Mobile user interface</h4> <p>In addition to the classic web user interface for desktop you can get a mobile user interface for your application from the same code.</p> <h4>Use your favorite platform</h4> <p>Browsers: Internet Explorer, Chrome, Firefox and Safari. Databases: Any supported by Hibernate, that is Oracle, DB2, AS/400, Informix, PostgreSQL, MySQL, MS SQL Server and practically all relational databases. Operating systems: Any with Java 8 (or better) support, that is Windows, Linux, Mac, Unix, AS/400, z/OS, etc. Application servers: Any with support for Servlets 3.0 (or better), including Tomcat, JBoss, WebSphere, Glassfish, WebLogic, etc. Enterprise portals: Any with JSR-168 or JSR-286 support including WebSphere Portal and Liferay.</p> <h4>Open source</h4> <p>LGPL license that allows you to develop commercial applications without paying any fees.</p> <h4>Multilingual</h4> <p>The standard labels and messages included in OpenXava are in English, Spanish, German, French, Chinese, Russian, Japanese, Portuguese, Catalan, Indonesian, Italian, Polish, Serbian and Swedish. Moreover, it's pretty easy to add a new language.</p> <h4>The most used Java domain-driven framework</h4> <p>250,000 downloads. 60 authors. Thousands of threads in the forums. Hundreds of applications developed. Hundreds of professionals in LinkedIn with OpenXava experience.</p> <h4>Exhaustive documentation</h4> <p>Complete Reference Guide in English, French, Russian, Chinese and Spanish. Free online course in English and Spanish where an invoicing application is developed from scratch. Book in English and Spanish available from Amazon. Hundreds of videos on YouTube.</p> <h4>Quality support</h4> <p>Free support in community forums, though they have thousands of threads all questions are answered. Moreover, there is professional support available.</p> <h4>Based on Java standards</h4> <p>Thus you can migrate your current Java code easily to OpenXava, and vice versa. You can use any Java tool for OpenXava. Your Java developers are already OpenXava developers. OpenXava supports: JSR-338, JSR-317, JSR-303, JSR-330, JSR-220, JSR-153, JSR-311, JSR-339, JSR-370, JSR-168 and JSR-286.</p> <h4>Third party tools</h4> <p>MinuteProject: Generates an OpenXava application from an already existing database. Mogwai ERDesigner: An entity relationship modeling/design tool(ERD) that generates OpenXava applications. Moskitt: A CASE tool, built on Eclipse, that generates OpenXava applications from UML models.</p> https://tedneward.github.io/Research/platforms/outsystems/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/outsystems/index.html <p><a href="https://www.outsystems.com/">Website</a></p> <blockquote> <p>OutSystems is a modern application platform designed to dramatically accelerate the development of your most critical applications while also delivering unprecedented levels of flexibility and efficiency. Address your top digital transformation priorities with applications that make a difference across all areas of the business—from customer experience transformation and workplace innovation to process automation and application modernization.</p> </blockquote> <h4>Personal conversation with @devhammer (7/1/2021):</h4> <p>Suports either a cloud (AWS)-based deployment model, or a "self-service" deployment model.</p> <p>Considers itself very "low-code"--a tool that developers will recognize and find useful, but not for people with no programming background. "Brought back all the power/love of using VB, but keeps people in that visual model/mindset longer than VB does."</p> <p>Uses Apache Cordova for multiplatform application client. Recently moved from ASP.NET WebForms-based UI to a React one. (Will continue to support WebForms for a "long time to come".)</p> https://tedneward.github.io/Research/platforms/platformatic/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/platformatic/index.html <p><a href="https://docs.platformatic.dev/">Website</a> | <a href="https://github.com/platformatic">Source</a></p> <p>Platformatic’s Open Source toolkit helps you build modern Node.js applications without the endless loop of setup and maintenance.<br> Break free from repetitive tasks, accelerate deployments and improve your Node.js development experience with our:</p> <ul> <li>Opinionated structures and composable templates</li> <li>Batteries-included API setup</li> <li>Out-of-the-box documentation, logs and metrics</li> <li>Shareable and reusable configurations</li> </ul> <p><strong>Platformatic for Teams</strong></p> <p>Node.js promises speed, agility, and scalability. In practice however, doing Node right has been a challenge for engineering teams, costing their companies cloud spend, downtime, and time-to-market.<br> Time and time again, we’ve seen companies stumble when dealing with the same 5 issues:</p> <ul> <li>Scaling based on CPU and memory.</li> <li>Preview environment limitations</li> <li>Getting the right Node.js metrics</li> <li>Not knowing what will break or which teams you’ll impact when deploying APIs</li> <li>Figuring out which logger is right for you</li> </ul> https://tedneward.github.io/Research/platforms/powerloom/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/powerloom/index.html <p><a href="https://www.isi.edu/isd/LOOM/PowerLoom/">Website</a></p> https://tedneward.github.io/Research/platforms/puter/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/puter/index.html <p><a href="https://puter.com/">Website/live platform</a> | <a href="https://docs.puter.com/#/">SDK Docs</a> | <a href="https://github.com/HeyPuter/">Source</a> (examples)</p> https://tedneward.github.io/Research/platforms/quickbase/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/quickbase/index.html <p><a href="https://www.quickbase.com/">Website</a></p> https://tedneward.github.io/Research/platforms/reactos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/reactos/index.html <p><a href="https://reactos.org/">Website</a> | <a href="https://github.com/reactos/reactos">Source</a></p> <p>Windows might be the most used operating system for home users, but what if you don't want to use Microsoft's OS but do want to use the apps written to run on it? Before SteamOS and the Proton compatibility layer added that functionality to Linux, ReactOS was the only answer to that question. It's an open-source clone of Windows NT that is fully compatible with any app, driver, and service that also runs on Windows NT.</p> <p>As such, it's probably the easiest retro-feeling OS to install and use because it feels familiar while offering modern conveniences and stability. It can also be run from a live USB, so you don't even have to install it on your internal hard drive if you prefer to test-drive things before proceeding.</p> https://tedneward.github.io/Research/platforms/retool/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/retool/index.html <p><a href="https://retool.com/">Website</a> Commercial with a free tier.</p> <p>All internal tools are made up of the building blocks: Tables, Lists, Charts, Forms, Wizards, Maps and so on. Retool provides a complete set of powerful building blocks out of the box. Spend your time getting UI in front of stakeholders, not hunting down the best React table library. Assemble your app in 30 seconds by dragging and dropping from our pre-built <a href="https://retool.com/components">components</a>.</p> <p>Retool has 4 fundamental pieces to it:</p> <ul> <li>Connect your data sources, like PostgreSQL, Salesforce, Firebase, and 20+ more</li> <li>Build your queries and logic in SQL or Javascript</li> <li>Connect your queries and logic to prebuilt components like tables, text inputs, and buttons</li> <li>Organize and connect your components into an app</li> </ul> <p>Retool isn't just a front end, though – we take care of a lot of the pesky logic that internal tools tend to require, like scheduling queries, updating and writing data, and triggers. Retool apps are easy to share with your teammates and stakeholders, and we offer granular access management and audit logs to keep things secure.</p> https://tedneward.github.io/Research/platforms/rubymotion/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/rubymotion/index.html <p><a href="http://www.rubymotion.com/">Website</a></p> <h3>Examples</h3> <p>Android HelloWorld</p> <pre><code class="language-ruby">class MainActivity &lt; Android::App::Activity def onCreate(savedInstanceState) puts "Hello World!" super view = Android::Widget::TextView.new(self) view.text = "Hello World!" self.contentView = view end end </code></pre> <p>iOS HelloWorld</p> <pre><code class="language-ruby">class AppDelegate def application(application, didFinishLaunchingWithOptions:launchOptions) true end end class AppDelegate def application(application, didFinishLaunchingWithOptions:launchOptions) alert = UIAlertView.new alert.message = "Hello World!" alert.show true end end </code></pre> https://tedneward.github.io/Research/platforms/serenity/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/serenity/index.html <p><a href="https://serenityos.org/">Website</a> | <a href="https://github.com/SerenityOS/serenity">Source</a></p> https://tedneward.github.io/Research/platforms/skyve/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/skyve/index.html <p><a href="https://skyve.org/">Website</a> | <a href="">Source</a><a href="https://github.com/skyvers/skyve">https://github.com/skyvers/skyve</a></p> https://tedneward.github.io/Research/platforms/strapi/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/strapi/index.html <p><a href="https://strapi.io/">Website</a> | <a href="https://github.com/strapi/strapi">Source</a></p> https://tedneward.github.io/Research/platforms/tabris/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tabris/index.html <p><a href="https://tabris.com/">Website</a> | <a href="https://github.com/eclipsesource/tabris-js/">Source</a></p> <p>(Looks like it builds on top of Apache Cordova somewhere/how?)</p> <p>Hello, world in Tabris:</p> <pre><code>import {Button, contentView, TextView} from 'tabris'; // in JS new Button({top: 16, centerX: true, text: 'Use native UI'}) .onSelect(() =&gt; $(TextView).only().text = 'Powered by Tabris.js') .appendTo(contentView); new TextView({top: 'prev() 16', centerX: true}) .appendTo(contentView); // or in JSX contentView.append( &lt;$&gt; &lt;Button top={16} centerX text='Use native UI' onSelect={() =&gt; $(TextView).only().text = 'Powered by Tabris.js'}/&gt; &lt;TextView top='prev() 16' centerX/&gt; &lt;/$&gt; ); </code></pre> <p>yields<br> <img src="https://github.com/eclipsesource/tabris-js/raw/master/doc/img/hello-example.png" alt=""><br> and<br> ~<a href="https://github.com/eclipsesource/tabris-js/raw/master/doc/img/hello-example.png"></a></p> <p>The code of the application is loaded dynamically - nothing is precompiled. JavaScript is executed Just-in-Time and passed via a native bridge to the device. Tabris.js accesses native controls and does not depend on webviews to render the app's UI.</p> <p>A Compiled Tabris.js app is written in JavaScript or TypeScript and offers support for the latest EcmaScript features and JSX. It is recommended for more complex Tabris.js projects. The only downside of this setup is that the code needs to be transformed before execution, which is done automatically when using the Tabris CLI or build service.</p> <p>A Vanilla Tabris.js app is written in plain JavaScript that is executed as-is in the JavaScript engine. As a result there may be minor differences between the platforms.</p> https://tedneward.github.io/Research/platforms/tink/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tink/index.html <p><a href="https://tink.com/">Website</a></p> <h3>Products</h3> <ul> <li> <p>Transactions: Access up-to-date transaction data from thousands of banks across Europe – already standardised and categorised so you can extract real value and insights.</p> </li> <li> <p>Account Check: Confirm account ownership with real-time data straight from a user’s bank account, creating a quick and simple payment setup process.</p> </li> <li> <p>Income Check: Optimise credit decisions and understand your customers’ true financial capacity by instantly verifying income using real-time data.</p> </li> <li> <p>Payment Initiation: Increase engagement and conversion by giving your customers a fully embedded payments experience – at a fraction of the cost.</p> </li> <li> <p>Money Manager: Build smart, intuitive personal finance management applications that give your customers tools and personalised insights to better manage their money.</p> </li> </ul> https://tedneward.github.io/Research/platforms/uefi/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/uefi/index.html <p>The replacement for BIOS on PCs and servers. An operating system in its own right, comparable in complexity to DOS.</p> <p><a href="https://uefi.org/">Website</a> | <a href="https://uefi.org/sites/default/files/resources/UEFI%20Spec%202_6.pdf">UEFI Specification</a></p> <p><a href="https://uefi.org/sites/default/files/resources/UEFI_Shell_2_2.pdf">Shell Specification</a> | <a href="https://manuais.iessanclemente.net/images/a/a6/EFI-ShellCommandManual.pdf">Intel Shell Command Reference Manual</a> | <a href="https://www.intel.com/content/dam/support/us/en/documents/motherboards/server/sb/efi_instructions.pdf">Intel Basic Instructions for Using the<br> Extensible Firmware Interface (EFI)</a></p> <p><a href="https://kb.stonegroup.co.uk/index.php?View=entry&amp;EntryID=84">"How to Access the EFI Shell to carry out Systems Diagnostics or Updates"</a>: Turns out the shell can read USB-key drives and execute .EFI executable files--which is interesting in of itself--and this article describes how to boot into an Intel machine and use the UEFI shell to do that.</p> <p><a href="https://www.rodsbooks.com/efi-programming/index.html">Programming for EFI</a>: Building programs for the EFI environment.</p> https://tedneward.github.io/Research/platforms/macos/usage/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/macos/usage/index.html <h2>Network utilities</h2> <ul> <li><a href="https://cyberhost.uk/the-hidden-macos-speedtest-tool-networkquality/">"networkQuality"</a>: "The networkQuality tool is a built-in tool released in macOS Monterey that can help diagnose network issues and measure network performance."</li> </ul> <h2>Terminal Tricks</h2> <ul> <li> <p><a href="https://josh.works/shell-script-basics-change-mac-address">"Change your MAC address with a shell script"</a></p> </li> <li> <p><a href="https://www.techrepublic.com/article/macos-terminal-commands-every-mac-user-should-know/">"macOS terminal commands every mac user should know"</a>:</p> <ul> <li> <p>Update macOS via SoftwareUpdate in the terminal: <code>softwareupdate -i -a</code></p> </li> <li> <p>Copy output of a command to the clipboard: <code>cat ~/Desktop/myfile.txt | pbcopy</code></p> </li> <li> <p>View system utilization: <code>top</code></p> </li> <li> <p>Keep Mac from turning off/sleeping: <code>caffeinate</code></p> </li> <li> <p>Clear the DNS cache: <code>sudo killall -HUP mDNSResponder;sudo killall mDNSResponderHelper;sudo dscacheutil -flushcache</code></p> </li> <li> <p>Find diffs in files: <code>opendiff ~/Desktop/text1.txt ~/Desktop/text2.txt</code></p> </li> <li> <p>Find out how long the Mac has been on: <code>uptime</code></p> </li> <li> <p>Force macOS to shut down: <code>shutdown -r now</code></p> </li> <li> <p>Get a QuickLook preview of a file: <code>qlmanage -p ~/Desktop/text.txt</code></p> </li> </ul> </li> <li> <p><a href="https://www.techrepublic.com/article/16-terminal-commands-every-user-should-know/">"16 Terminal commands every user should know"</a>:</p> <ul> <li> <p>Copy contents of a folder to a new folder: <code>ditto</code></p> </li> <li> <p>Get one-line description for a command: <code>whatis</code></p> </li> <li> <p>Keep the Mac from hibernating/sleeping: <code>caffeinate</code></p> </li> </ul> </li> <li> <p><a href="https://www.makeuseof.com/tag/useful-mac-osx-shortcuts-pdf/">"Useful macOS shortcuts"</a>:</p> <p>Startup: Intel</p> <table> <thead> <tr> <th>Action </th> <th> Keystroke(s)</th> </tr> </thead> <tbody> <tr> <td>Boot without automatic login </td> <td> Left shift (Press and hold when macOS progress indicator appears)</td> </tr> <tr> <td>Boot in Safe Mode (Intel) </td> <td> Shift (Press and hold)</td> </tr> <tr> <td>Boot in Single User Mode (Intel, unavailable in macOS Mojave and later) </td> <td> Cmd + S</td> </tr> <tr> <td>Boot in Verbose Mode (Intel) </td> <td> Cmd + V</td> </tr> <tr> <td>Boot in Target Disk Mode (Intel) </td> <td> T</td> </tr> <tr> <td>Boot from a NetBoot network server </td> <td> N</td> </tr> <tr> <td>Boot to Startup Manager (Intel) </td> <td> Option</td> </tr> <tr> <td>Load macOS Recovery (Intel) </td> <td> Cmd + R</td> </tr> <tr> <td>Load macOS Recovery over the internet (Intel) </td> <td> Option + Cmd + R</td> </tr> <tr> <td>Enter Apple Diagnostics (Intel) </td> <td> D</td> </tr> <tr> <td>Load Apple Diagnostics over the internet </td> <td> Option + D</td> </tr> <tr> <td>Reset NVRAM or PRAM (Intel, unnecessary on Apple Silicon Macs) </td> <td> Option + Cmd + P + R</td> </tr> <tr> <td>Eject removable media </td> <td> F12</td> </tr> </tbody> </table> <p>Startup: Apple Silicon</p> <table> <thead> <tr> <th>Action </th> <th> Keystroke(s)</th> </tr> </thead> <tbody> <tr> <td>Boot without automatic login </td> <td> Left shift (Press and hold when macOS progress indicator appears)</td> </tr> <tr> <td>Boot in Safe Mode (Apple Silicon) </td> <td> Power (Press and hold), then select a volume, then hold Shift while clicking Continue in Safe Mode</td> </tr> <tr> <td>Boot to Startup Manager (Apple Silicon) </td> <td> Power</td> </tr> <tr> <td>Enter Apple Diagnostics (Apple Silicon) </td> <td> Cmd + D</td> </tr> <tr> <td>Load macOS Recovery (Apple Silicon) </td> <td> Power, then select Options, then click Continue</td> </tr> <tr> <td>Load Apple Diagnostics over the internet </td> <td> Option + D</td> </tr> <tr> <td>Boot from a NetBoot network server </td> <td> N</td> </tr> <tr> <td>Eject removable media </td> <td> F12</td> </tr> </tbody> </table> <p>Finder</p> <table> <thead> <tr> <th>Action </th> <th> Keystroke(s)</th> </tr> </thead> <tbody> <tr> <td>Go to folder </td> <td> Cmd + Shift + G</td> </tr> <tr> <td>Show/hide the Path Bar </td> <td> Option + Cmd + P</td> </tr> <tr> <td>Show/hide the Status Bar </td> <td> Option + Cmd + / (Slash)</td> </tr> </tbody> </table> <p>General</p> <table> <thead> <tr> <th>Action </th> <th> Keystroke(s)</th> </tr> </thead> <tbody> <tr> <td>Lock the screen </td> <td> Cmd + Control + Q</td> </tr> <tr> <td>Log out </td> <td> Shift + Cmd + Q</td> </tr> <tr> <td>Log out without confirmation </td> <td> Option + Shift + Cmd + Q</td> </tr> <tr> <td>Display the macOS screenshotting tools </td> <td> Cmd + 5</td> </tr> <tr> <td>Take a screenshot of the entire screen </td> <td> Shift + Cmd + 3</td> </tr> <tr> <td>Take a screenshot of a portion of the screen </td> <td> Shift + Cmd + 4 (then hold Space to drag the selection, if needed)</td> </tr> <tr> <td>Take a screenshot of a specific window or menu </td> <td> Cmd + Shift + 4, then Space</td> </tr> <tr> <td>Close all windows </td> <td> Option + Cmd + W</td> </tr> <tr> <td>Paste and delete original items </td> <td> Option + Cmd + V</td> </tr> <tr> <td>Paste and match style </td> <td> Shift + Option + Cmd + V</td> </tr> <tr> <td>Zoom in </td> <td> Cmd + + (Plus)</td> </tr> <tr> <td>Zoom out </td> <td> Cmd + - (Minus)</td> </tr> <tr> <td>Open preferences for the current app </td> <td> Cmd + , (Comma)</td> </tr> <tr> <td>Open Spotlight search </td> <td> Cmd + Space</td> </tr> <tr> <td>Hide/show the Dock </td> <td> Option + Cmd + D</td> </tr> <tr> <td>Show the Character Viewer </td> <td> Control + Cmd + Space</td> </tr> <tr> <td>Show the emoji picker </td> <td> Fn</td> </tr> <tr> <td>Show desktop icons as stacks </td> <td> Cmd + Control + 0</td> </tr> <tr> <td>Switch the current app to fullscreen mode </td> <td> Control + Cmd + F</td> </tr> <tr> <td>Quit the selected app in the app switcher </td> <td> Cmd + Shift, then Q</td> </tr> <tr> <td>Force quit apps </td> <td> Option + Cmd + Esc</td> </tr> <tr> <td>Switch to the next window in the current app </td> <td> Cmd + ~ (Tilde)</td> </tr> <tr> <td>Put display(s) to sleep </td> <td> Shift + Control + Eject</td> </tr> <tr> <td>Put your computer to sleep </td> <td> Option + Cmd + Eject</td> </tr> <tr> <td>Quit all apps and restart </td> <td> Control + Cmd + Eject</td> </tr> <tr> <td>Quit all apps and shut down </td> <td> Control + Option + Cmd + Power</td> </tr> <tr> <td>Choose from Sleep, Restart, and Shutdown options </td> <td> Control + Eject</td> </tr> </tbody> </table> </li> </ul> <h2>A/V Tricks</h2> <ul> <li><a href="https://macreports.com/record-face-screen-mac/">"How to Record Your Face and Screen"</a>: <ol> <li> <p>1-Launch QuickTime player. It can be found in the applications folder.</p> </li> <li> <p>Select File &gt; New Movie Recording.</p> </li> <li> <p>Select View &gt; Float on top so that your camera window will be on top of any other application anywhere on your screen. If you do not select this, your video may not record you.</p> </li> <li> <p>Now you can resize the camera window. You can move this camera view anywhere you want (left, right, top, bottom, etc).</p> </li> <li> <p>Now again, in QuickTime Player, select File &gt; New Screen Recording. You can change some settings by clicking the arrow next to the Record button. You can record your full screen or part of your Mac screen. You may want to arrange your screen (where will your camera be etc?) before you start your recording.</p> </li> <li> <p>When you are ready to start recording, click the red record button. You have two options: (a) if you want to record your entire screen, click anywhere on your screen to start the actual recording (b) or you may drag to select an area.</p> </li> <li> <p>Now you are recording your screen and your camera view (e.g. your face). When you are done recording you may press the Command-Control-Esc keys. After you done, you may also edit your video using Quicktime. For instance, you may go to QuickTime &gt; Edit &gt; Trim to remove the unwanted parts of your video.</p> </li> </ol> <p>Additional notes:</p> <ul> <li>You can also press Shift-Command (⌘)-5 to create video and capture still images of your screen.</li> <li>Please note that if your Mac does not have a built-in camera, you can use an external camera.</li> <li>You can use an iOS device connected to your Mac to record a video on your device.</li> </ul> </li> </ul> https://tedneward.github.io/Research/platforms/mendix/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mendix/index.html <p><a href="https://www.mendix.com/">Website</a></p> https://tedneward.github.io/Research/platforms/minix/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/minix/index.html <p><a href="http://www.minix3.org/">Website</a> | <a href="http://git.minix3.org/?p=minix.git;a=summary">Source</a></p> <p>From the book <a href="https://www.amazon.com/Operating-Systems-Design-Implementat-Implementation/dp/9332550514/ref=sr_1_1">"Operating Systems: Design and Implementation"</a> (3rd Ed) by Tanenbaum</p> <p>Generally considered the starting point for most classic operating systems study. At some point I want to pull down the Minix ISO and install it into a VirtualBox so that I can hack on an OS for a while. <em>sigh</em> Someday....</p> <p>Dormant since 2018.</p> https://tedneward.github.io/Research/platforms/mobile/lineageos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/lineageos/index.html <p><a href="https://www.lineageos.org/">Website</a> | <a href="https://gitlab.com/LineageOS">Source</a></p> <p>Article: <a href="https://thenewleafjournal.com/installing-lineageos-on-a-2013-nexus-7-wi-fi/">Installing LineageOS on a 2013 Nexus 7</a></p> https://tedneward.github.io/Research/platforms/nativescript/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/nativescript/index.html <p><a href="https://www.nativescript.org/">Website</a></p> https://tedneward.github.io/Research/platforms/nodejs/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/nodejs/index.html <p><a href="https://nodejs.org/">Website</a> | <a href="https://github.com/nodejs/node">Github</a></p> <h2>Libraries</h2> <p><a href="https://www.prisma.io/">Prisma</a>: helps app developers build faster and make fewer errors with an open source database toolkit for PostgreSQL, MySQL, SQL Server, and SQLite. <a href="https://softwareengineeringdaily.com/2020/06/04/prisma-modern-database-tooling-with-johannes-schickling/">Software Engineering Daily interview</a></p> <h2>Debugging</h2> <p><a href="https://github.com/GoogleChromeLabs/ndb">ndb</a>: improved debugging experience for NodeJS</p> <h2>Interoperability</h2> <p><a href="https://github.com/microsoft/node-api-dotnet">node-api-dotnet</a>: Advanced interoperability between .NET and JavaScript in the same process.</p> <ul> <li><a href="https://www.youtube.com/watch?v=G36lrPrF09c&amp;ab_channel=NearForm">Node.js startup performance</a></li> </ul> https://tedneward.github.io/Research/platforms/openfin/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/openfin/index.html <p><a href="https://www.openfin.co/">Website</a> | <a href="https://github.com/openfin">Source</a> | <a href="https://developers.openfin.co/of-docs">Docs</a></p> <p>Basically looks like a unified Web "workspace" concept. May be low-code, hard to tell without diving into it more deeply. Dev environment appears to rely on NodeJS.</p> <h2>What is OpenFin?</h2> <blockquote> <p>OpenFin is a web runtime and operating environment designed for enterprise app development. Built on web standards, OpenFin dramatically enhances the capabilities of web apps, enabling end-user experience and enterprise controls well beyond what is possible with browsers like Chrome and Edge or with frameworks like Electron and Chromium Embedded Framework.</p> <p>We offer an out of the box platform solution through OpenFin Workspace or the ability to build from scratch using APIs provided by OpenFin Container. Whichever path you choose, OpenFin dramatically improves the desktop experience by securely connecting disparate apps and content in intuitive, efficient, and meaningful ways.</p> </blockquote> <p>Architecture diagram:<br> <img src="https://files.readme.io/3f54afa-Architecture_Diagram_v3.png" alt=""></p> <p>Sounds like a portal/"work canvas"/Smalltalk-browser-like experience inside a web browser. "Notification Center" seems to imply a messaging-based backplane. Architecture diagram shows Electron and Chromium, possibly to allow for desktop apps?</p> <h2>Workspace apps</h2> <blockquote> <p>Workspace apps organize and connect content across the desktop with Home, Browser, Notification Center &amp; Content Store.</p> <p>OpenFin Workspace is a unified space for work, powered by an extensive set of tools designed to help your firm increase productivity and deliver exceptional employee experiences. Key components of Workspace include a keyboard-driven digital assistant for app discovery and search; a browser for displaying content with complex layouts and shared context; and a rich, actionable notification center, experiences that accelerate time to market, and easy-to-use APIs for hyper-customization.</p> <p><strong>Built by OpenFin.</strong> Workspace components are built and managed by OpenFin. We enable customization through configuration. <strong>Component Hosting.</strong> Workspace components are hosted securely on OpenFin’s CDN. <strong>Desktop Owner Settings.</strong> Workspace can be configured using the OpenFin Desktop Owner Settings file. For example, this allows you to point Workspace at your own Content Store. <strong>Content Discovery Service.</strong> By default, Workspace components are configured to point at OpenFin’s Content Discovery Service, which powers the OpenFin Content Store. You can point Workspace at your own Content Discovery Service hosted on your own infrastructure. <strong>Versioning.</strong> Each release of Workspace is versioned. You can lock the version of Workspace that you run via Desktop Owner Settings.</p> </blockquote> <h2>What is OpenFin Container?</h2> <p>OpenFin Container is a desktop runtime application environment that enables communication between all of your apps (web, legacy, third-party) for the enterprise desktop. It is built on top of Google Chromium and GitHub Electron and combines advanced interoperability, lightning-fast distribution, and robust security with an agile web development and deployment model.</p> <p>Any web app that runs in Google Chrome can run on OpenFin OS unmodified and in a matter of seconds. With OpenFin, developers write 99% of their code with standard HTML5 and the JavaScript frameworks of their choice, including React and Angular. OpenFin APIs (usually &lt;1% of code) enable windowing, interoperability and system access.</p> <h3>Platform Provider</h3> <p>The Platform Provider is the main application window which is the communication hub that coordinates between all the windows in a Platform Application. The provider runs in a hidden window and enables application providers to extend or overwrite default Platform behavior through Platform Overrides. This is where you can control how your application will look and feel to create a custom branded experience.</p> <h3>Platform Window</h3> <p>Platform Windows act as a “frame” for your Platform Views and can be leveraged to display UI controllers such as Minimize, Maximize and Close. These are child windows of the Provider and may contain 1 or many Platform Views.</p> <h3>Platform View</h3> <p>Platform Views are your applications content and they reside within a Platform Window. The content (a web application) is loaded into a view and attached to a Platform Window. Views have their own JavaScript context that is distinct and unconnected to the window’s context, that have no DOM representation within the window. This allows views to move between windows without refreshing or otherwise destroying the context. Views can be tabbed/tabless and provide the end-user ease to arrange the layout of a Platform Window. For optimal performances, it is worth noting that views can be split into as many render processes as needed through processAffinity.</p> <h3>Layout Management</h3> <p>Layout management enables applications providers to programmatically embed multiple views or “web apps” in a single Platform Window.</p> <h3>Snapshot</h3> <p>A Snapshot defines the Platform Window and View configuration to be launched into a Platform. When added as a top-level option in your Platform manifest, the Platform will launch with that Snapshot by default. Snapshots also enable application providers the ability to replenish/restore saved configurations while the Platform is running.</p> <h3>Platform Manifest</h3> <p>This is the file that defines how an OpenFin Platform works and tells the OpenFin Runtime what your workspace will look like at Launch.</p> <h3>Process Affinity</h3> <p>By default, all child windows of a Platform Provider share a single renderer process, based on same-origin policy. Views can be separated into further processes by providing a processAffinity option. This enables application providers to have two or more views share a process separate from other views. This can be quite useful in architecting your application for views that have a heavier memory profile.</p> <hr> <p><strong>Content Discovery Service.</strong> "Implements the GET methods of the FDC3 App Directory specification, with some additional optional properties for defining the content items. A content discovery service can provide apps or workspaces."</p> https://tedneward.github.io/Research/platforms/oqtane/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/oqtane/index.html <p><a href="https://www.oqtane.org/">Website</a> | <a href="https://github.com/oqtane/framework">Github</a></p> https://tedneward.github.io/Research/platforms/photonos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/photonos/index.html <p><a href="https://vmware.github.io/photon/">Website</a> | <a href="https://github.com/vmware/photon">Source</a></p> https://tedneward.github.io/Research/platforms/play/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/play/index.html <p><a href="https://www.createwithplay.com/">Website</a></p> https://tedneward.github.io/Research/platforms/prefect/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/prefect/index.html <p><a href="https://www.prefect.io/">Website</a> | <a href="https://github.com/prefecthq/prefect">Source</a></p> <p>Hello world</p> <pre><code class="language-python">from prefect import task, Flow @task def say_hello(): print("Hello, World!") with Flow("My First Flow") as flow: say_hello() flow.run() # "Hello, world!" </code></pre> <p>Source appears to be <a href="https://www.prefect.io/core/">PrefectCore</a> (Python library), which suggests a "freemium" open-source model.</p> <p>Features described:</p> <ul> <li>Realtime UI: The Prefect UI updates in realtime so you're never behind.</li> <li>Universal Deploy: Anywhere you can run Python, you can run Prefect. Instantly deploy your flows and monitor runs from Prefect's UI, no Docker required.</li> <li>Flow code: Prefect flows are plain old Python, so you can build and modify them however you like.</li> <li>Parameters: Add parameters to any flow for easy runtime templating and reuse.</li> <li>Robust states: Prefect handles every error, whether expected or not. Some tasks might only run if upstream tasks fail.</li> <li>Dataflow: Pass data between tasks for complex processing and advanced analytics.</li> <li>Mapping: Powerful map/reduce operators generate dynamic tasks for each element of an input. Mapped tasks can be linked to create parallel pipelines.</li> <li>Environments: A flexible environment model means flows can be deployed anywhere from a laptop to multi-cloud clusters.</li> <li>Realtime: When paired with Dask, Prefect's event-driven scheduler can execute tasks with millisecond latency.</li> <li>Time Travel: Prefect task outputs can be cached or updated at different intervals, even within the same workflow.</li> <li>Result Handlers: Serialize data in and out of your tasks with customizable result handlers, including local filesystems, S3, and GCS.</li> <li>Custom Schedules: Specify custom schedule logic including business days, offsets, and blackout windows, or fall back on good old cron.</li> <li>Looping: Loop tasks with arbitrary control logic.</li> <li>Depth-First Execution: Race through mapped pipelines by allowing tasks to start before all tasks of the previous stage have finished.</li> <li>COMING SOON -- Event-Driven Flows: Fire off flow runs in response to external events of any frequency.</li> <li>COMING SOON -- Task Affinity: Run each of a flow's tasks in a completely different environment, including new dependencies or platforms.</li> </ul> https://tedneward.github.io/Research/platforms/qemu/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/qemu/index.html <p><a href="https://airbus-seclab.github.io/qemu_blog/">Qemu Internals</a></p> https://tedneward.github.io/Research/platforms/quix/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/quix/index.html <p><a href="https://quix.io/docs/get-started/welcome.html">Website</a> | <a href="https://github.com/quixio">Source</a></p> https://tedneward.github.io/Research/platforms/redox/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/redox/index.html <p><a href="https://www.redox-os.org/">Website</a> | <a href="https://gitlab.redox-os.org/redox-os/redox/">Source</a></p> https://tedneward.github.io/Research/platforms/rhodes/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/rhodes/index.html <p><a href="https://tau-platform.com/en/">Website</a> | <a href="https://github.com/rhomobile">Source</a></p> <h2>Overview</h2> <p>The Rhodes framework is a platform for building locally executing, device-optimized mobile applications for all major smartphone devices.</p> <h3>Installation</h3> <p><a href="http://docs.tau-platform.com/en/7.5/guide/rhomobile-install#windows-first-time-install-32--and-64-bit">Windows First-time Install (32- and 64-bit)</a> - How to install RhoMobile Suite on Windows (32- and 64-bit).</p> <p><a href="http://docs.tau-platform.com/en/7.5/guide/rhomobile-install#mac-os-x-first-time-install">Mac OS X First-time Install</a> - How to install RhoMobile Suite on Mac OS.</p> <h3>Building apps</h3> <p><a href="http://docs.tau-platform.com/en/7.5/guide/build_android">Build for Android</a> - How to build apps for Android.</p> <p><a href="http://docs.tau-platform.com/en/7.5/guide/build_ios">Build for iOS</a> - How to build apps for iOS.</p> <p><a href="http://docs.tau-platform.com/en/7.5/guide/build_win">Building for Windows</a> - How to build apps for Windows.</p> <h3>RhoConnect Push Synchronization</h3> <p><a href="http://docs.tau-platform.com/en/7.5/rhoconnect/push">Introduction to Push</a></p> <p><a href="http://docs.tau-platform.com/en/7.5/rhoconnect/push-client-setup-android">Set up Google Cloud Messaging on client and server for Rhodes Android Device</a> - Setting Up RhoConnect Push-Based Synchronization with Google Cloud Messaging.</p> <p><a href="http://docs.tau-platform.com/en/7.5/rhoconnect/push-client-setup-ios">Set up Apple Push Notification Service on client and server for Rhodes iOS Device</a> - Setting Up RhoConnect Push-Based Synchronization with Apple Push Notification Service.</p> <p><a href="http://docs.tau-platform.com/en/7.5/rhoconnect/push-client-setup-rps">Set up RhoConnect Push Service on client and server for Rhodes Android and Windows Mobile/CE Devices</a> - Setting Up for RhoConnect Push Service on Client Application for Windows Mobile/CE Devices.</p> <h3>Android Docs</h3> <p><a href="https://github.com/rhomobile/rhodes/blob/master/doc/oss/Barcode_support_doc.md">Guidelines To Support Barcode API On Android JellyBean Devices</a> - Lists the Zebra Android devices capable of barcode scanning, their OS image versions and steps for activating the feature.</p> <h3>Repositories Released as Open Source</h3> <p>Clone these repo(s) as required for your needs to generate a copy of the open source code to the local development machine.</p> <p><a href="https://github.com/rhomobile/rhodes/">Rhodes</a></p> <p><a href="https://github.com/rhomobile/rhoconnect/">RhoConnect</a></p> <p><a href="https://github.com/rhomobile/rhoconnect-client/">RhoConnect Client</a></p> <p><a href="https://github.com/rhomobile/rhoconnect-push">RhoConnect Push</a></p> <p><a href="https://github.com/rhomobile/rhoconnect-push-service">RhoConnect Push Service</a></p> <p><a href="https://github.com/rhomobile/rhostudio/">RhoStudio</a></p> <p><a href="https://github.com/rhomobile/rhoinstaller/">RhoInstaller</a></p> <p><a href="https://github.com/rhomobile/RMS-Testing">RMS-Testing</a></p> <h2>Reading</h2> <p><a href="https://reintech.io/blog/developing-cross-platform-mobile-apps-with-ruby">Developing Cross-platform mobile apps with Ruby</a></p> https://tedneward.github.io/Research/platforms/saltcorn/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/saltcorn/index.html <p><a href="https://saltcorn.com/">Website</a> | <a href="https://github.com/saltcorn/saltcorn">Source</a></p> https://tedneward.github.io/Research/platforms/sharemind/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/sharemind/index.html <p><a href="https://sharemind.cyber.ee/">Website</a> | <a href="https://github.com/sharemind-sdk/sharemind-sdk.github.io">SDK</a> | <a href="https://github.com/sharemind-sdk/vm_m4">VM</a></p> <h2>Tools:</h2> <ul> <li><a href="https://github.com/sharemind-sdk/sbdump">sbdump</a>: Sharemind Executable Disassembler tool</li> <li><a href="https://github.com/sharemind-sdk/secrec">secrec</a>: Sharemind SecreC Compiler and Analyzer</li> <li><a href="https://github.com/sharemind-sdk/secrec-stdlib">secrec-stdlib</a></li> </ul> https://tedneward.github.io/Research/platforms/spacecloud/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/spacecloud/index.html <p><a href="https://space-cloud.io/">Website</a> | <a href="https://github.com/spacecloud-io/space-cloud">Source</a></p> https://tedneward.github.io/Research/platforms/sulu/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/sulu/index.html <p><a href="https://sulu.io/">Website</a> | <a href="https://github.com/sulu/sulu">Source</a></p> https://tedneward.github.io/Research/platforms/tangled/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tangled/index.html <p><a href="https://tangled.sh/">Website</a> | <a href="https://tangled.sh/@tangled.sh/core">Source</a></p> <h2>Reading</h2> <p><a href="https://anil.recoil.org/notes/disentangling-git-with-bluesky">Socially self-hosting source code with Tangled on Bluesky</a></p> https://tedneward.github.io/Research/platforms/tooljet/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tooljet/index.html <p><a href="https://tooljet.io/">Website</a> | <a href="https://github.com/tooljet/tooljet">Source</a></p> <blockquote> <p>You can connect to your data sources such as databases ( like PostgreSQL, MongoDB, Elasticsearch, etc ), API endpoints ( ToolJet supports importing OpenAPI spec &amp; OAuth2 authorization) and external services ( like Stripe, Slack, Google Sheets, Airtable ) and use our pre-built UI widgets to build internal tools.</p> </blockquote> <p>Features</p> <ul> <li>Visual app builder with widgets such as tables, charts, modals, buttons, dropdowns and more</li> <li>Mobile 📱 &amp; desktop layouts 🖥</li> <li>Connect to databases, APIs and external services</li> <li>Deploy on-premise ( supports docker, kubernetes, heroku and more )</li> <li>Granular access control on organization level and app level</li> <li>Write JS code almost anywhere in the builder</li> <li>Query editors for all supported data sources</li> <li>Transform query results using JS code</li> <li>Import endpoints from OpenAPI specs</li> <li>All the credentials are securely encrypted using aes-256-gcm.</li> <li>ToolJet acts only as a proxy and doesn't store any data.</li> <li>Support for OAuth</li> </ul> <p><a href="https://www.marktechpost.com/2023/10/13/meet-tooljet-an-open-source-low-code-framework-to-build-and-deploy-internal-tools-with-minimal-engineering-effort/">"Meet ToolJet"</a></p> https://tedneward.github.io/Research/platforms/uibakery/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/uibakery/index.html <p>Glue your databases, APIs, and third-party services together with beautiful UI components. Write custom JavaScript code if needed almost anywhere inside your web app.</p> <p><a href="https://uibakery.io/">Website</a> | <a href="https://github.com/akveo">Github links</a> (not sure it's source to UIBakery)</p> <p>Features:</p> <ul> <li>Autogenerate UI components: Automatically configure Tables, Forms, Charts, Maps. Mirror a data structure into UI components.</li> <li>Connect, read and write your data: Natively connect a UI to data sources. Read, write, search data using built-in functions, SQL queries, HTTP requests.</li> <li>Create workflows and business logic: Add sequences and conditions, create advanced business logic for your internal app. Modify everything visually.</li> <li>Add custom code &amp; create new components: Connect JavaScript libraries. Add custom code to map data. Build components using React or jQuery.</li> <li>Debug and test your app: Debug app state, handle runtime errors in the development stage. Get error notifications – we’ll help you fix them.</li> <li>Publish your internal tool: No manual app deployment, hosting, and environment setup. Publish an app securely, invite your end-users.</li> </ul> https://tedneward.github.io/Research/platforms/macos/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/macos/index/index.html <ul> <li><a href="../macos/usage">Usage</a></li> <li><a href="../macos/vm">Virtual machine images</a></li> <li><a href="../macos/development">Development</a></li> </ul> <h2>Miscellaneous tools</h2> <p><a href="https://github.com/HamburgChimps/apple-notes-liberator">Apple Notes Liberator</a>: Free your Apple Notes data from Notes.app</p> https://tedneward.github.io/Research/platforms/macos/vm/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/macos/vm/index.html <h1>Docker</h1> <p><a href="https://github.com/sickcodes">sick.codes</a>: collection of interesting macOS-related repositories</p> <ul> <li><a href="https://github.com/sickcodes/Docker-OSX">Docker-OSX</a>: Run macOS VM in a Docker! Run near native OSX-KVM in Docker! X11 Forwarding! CI/CD for OS X Security Research! Docker mac Containers. <ul> <li><a href="https://github.com/sickcodes/Docker-eyeOS">Docker-eyeOS</a>: Run iPhone (xnu-arm64) in a Docker container! Supports KVM + iOS kernel debugging (GDB)! Run xnu-qemu-arm64 in Docker! Works on ANY device.</li> <li><a href="https://github.com/sickcodes/osx-optimizer">osx-optimizer</a>: OSX Optimizer: Optimize MacOS - Shell scripts to speed up your mac boot time, accelerate loading, and prevent unnecessary throttling.</li> </ul> </li> </ul> <h1>VirtualBox</h1> <p><a href="https://www.xda-developers.com/how-install-macos-virtualbox/">"How to install macOS in VirtualBox"</a></p> <h2>Running macOS in a VirtualBox VM</h2> <p>(Some details swiped from <a href="https://techsviewer.com/install-macos-10-15-catalina-on-virtualbox-on-windows-pc/">https://techsviewer.com/install-macos-10-15-catalina-on-virtualbox-on-windows-pc/</a>)</p> <p>Basic steps:<br> * Make sure Virtual Box Extension Pack is installed. Necessary (?) for USB setting below.</p> <ul> <li> <p><em>Obtain a macOS ISO.</em> The most "legal" thing to do seems to be to download the "Install macOS {version}" where "{version}" is Catalina, High Sierra, whatever, and then convert the installer application into an ISO via <a href="https://gist.github.com/tedneward/5d4710983caf55906f834dc7c576a418">this Gist</a>.</p> </li> <li> <p><em>Create the VBox Guest.</em> Select "Type: macOS" and "Version: (Latest macOS version)", where the latest macOS version offered was High Sierra when I tried this. Set RAM to be high--I chose 16GB on my 64GB MBP--and create a hard disk--I chose 100GB dynamic. Then set the following options:</p> </li> <li>"System"/"Motherboard": set "Chipset: PIIX 3"; set "Enable EFI" on.</li> <li> <p>"System"/"Processor": set "Cores" to 2 (or 4, if you can); set "Enable PAE/NX" on.</p> </li> <li> <p>"Display"/"Screen": set "Video Memory" to 128MB; set "Graphics Controller" to "VMSVGA". (On my running VBox right now, it's set to VBoxVGA, and wouldn't be set to anything else, so this may only be required for Windows machines. Verify.); I left 3D and 2D acceleration turned off.</p> </li> <li> <p>"Ports"/"USB": set "Enable USB Controller" on; select "USB 3.0 (xHCI) Controller". Apparently macOS doesn't support USB 2.0 or 1.1 Controllers.</p> </li> </ul> <p>Then mount the macOS ISO as the ISO image.</p> <ul> <li><em>Tweak the VBox settings by hand.</em> Next run the following shell commands (with VirtualBox on the command-line path):</li> </ul> <pre><code>VBoxManage modifyvm "Your VM Name" --cpuidset 00000001 000106e5 00100800 0098e3fd bfebfbff VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemProduct" "iMac11,3" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiSystemVersion" "1.0" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/efi/0/Config/DmiBoardProduct" "Iloveapple" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/DeviceKey" "ourhardworkbythesewordsguardedpleasedontsteal(c)AppleComputerInc" VBoxManage setextradata "Your VM Name" "VBoxInternal/Devices/smc/0/Config/GetKeyFromRealSMC" 1 </code></pre> <p>No clue what these do but they appear to be necessary. I suspect they're Apple anti-piracy settings.</p> <ul> <li> <p><em>Boot the ISO.</em> The ISO won't immediately install to the created hard drive because it needs to be formatted first. From the "macOS Utilities" dialog select "Disk Utility" and Erase the VirtualBox hard drive as "Format: MacOS Extended (Journaled)" and "Scheme: GUID Partition Map". Name can be whatever. When that's done, quit "Disk Utility" and "Install macOS" to the VBox hard drive.</p> </li> <li> <p><em>Finish the macOS install.</em> Remove the ISO, boot the VBox, and it should begin the installation process as per normal installation procedures. Create an account, and by the time that's all done, it should be good to go.</p> </li> </ul> https://tedneward.github.io/Research/platforms/meteor/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/meteor/index.html <p><a href="https://www.meteor.com/">Website</a> | <a href="https://docs.meteor.com/#/full/">Docs</a> | <a href="https://github.com/meteor/meteor">Source</a></p> <p>Meteor is a full-stack JavaScript platform for developing modern web and mobile applications. Meteor includes a key set of technologies for building connected-client reactive applications, a build tool, and a curated set of packages from the Node.js and general JavaScript community.</p> <ul> <li> <p>Meteor allows you to develop in one language, JavaScript, in all environments: application server, web browser, and mobile device.</p> </li> <li> <p>Meteor uses data on the wire, meaning the server sends data, not HTML, and the client renders it.</p> </li> <li> <p>Meteor embraces the ecosystem, bringing the best parts of the extremely active JavaScript community to you in a careful and considered way.</p> </li> <li> <p>Meteor provides full stack reactivity, allowing your UI to seamlessly reflect the true state of the world with minimal development effort.</p> </li> </ul> https://tedneward.github.io/Research/platforms/mobile/caylixos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/caylixos/index.html <p><a href="https://calyxos.org/">Website</a></p> https://tedneward.github.io/Research/platforms/mobile/replicant/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/replicant/index.html <p><a href="https://www.replicant.us/">Website</a></p> https://tedneward.github.io/Research/platforms/netlifycms/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/netlifycms/index.html <p><a href="https://www.netlifycms.org/">Website</a> | <a href="https://github.com/netlify/netlify-cms">Source</a></p> <p>Seems related to <a href="/clouds/netlify">Netlify</a>?</p> https://tedneward.github.io/Research/platforms/nscde/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/nscde/index.html <p><a href="https://github.com/NsCDE/NsCDE">Source</a></p> <p>One of the main aspects of Linux that attracts users is how much choice it offers. Not so Common Desktop Environment is a desktop environment to run over a Linux installation that looks and feels like the Common Desktop Environment (CDE) that ran on HP Unix systems. Enjoy a desktop full of 90s-era pastel shades while enjoying the modern conveniences of a recent Linux distro.</p> https://tedneward.github.io/Research/platforms/openindiana/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/openindiana/index.html <p><a href="https://openindiana.org/">Website</a> | <a href="https://github.com/openindiana">Source</a> | <a href="https://docs.openindiana.org/">Docs</a> <em>(very sketchy/WIP as of August 2025)</em> and <a href="https://docs.openindiana.org/books/about/">OpenSolaris books</a></p> <h2>What is illumos?</h2> <p>illumos is a collection of software that forms the core of an Operating System. It includes the kernel, device drivers, core system libraries, and utilities. Conceptually, the illumos idea of "the operating system" lies between something like Linux (which is the kernel only; all of userspace is delivered by vendors in a "distribution") and the BSD family of operating systems, which are packaged as a complete unit (kernel, core libraries, userspace utilities, and even end-user software packages).</p> <p>illumos is the home of many technologies including ZFS, DTrace, Zones, ctf, FMA, and more. We pride ourselves on having a stable, highly observable, and technologically different system. Finally, illumos has a proud engineering heritage, tracing it roots back through Sun Microsystems to the original releases of UNIX and BSD.</p> https://tedneward.github.io/Research/platforms/oracle-apex/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/oracle-apex/index.html <p><a href="https://apex.oracle.com/en/">Website</a> Commercial</p> https://tedneward.github.io/Research/platforms/plan9/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/plan9/index.html <p><a href="http://plan9.bell-labs.com/plan9/">Website</a> | <a href="http://doc.cat-v.org/plan_9/">Docs website</a></p> <p>There's a lot of interesting ideas hiding out in here.</p> <h1><a href="http://doc.cat-v.org/plan_9/1st_edition/">First Edition</a></h1> <ul> <li><a href="http://doc.cat-v.org/plan_9/1st_edition/designing_plan_9">Designing Plan 9 - Bell Labs' Plan 9 research project looks to tomorrow</a> by Rob Pike, Dave Presotto, Ken Thompson, and Howard Trickey.</li> <li><a href="http://doc.cat-v.org/plan_9/1st_edition/help/">Help: A Minimalist Global User Interface</a> by Rob Pike</li> <li><a href="http://doc.cat-v.org/plan_9/1st_edition/cda/">Circuit Design Aids (CDA) on Plan 9</a> by A. G. Hume, M. Kahrs, T. J. Killian.</li> <li><a href="http://doc.cat-v.org/plan_9/1st_edition/multiprocessor-streams">Multiprocessor Streams for Plan 9</a> by David Leo Presotto.</li> <li><a href="http://doc.cat-v.org/plan_9/1st_edition/manual.pdf">The complete Plan 9 from Bell Labs Programmer’s Manual First Edition</a></li> </ul> <h1><a href="http://doc.cat-v.org/plan_9/2nd_edition/">Second Edition</a></h1> <ul> <li><a href="http://doc.cat-v.org/plan_9/2nd_edition/README">The "README"</a> by Brian Kernighan - somewhat outdated, but still relevant today.</li> <li><a href="/languages/alef">Alef</a></li> <li><a href="http://doc.cat-v.org/plan_9/2nd_edition/papers/panel/">A Quick Introduction to the Panel Library</a> by Tom Duff - used by the Mothra web browser</li> <li><a href="http://doc.cat-v.org/plan_9/2nd_edition/papers/gfx/">Raster Graphics in plan9</a> by Tom Duff - Ironically obsoleted the adoption in 3rd Edition of the Porter/Duff graphics model.</li> </ul> <h1><a href="http://doc.cat-v.org/plan_9/3rd_edition/">Third Edition</a></h1> <ul> <li><a href="http://doc.cat-v.org/plan_9/3rd_edition/rio/">Rio: Design of a Concurrent Window System</a></li> <li><a href="http://doc.cat-v.org/bell_labs/concurrent_window_system/">A Concurrent Window System</a></li> <li><a href="http://doc.cat-v.org/bell_labs/transparent_wsys/">Window Systems Should Be Transparent</a></li> </ul> <h1><a href="http://doc.cat-v.org/plan_9/4th_edition/">Fourth Edition</a></h1> <ul> <li><a href="http://doc.cat-v.org/plan_9/4th_edition/papers/">Papers</a></li> <li><a href="http://man.cat-v.org/plan_9/">Manual Pages</a></li> </ul> https://tedneward.github.io/Research/platforms/porter/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/porter/index.html <p><a href="https://porter.run/">Website</a> | <a href="https://github.com/porter-dev/porter">Source</a></p> https://tedneward.github.io/Research/platforms/projectreactor/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/projectreactor/index.html <p><a href="https://projectreactor.io/">Website</a> | <a href="https://github.com/reactor">Source</a></p> <p>Several sub-projects:<br> * <a href="https://github.com/reactor/reactor-core">Reactor Core</a>: A Reactive Streams foundation for Java 8<br> * <a href="https://github.com/reactor/reactor-core/tree/master/reactor-test">Reactor Test</a>: Test utilities<br> * <a href="https://github.com/reactor/reactor-addons/#reactor-extra">Reactor Extra</a>: Additional operators for Flux<br> * <a href="https://github.com/reactor/reactor-netty">Reactor Netty</a>: HTTP, TCP, UDP Clients/Servers using Netty<br> * <a href="https://github.com/reactor/reactor-addons/#reactor-adapter">Reactor Adapter</a>: Adapt to/from other reactive libraries<br> * <a href="https://github.com/reactor/reactor-kafka">Reactor Kafka</a>: Reactive bridge to Apache Kafka<br> * <a href="https://github.com/reactor/reactor-kotlin-extensions">Reactor Kotlin Extensions</a>: Kotlin extensions for all Java artifacts in Dysprosium+<br> * <a href="https://github.com/reactor/reactor-rabbitmq">Reactor RabbitMQ</a>: Reactive bridge to RabbitMQ<br> * <a href="https://github.com/reactor/reactor-pool">Reactor Pool</a>: Generic object pool for reactive applications<br> * <a href="https://github.com/reactor/reactor-core-dotnet">Reactor Core.NET</a>: Incubating Reactive Streams foundation for .NET<br> * <a href="https://github.com/reactor/reactor-core-js">Reactor CoreJS</a>: Incubating Reactive Streams foundation for JavaScript</p> https://tedneward.github.io/Research/platforms/qovery/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/qovery/index.html <p><a href="https://www.qovery.com/">Website</a> | <a href="https://github.com/Qovery/engine">Source</a></p> https://tedneward.github.io/Research/platforms/racket/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/racket/index.html <p><a href="https://racket-lang.org/">Website</a> | <a href="https://docs.racket-lang.org/index.html">Docs</a> | <a href="https://github.com/racket/racket">Source</a></p> <p>General-purpose programming language, as well as ecosystem for language-oriented programming. Languages available (through the <code>#lang</code> directive):</p> <ul> <li>racket</li> <li>scribble/base</li> <li>typed/racket</li> <li><a href="datalog.html">datalog</a></li> <li>racket/gui</li> <li>web-server/insta</li> <li>plait (plai language with a type system close to that of ML)</li> <li>profj (ProfessorJ: Java in Racket)</li> <li>racklog (Prolog-style logic programming)</li> </ul> <p>Really broad ecosystem; deserves much more attention than it gets.</p> <p><a href="https://docs.racket-lang.org/quick/index.html">"An introduction to Racket, with pictures"</a></p> <p>Example: HelloWorld</p> <pre><code class="language-lisp">#lang racket/gui (define my-language 'English) (define translations #hash([English . "Hello world"] [French . "Bonjour le monde"] [German . "Hallo Welt"] [Greek . "Γειά σου, κόσμε"] [Portuguese . "Olá mundo"] [Spanish . "Hola mundo"] [Thai . "สวัสดีชาวโลก"])) (define my-hello-world (hash-ref translations my-language "hello world")) (message-box "" my-hello-world) </code></pre> <p>Example: Language-Oriented Programming</p> <pre><code>#lang typed/racket ;; Using higher-order occurrence typing (define-type SrN (U String Number)) (: tog ((Listof SrN) -&gt; String)) (define (tog l) (apply string-append (filter string? l))) (tog (list 5 "hello " 1/2 "world" (sqrt -1))) </code></pre> <pre><code>#lang scribble/base @; Generate a PDF or HTML document @(require (only-in racket ~a)) @(define N 99) @title{Bottles: @italic{Abridged}} @(apply itemlist (for/list ([n (in-range N 0 -1)]) @item{@(~a n) bottles.})) </code></pre> <pre><code>#lang datalog ancestor(A, B) :- parent(A, B). ancestor(A, B) :- parent(A, C), ancestor(C, B). parent(john, douglas). parent(bob, john). ancestor(A, B)? </code></pre> https://tedneward.github.io/Research/platforms/relate/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/relate/index.html <p><a href="https://relate.app/">Website</a> | Commercial with free tier</p> https://tedneward.github.io/Research/platforms/rintagi/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/rintagi/index.html <p><a href="https://www.rintagi.com/">Website</a> | <a href="https://github.com/Rintagi/Low-Code-Development-Platform">Source</a></p> https://tedneward.github.io/Research/platforms/seatable/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/seatable/index.html <p><a href="https://seatable.io/">Website</a> | <a href="https://github.com/seatable/seatable">Source</a> | <a href="https://manual.seatable.io/">Docs</a></p> <p><a href="https://manual.seatable.io/docker/Developer-Edition/Deploy%20SeaTable-DE%20with%20Docker/">Developer (free) Edition deployed with Docker</a></p> <h4>Software components</h4> <p>SeaTable consists of following component</p> <ul> <li>dtable-web: The web site for manage tables.</li> <li>dtable-server: Store the tables and provide collaborating feature.</li> <li>dtable-events: Background tasks likes email sending and so on.</li> <li>seaf-server: Store attachments (files and images)</li> </ul> https://tedneward.github.io/Research/platforms/simpl/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/simpl/index.html <p><a href="https://tpso.com/repo/webdemo/start.html#0">Website</a> | <a href="https://github.com/tpso-src/simpl4-src">Source</a></p> https://tedneward.github.io/Research/platforms/stackstorm/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/stackstorm/index.html <p><a href="https://stackstorm.com/">Website</a> | <a href="https://github.com/StackStorm/st2">Source</a></p> <p>Event-driven automation for auto-remediation, security responses, troubleshooting, deployments, and more. Includes rules engine, workflow, 160 integration packs with 6000+ actions (see <a href="https://exchange.stackstorm.org">https://exchange.stackstorm.org</a>) and ChatOps.</p> https://tedneward.github.io/Research/platforms/supercollider/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/supercollider/index.html <p><a href="http://supercollider.github.io/">Website</a> | <a href="https://github.com/supercollider/supercollider">Source</a> | <a href="https://mitpress.mit.edu/9780262232692/the-supercollider-book/">The Supercollider Book</a> (MIT Press)</p> <p>Programming language, IDE, and platform for building audio/sounds.</p> https://tedneward.github.io/Research/platforms/tikitrackers/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tikitrackers/index.html <p><a href="https://tiki.org/HomePage">Website</a> | <a href="https://gitlab.com/tikiwiki/tiki/">Source</a></p> <p><a href="https://doc.tiki.org/Features">Feature list</a>:</p> <ul> <li>Wiki pages</li> <li>Trackers (web forms and reports)</li> <li>Blogs</li> <li>Forums</li> <li>RSS Syndication</li> <li>WYSIWYG Editing</li> <li>Calendars and Events</li> <li>File and Image Galleries</li> <li>User and Group Management</li> <li>Surveys, Quizzes, and Polls</li> </ul> https://tedneward.github.io/Research/platforms/totaljs/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/totaljs/index.html <p><a href="https://www.totaljs.com/">Website</a> | <a href="https://github.com/totaljs/">Source</a></p> <p>Competitor to <a href="/languages/nodered/">Node-RED</a></p> https://tedneward.github.io/Research/platforms/universe/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/universe/index.html <p><a href="https://www.onuniverse.com/">Website</a> Looks to be iOS/macOS-specific.</p> https://tedneward.github.io/Research/platforms/macos/oss/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/macos/oss/index.html <p><a href="https://opensource.apple.com/releases/">Apple Open Source</a></p> https://tedneward.github.io/Research/platforms/mastodon/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mastodon/index.html <p><a href="https://joinmastodon.org/">Website</a> | <a href="https://github.com/tootsuite/mastodon">Source</a> or <a href="https://github.com/mastodon/mastodon">Source</a></p> <p>A Vagrant configuration is included for development purposes. To use it, complete following steps:</p> <ul> <li>Install Vagrant and Virtualbox</li> <li>Run <code>vagrant up</code></li> <li>Run <code>vagrant ssh -c "cd /vagrant &amp;&amp; foreman start"</code></li> <li>Open <a href="http://mastodon.local">http://mastodon.local</a> in your browser</li> </ul> https://tedneward.github.io/Research/platforms/mingw32/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mingw32/index.html <p><a href="http://www.mingw.org/">Website</a></p> https://tedneward.github.io/Research/platforms/mobile/e/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/e/index.html <p><a href="https://e.foundation">Website</a> | <a href="https://gitlab.e.foundation/e">Source</a></p> <p>Article: <a href="https://thenewleafjournal.com/review-of-e-an-android-alternative-for-mobile-phones/">Review of /e/ -- An Android Alternative for Mobile Phones</a></p> https://tedneward.github.io/Research/platforms/mobile/ubuntutouch/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/mobile/ubuntutouch/index.html <p><a href="https://ubuntu-touch.io/">Website</a></p> https://tedneward.github.io/Research/platforms/newjs/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/newjs/index.html <p><a href="https://nwjs.io/">Website</a> | <a href="https://github.com/nwjs/nw.js">Source</a></p> <h2>Getting Started</h2> <p>NW.js is based on <a href="http://www.chromium.org">Chromium</a> and <a href="http://nodejs.org/">Node.js</a>. It lets you call Node.js code and modules directly from browser and also use Web technologies in your app. Further, you can easily package a web application to a native application.</p> <h3>Get NW.js</h3> <p>You can get the latest binaries from official website <a href="http://nwjs.io">http://nwjs.io</a>.</p> <h3>Write NW.js App</h3> <h4>Example 1 - Hello World</h4> <p>This the basic example shows how to write an NW.js app.</p> <p><strong>Step 1.</strong> Create <code>package.json</code>:</p> <pre><code class="language-json">{ "name": "helloworld", "main": "index.html" } </code></pre> <p><code>package.json</code> is the manifest file in your app. It is written in <a href="http://www.json.org/">JSON format</a>. The <code>main</code> field figures out the first page opened by the NW.js, i.e. <code>"index.html"</code> in this example. And the <code>name</code> field is the unique name used among NW.js apps. See [Manifest Format](../References/Manifest Format.md) for more details.</p> <p><strong>Step 2.</strong> Create <code>index.html</code>:</p> <pre><code class="language-html">&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;Hello World!&lt;/title&gt; &lt;/head&gt; &lt;body&gt; &lt;h1&gt;Hello World!&lt;/h1&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>This is the normal HTML file. You can use any web technologies supported by latest browsers.</p> <p>!!! note "Chromium Specific Features"<br> NW.js is based on Chromium, which also enables you to use Chrome specific features. Such as <a href="https://developer.mozilla.org/en-US/docs/Web/API/File_System_API">File System API</a>, experimental CSS styles with <code>-webkit-</code> prefix. <strong>Be careful with these non-standard features since they may be deprecated shortly.</strong></p> <p><strong>Step 3.</strong> Run your app</p> <pre><code class="language-bash">cd /path/to/your/app /path/to/nw . </code></pre> <p><code>/path/to/nw</code> is the binary file of NW.js. On Windows, it's <code>nw.exe</code>; On Linux, it's <code>nw</code>; On Mac, it's <code>nwjs.app/Contents/MacOS/nwjs</code>.</p> <p>!!! tip "Drag &amp; Drop on Windows"<br> On Windows, you can drag the <code>folder containing package.json</code> to <code>nw.exe</code> to run your app.</p> <h4>Example 2 - Using NW.js APIs</h4> <p>All NW.js APIs are loaded in <code>nw</code> object globally and can be used directly in JavaScript files.</p> <p>This example shows how to create a native context menu in your NW.js app. You can create <code>index.html</code> with following content:</p> <pre><code class="language-html">&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;Context Menu&lt;/title&gt; &lt;/head&gt; &lt;body style="width: 100%; height: 100%;"&gt; &lt;p&gt;'Right click' to show context menu.&lt;/p&gt; &lt;script&gt; // Create an empty context menu var menu = new nw.Menu(); // Add some items with label menu.append(new nw.MenuItem({ label: 'Item A', click: function(){ alert('You have clicked at "Item A"'); } })); menu.append(new nw.MenuItem({ label: 'Item B' })); menu.append(new nw.MenuItem({ type: 'separator' })); menu.append(new nw.MenuItem({ label: 'Item C' })); // Hooks the "contextmenu" event document.body.addEventListener('contextmenu', function(ev) { // Prevent showing default context menu ev.preventDefault(); // Popup the native context menu at place you click menu.popup(ev.x, ev.y); return false; }, false); &lt;/script&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>... then run your app:</p> <pre><code class="language-bash">cd /path/to/your/app /path/to/nw . </code></pre> <p>!!! tip "require('nw.gui')"<br> The legacy way of loading NW.js APIs using <code>require('nw.gui')</code> is also supported. It returns the same <code>nw</code> object.</p> <h4>Example 3 - Using Node.js API</h4> <p>You can call node.js and modules directly from the DOM. So it enables endless possibilities for writing apps with nw.js.</p> <p>This example shows how to query the OS platform with <code>os</code> module of Node.js. Simply create the <code>index.html</code> file with following content and run it with NW.js.</p> <pre><code class="language-html">&lt;!DOCTYPE html&gt; &lt;html&gt; &lt;head&gt; &lt;title&gt;My OS Platform&lt;/title&gt; &lt;/head&gt; &lt;body&gt; &lt;script&gt; // get the system platform using node.js var os = require('os'); document.write('You are running on ', os.platform()); &lt;/script&gt; &lt;/body&gt; &lt;/html&gt; </code></pre> <p>You could also use the modules installed by <a href="https://www.npmjs.com/"><code>npm</code></a> with NW.js.</p> <p>!!! note "Native Node Modules"<br> Native Node modules, built when running <code>npm install</code>, are not compatible with NW.js ABI. To use them, you have to rebuild it from source code with <a href="https://github.com/nwjs/nw-gyp"><code>nw-gyp</code></a>. See [Use Native Node Modules](Advanced/Use Native Node Modules.md) for details.</p> https://tedneward.github.io/Research/platforms/objc/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/objc/index.html <p><a href="https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/ObjCRuntimeGuide/Introduction/Introduction.html">Objective-C Runtime Guide</a></p> https://tedneward.github.io/Research/platforms/openresty/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/openresty/index.html <p><a href="https://openresty.com/en/">Website</a></p> <p>Products:</p> <ul> <li>OpenResty Plus: "A premium Web application server that covers all your needs. An enterprise-ready Web server solution with rich built-in support features and better than ever programmability and compatibility."</li> <li>OpenResty Edge: "Gateway management platform built to maximize your productivity. A gateway software best to manage distributed web traffic in the multi-cloud and service mesh era."</li> <li>OpenResty Xray: "An all-in-one troubleshooting and profiling tool built for OSS. A real-time, lightweight, non-invasive monitoring tool that helps you optimize resources and quickly troubleshoot and resolve complex issues."</li> </ul> https://tedneward.github.io/Research/platforms/os-implementation/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/os-implementation/index.html <h2>Reading</h2> <h3>Articles/Blogs/Essays</h3> <ul> <li><a href="https://popovicu.com/posts/writing-an-operating-system-kernel-from-scratch/">Writing an Operating System Kernel from Scratch</a></li> <li><a href="https://www.howtogeek.com/what-is-an-agentic-os-and-why-microsoft-thinks-windows-will-soon-do-your-work-for-you/">How agentic OS will change the way you use Windows</a></li> </ul> <h2>Implementations</h2> <ul> <li><a href="https://github.com/jgarzik/hk">hk</a>: an operating system kernel written in Rust</li> </ul> https://tedneward.github.io/Research/platforms/plasmic/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/plasmic/index.html <p><a href="https://www.plasmic.app/">Website</a> | <a href="https://github.com/plasmicapp/plasmic">Source</a> | <a href="https://docs.plasmic.app/learn/">Docs</a></p> <p>Looks like it basically generates a <a href="../presentation/react">React</a> component that can then be referenced from React pages.</p> https://tedneward.github.io/Research/platforms/powerapps/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/powerapps/index.html <p><a href="https://powerapps.microsoft.com/en-us/">Website</a></p> https://tedneward.github.io/Research/platforms/pumpkinos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/pumpkinos/index.html <p><a href="https://github.com/migueletto/PumpkinOS">Source</a></p> https://tedneward.github.io/Research/platforms/quarkly/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/quarkly/index.html <p><a href="https://quarkly.io/">Website</a></p> <p><a href="../presentation/react">React</a>-based (using React components).</p> https://tedneward.github.io/Research/platforms/ravynos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/ravynos/index.html <p><a href="https://ravynos.com/">Website</a> | <a href="https://github.com/ravynsoft/ravynos">Source</a></p> https://tedneward.github.io/Research/platforms/remult/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/remult/index.html <p><a href="https://remult.dev/">Website</a> | <a href="https://github.com/remult/remult">Source</a></p> <p>Remult is a full-stack CRUD framework that uses your TypeScript model types to provide Secure REST API (highly configurable); Type-safe frontend API client; Type-safe backend query builder.</p> https://tedneward.github.io/Research/platforms/rowy/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/rowy/index.html <p><a href="https://www.rowy.io/">Website</a> | <a href="https://github.com/rowyio/rowy">Source</a></p> https://tedneward.github.io/Research/platforms/selfie/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/selfie/index.html <p><a href="http://selfie.cs.uni-salzburg.at/">Website</a> | <a href="https://github.com/cksystemsteaching/selfie">Source</a></p> <p>The Selfie Project provides an educational platform for teaching undergraduate and graduate students the design and implementation of programming languages and runtime systems. The focus is on the construction of compilers, libraries, operating systems, and even virtual machine monitors. The common theme is to identify and resolve self-reference in systems code which is seen as the key challenge when teaching systems engineering, hence the name.</p> <h2>Installing Selfie</h2> <p>Selfie runs in the cloud and natively on Linux, macOS, and Windows machines and possibly other systems that have a terminal and a C compiler installed. However, even if there is no C compiler installed on your machine or you only have access to a web browser you can still run selfie.</p> <p>There are at least three ways to install and run selfie, from real simple to a bit more difficult:</p> <ol> <li> <p>In the cloud: if you only have access to a web browser, just click <a href="https://replit.com/new/github/cksystemsteaching/selfie">here</a>. Alternatively, create a <a href="https://github.com">github</a> account, unless you already have one, and fork <a href="https://github.com/cksystemsteaching/selfie">selfie</a> into your github account. Then, create a <a href="https://c9.io">cloud9</a> student account, connect it to your github account, verify your email address and set a password (important!), and finally clone your fork of selfie into a new cloud9 workspace.</p> </li> <li> <p>In docker on your machine: if you have access to a Linux, macOS, or Windows machine download and install <a href="https://docker.com">docker</a>. Then, open a terminal window and type <code>docker run -it cksystemsteaching/selfie</code>. Besides simplicity, the key advantage of using docker is that you can run selfie out of the box on your machine but also on QEMU as well as on spike. Both emulators and the SMT solver boolector are pre-installed in the <a href="https://hub.docker.com/r/cksystemsteaching/selfie">selfie docker image</a>.</p> </li> <li> <p>Natively on your machine: instead of using docker, you may also just download and unzip <a href="https://github.com/cksystemsteaching/selfie/archive/main.zip">selfie</a>, and then open a terminal window to run selfie natively on your machine. However, for this to work you need to have a C compiler installed on your machine. We recommend using <a href="https://clang.llvm.org">clang</a> or <a href="https://gcc.gnu.org">gcc</a> (with <a href="https://www.cygwin.com">cygwin</a> on Windows).</p> </li> </ol> <p>At this point we assume that you have a system that supports running selfie. Below we use the <code>make</code> command assuming it is installed on your system which is usually the case. However, we also show the command invoked by <code>make</code> so that you can always invoke that command manually if your system does not have <code>make</code> installed.</p> <p>The next step is to produce a selfie binary. To do that <code>cd</code> to the selfie folder in your terminal and then type <code>make</code>. With docker, the system will respond <code>make: 'selfie' is up to date</code> since there is already a selfie binary pre-installed. Without docker, <code>make</code> will invoke the C compiler on your machine or in the cloud9 workspace:</p> <pre><code class="language-bash">cc -Wall -Wextra -O3 -D'uint64_t=unsigned long' selfie.c -o selfie </code></pre> <p>and then compile <code>selfie.c</code> into an executable called <code>selfie</code> as directed by the <code>-o</code> option. The executable contains the C* compiler, the mipster emulator, and the hypster hypervisor. The <code>-Wall</code> and <code>-Wextra</code> options enable all compiler warnings which is useful during further development of selfie. The <code>-O3</code> option instructs the compiler to generate optimized code. The <code>-D'uint64_t=unsigned long'</code> option is needed to bootstrap the code. It defines the data type <code>uint64_t</code> which would otherwise be undefined since C* does not include the necessary definitions. If your system supports compiling and executing 32-bit binaries you may also try <code>make selfie-32</code> which will produce a 32-bit system that in turn generates and executes 32-bit binaries.</p> <h2>Running Selfie</h2> <p>Once you have successfully compiled <code>selfie.c</code> you may invoke <code>selfie</code> without any arguments as follows:</p> <pre><code class="language-bash">$ ./selfie ./selfie { -c { source } | -o binary | [ -s | -S ] assembly | -l binary } [ ( -m | -d | -r | -y ) 0-4096 ... ] </code></pre> <p>In this case, <code>selfie</code> responds with its usage pattern.</p> <p>The order in which the options are provided matters for taking full advantage of self-referentiality.</p> <p>The <code>-c</code> option invokes the C* compiler on the given list of <code>source</code> files compiling and linking them into RISC-U code that is stored internally. For example, <code>selfie</code> may be used to compile its own source code <code>selfie.c</code> as follows:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c </code></pre> <p>The <code>-o</code> option writes RISC-U code produced by the most recent compiler invocation to the given <code>binary</code> file. For example, <code>selfie</code> may be instructed to compile itself and then output the generated RISC-U code into a RISC-U binary file called <code>selfie.m</code>:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -o selfie.m </code></pre> <p>The <code>-s</code> option writes RISC-U assembly of the RISC-U code produced by the most recent compiler invocation to the given <code>assembly</code> file while the <code>-S</code> option additionally includes approximate line numbers and the binary representation of the instructions. Similarly as before, <code>selfie</code> may be instructed to compile itself and then output the generated RISC-U code into a RISC-U assembly file called <code>selfie.s</code>:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -s selfie.s </code></pre> <p>The <code>-l</code> option loads RISC-U code from the given <code>binary</code> file. The <code>-o</code> and <code>-s</code> options can also be used after the <code>-l</code> option. However, in this case the <code>-s</code> option does not generate approximate source line numbers. For example, the previously generated RISC-U binary file <code>selfie.m</code> may be loaded as follows:</p> <pre><code class="language-bash">$ ./selfie -l selfie.m </code></pre> <p>The <code>-m</code> option invokes the mipster emulator to execute RISC-U code most recently loaded or produced by a compiler invocation. The emulator creates a machine instance with <code>0-4096</code> MB of memory. The <code>source</code> or <code>binary</code> name of the RISC-U code and any remaining <code>...</code> arguments are passed to the main function of the code. For example, the following invocation executes <code>selfie.m</code> using mipster:</p> <pre><code class="language-bash">$ ./selfie -l selfie.m -m 1 </code></pre> <p>This is in fact semantically equivalent to executing <code>selfie</code> without any arguments:</p> <pre><code class="language-bash">$ ./selfie </code></pre> <p>The <code>-d</code> option is similar to the <code>-m</code> option except that mipster outputs each executed instruction, its approximate source line number, if available, and the relevant machine state. Alternatively, the <code>-r</code> option limits the amount of output created with the <code>-d</code> option by having mipster merely replay code execution when runtime errors such as division by zero occur. In this case, mipster outputs only the instructions that were executed right before the error occurred.</p> <p>If you are using docker you can also execute <code>selfie.m</code> directly on spike and pk as follows:</p> <pre><code class="language-bash">$ spike pk selfie.m </code></pre> <p>which is again semantically equivalent to executing <code>selfie</code> without any arguments.</p> <p>The <code>-y</code> option invokes the hypster hypervisor to execute RISC-U code similar to the mipster emulator. The difference to mipster is that hypster creates RISC-U virtual machines rather than a RISC-U emulator to execute the code. See below for an example.</p> <h3>Self-compilation</h3> <p>Here is an example of how to perform self-compilation of <code>selfie.c</code> and then check if the RISC-U code <code>selfie1.m</code> generated for <code>selfie.c</code> by executing the <code>./selfie</code> binary is equivalent to the code <code>selfie2.m</code> generated by executing the just generated <code>selfie1.m</code> binary:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -o selfie1.m -m 2 -c selfie.c -o selfie2.m $ diff -s selfie1.m selfie2.m Files selfie1.m and selfie2.m are identical </code></pre> <p>Note that at least 2MB of memory is required for this to work.</p> <h3>Self-execution</h3> <p>The following example shows how to perform self-execution of the mipster emulator. In this case we invoke mipster to invoke itself to execute <code>selfie</code>:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -o selfie.m -m 2 -l selfie.m -m 1 </code></pre> <p>which is again semantically equivalent to executing <code>selfie</code> without any arguments but this time with <code>selfie</code> printing its usage pattern much slower since there is a mipster running on top of another mipster.</p> <h3>Self-hosting</h3> <p>The previous example can also be done by running hypster on mipster. This is significantly faster and requires less memory since hypster does not create a second emulator instance on top of the first emulator instance. Instead, hypster creates a virtual machine to execute selfie that runs concurrently to hypster on the first emulator instance:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -o selfie.m -m 1 -l selfie.m -y 1 </code></pre> <p>We may even run hypster on hypster on mipster which is still reasonably fast since there is still only one emulator instance involved and hypster itself does not add much overhead:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -o selfie.m -m 2 -l selfie.m -y 1 -l selfie.m -y 1 </code></pre> <h3>Workflow</h3> <p>To compile any C* source code and execute it right away in a single invocation of <code>selfie</code> without generating a RISC-U binary use:</p> <pre><code class="language-bash">$ ./selfie -c any-cstar-file.c -m 1 "arguments for any-cstar-file.c" </code></pre> <p>Equivalently, you may also use a selfie-compiled version of <code>selfie</code> and have the mipster emulator execute selfie to compile any C* source code and then execute it right away with hypster on the same emulator instance:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -m 1 -c any-cstar-file.c -y 1 "arguments for any-cstar-file.c" </code></pre> <p>You may also generate RISC-U binaries both ways which will then be identical:</p> <pre><code class="language-bash">$ ./selfie -c any-cstar-file.c -o any-cstar-file1.m $ ./selfie -c selfie.c -m 1 -c any-cstar-file.c -o any-cstar-file2.m $ diff -s any-cstar-file1.m any-cstar-file2.m Files any-cstar-file1.m and any-cstar-file2.m are identical </code></pre> <p>This can also be done in a single invocation of <code>selfie</code>:</p> <pre><code class="language-bash">$ ./selfie -c any-cstar-file.c -o any-cstar-file1.m -c selfie.c -m 1 -c any-cstar-file.c -o any-cstar-file2.m $ diff -s any-cstar-file1.m any-cstar-file2.m Files any-cstar-file1.m and any-cstar-file2.m are identical </code></pre> <p>The generated RISC-U binaries can then be loaded and executed as follows:</p> <pre><code class="language-bash">$ ./selfie -l any-cstar-file1.m -m 1 "arguments for any-cstar-file1.m" </code></pre> <h4>Linking</h4> <p>To compile and link any C* source code from multiple source files use:</p> <pre><code class="language-bash">$ ./selfie -c any-cstar-file1.c any-cstar-file2.c ... -m 1 </code></pre> <p>For example, to make the source code of <code>selfie.c</code> available as library code in any C* source code use:</p> <pre><code class="language-bash">$ ./selfie -c any-cstar-file.c selfie.c -m 1 </code></pre> <p>Note that multiple definitions of symbols are ignored by the compiler with a warning.</p> <h4>Debugging</h4> <p>Selfie's console messages always begin with the name of the source or binary file currently running. The mipster emulator also shows the amount of memory allocated for its machine instance and how execution terminated (exit code).</p> <p>As discussed before, RISC-U assembly for <code>selfie</code> and any other C* file is generated as follows:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -s selfie.s </code></pre> <p>If the assembly code is generated from a binary generated by the compiler (and not loaded from a file) approximate source line numbers are included in the assembly file.</p> <p>Verbose debugging information is printed with the <code>-d</code> option, for example:</p> <pre><code class="language-bash">$ ./selfie -c selfie.c -d 1 </code></pre> <p>Similarly, if the executed binary is generated by the compiler (and not loaded from a file) approximate source line numbers are included in the debug information.</p> https://tedneward.github.io/Research/platforms/singularity/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/singularity/index.html <p><a href="https://github.com/lastweek/source-singularity">Source</a> | <a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2005/10/tr-2005-135.pdf">Overview</a> (PDF) | <a href="https://www.microsoft.com/en-us/research/project/singularity/">Official site</a></p> <p>Discontinued since 2008, it seems like?</p> <h2>Reading</h2> <ul> <li><a href="https://codingkaiser.blog/2021/07/23/operating-systems-are-more-exciting-than-you-think/">"Operating Systems are More Exciting Than You Think"</a> -- good overview of Singularity and how it provides isolation/security</li> <li><a href="https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/osr2007_rethinkingsoftwarestack.pdf">"Rethinking the software stack"</a></li> </ul> https://tedneward.github.io/Research/platforms/stirling-pdf/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/stirling-pdf/index.html <p><a href="https://www.stirling.com/">Website</a> | <a href="https://github.com/Stirling-Tools/Stirling-PDF">Source</a></p> https://tedneward.github.io/Research/platforms/svardos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/svardos/index.html <p><a href="http://svardos.org/">Website</a> | <a href="https://github.com/SvarDOS/edrdos">GitHub</a></p> https://tedneward.github.io/Research/platforms/tina/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tina/index.html <p><a href="https://tina.io/">Website</a> | <a href="https://github.com/tinacms/tinacms">Source</a></p> https://tedneward.github.io/Research/platforms/tsuru/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/tsuru/index.html <p><a href="https://tsuru.io/">Website</a> | <a href="https://github.com/tsuru/tsuru">Source</a></p> https://tedneward.github.io/Research/platforms/linux/distros/hackeros/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/linux/distros/hackeros/index.html <p><a href="">Website</a></p> <h2>Resources</h2> <h3>Articles, Blogs, Essays</h3> <ul> <li><a href="https://thenewstack.io/hackeros-is-what-a-linux-enthusiasts-os-should-be/">HackerOS is what a Linux Enthusiast's OS Should Be</a></li> </ul> https://tedneward.github.io/Research/platforms/linux/tools/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/linux/tools/index.html <p><a href="./PerfObsTools.jpeg">Performance Observability Tools</a> (JPEG)</p> <p><a href="https://www.howtogeek.com/how-to-add-terminal-completion-to-your-command-line-apps/">"How to Add Terminal Completion to your Command-Line Apps</a></p> https://tedneward.github.io/Research/platforms/llvm/index/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/llvm/index/index.html <p><a href="https://llvm.org/">Website</a> | <a href="https://github.com/llvm/llvm-project">Github</a></p> <h3>Implementations</h3> <ul> <li>.NET: <a href="https://github.com/dotnet/llvmsharp/">LLVMSharp</a></li> <li>JVM: <a href="https://github.com/davidar/lljvm">LLJVM</a></li> </ul> <h3>Articles/Books</h3> <p><a href="http://www.aosabook.org/en/llvm.html">LLVM</a> - Chris Lattner, The Architecture of Open Source Applications</p> <p><a href="https://llvm.org/docs/GarbageCollection.html">Garbage Collection with LLVM</a> | <a href="https://llvm.org/docs/Statepoints.html">Garbage Collection Safepoints in LLVM</a></p> <h3>Tutorials</h3> <p><a href="https://llvm.org/docs/tutorial/MyFirstLanguageFrontend/index.html">My First Language Frontend with LLVM Tutorial</a> | <a href="http://llvm.org/docs/tutorial/">Kaleidoscope</a> or <a href="http://www.stephendiehl.com/llvm/">in Haskell</a></p> <p><a href="https://mukulrathi.com/create-your-own-programming-language/llvm-ir-cpp-api-tutorial/">A Complete Guide to LLVM for Programming Language Creators</a></p> <p><a href="https://www.youtube.com/watch?v=OhkwPSvyBu0">Make your own LLVM compiler</a></p> <p><a href="http://adriansampson.net/blog/llvm.html">LLVM for Grad Students</a></p> <p><a href="https://jonathan2251.github.io/lbd/">Tutorial: Creating an LLVM Backend for the Cpu0 Architecture</a></p> <p><a href="http://jonathan2251.github.io/lbt/">Tutorial: Creating an LLVM Toolchain for the Cpu0 Architecture</a></p> <p><a href="http://blog.regehr.org/archives/1453">A Tourist’s Guide to the LLVM Source Code</a></p> <p><a href="http://eli.thegreenplace.net/2012/11/24/life-of-an-instruction-in-llvm">Life of an instruction in LLVM</a> | <a href="https://github.com/thegameg/llvm-life/">in Clang/LLVM</a></p> <p><a href="http://blog.regehr.org/archives/1450">Testing LLVM</a></p> <h3>Related</h3> <p><a href="https://github.com/decomp/decomp">Decompilation pipeline</a> project: "The aim of this project is to implement a decompilation pipeline composed of independent components interacting through well-defined interfaces."</p> https://tedneward.github.io/Research/platforms/lowdefy/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/lowdefy/index.html <p><a href="https://lowdefy.com/">Website</a> | <a href="https://github.com/lowdefy/lowdefy">Source</a> | <a href="https://docs.lowdefy.com/">Docs</a></p> https://tedneward.github.io/Research/platforms/linux/distros/vanillaos/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/linux/distros/vanillaos/index.html <p><a href="https://vanillaos.org/">Website</a> | <a href="https://github.com/vanilla-os">Source</a></p> <p>"At the first start you can choose which package format to use in Vanilla OS (Flatpak, Snap, Appimage ..), you choose and Vanilla OS will take care of the rest, putting you in the situation to start without problems.</p> <p>"Vanilla OS is an immutable operating system, core parts of the system are locked down to prevent unwanted changes and corruption from third-party applications or a faulty update. Some paths are still writable, such as the home and configurations directories, this allows the user to keep their files and ensure the normal functioning of applications.</p> <p>"Core components are only updated via controlled and atomic transactions, which are applied only on success and made available on reboot."</p> https://tedneward.github.io/Research/platforms/linux/winboat/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/linux/winboat/index.html <p><a href="https://www.winboat.app/">Website</a> | <a href="https://github.com/TibixDev/winboat">Source</a></p> <p>The developer released a few new versions since last covering it, some highlights of what's been added recently:</p> <ul> <li>Custom install path.</li> <li>The ability to share your /home folder.</li> <li>It will remember where it was when you close the app.</li> <li>Multi-monitor support with MultiMon.</li> <li>An option to auto-start a container.</li> <li>A warning on disk space if you're getting full.</li> <li>Some improvements to the general flow and user help for the app itself.</li> <li>Added noVNC as a preset app if you want to access WinBoat from the browser.</li> <li>Experimental Dynamic USB pass-through.</li> </ul> https://tedneward.github.io/Research/platforms/llvm/mlir/index.html Mon, 13 Apr 2026 14:04:55 +0000 platforms/llvm/mlir/index.html <p><a href="https://mlir.llvm.org/">Website</a> | <a href="https://mlir.llvm.org/users/">Users</a> -- that is to say, projects and languages that use MLIR</p> <p>MLIR is intended to be a hybrid IR which can support multiple different requirements in a unified infrastructure. For example, this includes:</p> <ul> <li>The ability to represent dataflow graphs (such as in TensorFlow), including dynamic shapes, the user-extensible op ecosystem, TensorFlow variables, etc.</li> <li>Optimizations and transformations typically done on such graphs (e.g. in Grappler).</li> <li>Ability to host high-performance-computing-style loop optimizations across kernels (fusion, loop interchange, tiling, etc.), and to transform memory layouts of data.</li> <li>Code generation “lowering” transformations such as DMA insertion, explicit cache management, memory tiling, and vectorization for 1D and 2D register architectures.</li> <li>Ability to represent target-specific operations, e.g. accelerator-specific high-level operations.</li> <li>Quantization and other graph transformations done on a Deep-Learning graph.</li> <li>Polyhedral primitives.</li&