英語に関する驚きの事実 - にょきにょきブログ

英語について、統計をとってみた。なんとも驚いた。

今回統計を取る対象となったテスト文章は Wikibooks 英語版(http://en.wikibooks.org/)の全ページ。ダウンロードはこの辺から。http://dumps.wikimedia.org/backup-index.html
以下の点を妥協して、単語を抜き出す。XML タグや記号付きの単語 (-first みたいな)は省いているので純粋に記事に関しての単語が取得できているはず。
・活用や複数形を考慮しているわけではない。例えば、student と students は別の単語として計算される。
・人物名や地名は省いてない。

すると、全単語数は 24923806 個、ユニークな単語(全単語数から重複を取り除いた単語数)は 139333 個ということがわかった。
このパースした結果を元に統計情報を取ると、以下の驚きの結果が出た。

・ユニークな単語のうち、およそ半数は1度しか使用されていない。
・全文章を構成する単語のうち、以下の7単語で30%を占める。
the, of, to, and, a, in, is.
・上記単語に以下の単語を加えたものが、全文章のうちの50%を占める。計 33 単語也。
that, be, as, with, it, this, on, you, by, can, an, not, have, if, at, will, witch, we, was, they, but, one, had, all, i, when.

・全文章を構成する単語のうち、250単語で70%を占める。
・全文章を構成する単語のうち、700単語で80%を占める。
・全文章を構成する単語のうち、2300単語で90%を占める。

なんと、Wikibooks 英語版の全文章の 9 割は、たった 2300 個の単語によって構成されているのだ！

というわけで文章の90%を占める2300単語はこいつら。(出現頻度順)

the, of, to, and, a, in, is, that, be, as, with, it, this, on, you, by, can, an, not, have, if, at, will, which, we, was, they, but, one, has, all, i, when, use, also, some, may, used, into, these, what, so, do, about, he, then, would, only, should, two, his, new, than, no, any, how, its, most, such, using, many, each, like, been, make, because, up, who, time, between, out, same, them, see, guide, need, just, had, people, must, does, set, could, being, type, while, even, way, both, get, students, data, system, function, called, book, solid, page, following, much, good, now, value, made, often, name, those, without, take, want, find, well, since, end, x, example, add, my, know, file, change, within, given, point, help, common, might, small, own, high, found, line, list, means, code, text, language, usually, class, known, long, until, still, case, go, second, school, think, back, she, able, object, place, based, us, public, state, less, look, open, left, last, possible, always, method, next, few, main, give, english, down, specific, include, put, available, image, something, him, above, things, become, de, single, let, keep, did, pages, simple, level, why, note, types, best, development, once, c, making, show, too, section, little, web, due, table, contains, having, done, although, basic, light, against, least, come, study, move, content, top, makes, education, uses, social, states, off, life, cannot, including, body, local, though, student, say, going, access, allows, allow, changes, useful, instead, technology, simply, human, side, values, model, systems, test, key, united, needs, amount, cell, field, b, functions, y, knowledge, meaning, books, me, space, opening, sometimes, moved, black, actually, multiple, along, special, size, question, call, times, mass, defined, idea, thus, management, low, complete, select, full, questions, added, int, input, position, examples, likely, day, enough, individual, wikibooks, white, design, subject, seen, cause, upon, files, old, takes, points, output, below, analysis, check, methods, application, view, patients, business, link, national, whole, am, elements, complex, command, step, among, solution, especially, click, sound, template, gives, et, ways, cells, sample, blood, please, fact, shows, effect, positive, inside, objects, native, almost, equal, taken, sentence, easy, additional, child, act, doing, yet, physical, play, family, seems, negative, element, law, shown, database, comes, quite, bit, feel, potential, links, difficult, e, rt, ability, needed, away, associated, total, title, default, chemical, double, said, easily, developed, hand, quality, choose, else, lines, outside, tools, unit, j, action, discussion, contain, pmid, communication, lot, someone, shall, bottom, n, active, base, distance, final, becomes, length, home, man, includes, million, mean, edit, video, issues, heat, windows, schools, food, god, digital, educational, style, equation, build, thing, dna, effective, close, window, lead, voltage, statement, game, community, half, again, science, teaching, define, past, cut, f, kind, moving, talk, definition, plant, age, team, typically, acid, images, la, five, online, stage, adding, volume, effects, follow, youth, taking, became, significant, date, via, tell, speed, avoid, looking, included, thought, site, d, ask, box, begin, option, anything, s, sequence, licensing, believe, block, head, addition, beginning, women, days, itself, cases, names, species, policy, big, stamp, applied, designed, cost, constant, ideas, message, develop, copy, john, live, advanced, t, goes, june, sense, phase, men, explain, os, placed, apply, involved, followed, focus, came, middle, m, flow, display, actual, oil, hold, initial, leave, depending, levels, august, media, document, fixed, july, death, tool, nothing, except, building, says, health, house, seem, getting, cup, map, existing, device, vs, account, limited, issue, plan, named, languages, applications, signal, g, causes, commonly, notice, themselves, exactly, completely, pass, void, components, location, gas, molecules, global, continue, living, society, assessment, allowed, face, behind, concept, changed, choice, disease, basis, studies, sets, jesus, mode, wikipedia, individuals, unless, module, whose, containing, piece, wide, evidence, manual, south, mind, looks, conditions, company, domain, events, event, skills, save, button, leaves, identify, teach, activity, task, notes, gy, match, money, young, items, loop, land, selected, months, options, giving, changing, consists, stop, delete, linux, music, le, lack, gets, setting, highly, caused, visual, took, package, benefit, onto, essential, lesson, began, deal, job, plants, techniques, component, java, passed, equivalent, quickly, contents, goal, index, collection, maximum, specified, scale, exist, medical, connected, sign, classes, units, unique, v, bot, path, anyone, steps, activities, six, built, attention, bad, un, meet, contact, technical, instance, plus, column, sent, sum, developing, item, economic, xml, send, amino, helps, movement, love, attempt, machine, independent, implementation, listed, depends, post, angle, fall, install, wikibook, status, tend, political, affect, held, menu, pay, slightly, shape, late, z, blue, weight, node, p, technique, combination, expected, intended, automatically, discuss, audio, details, context, assume, r, population, minutes, immediately, testing, hot, condition, typos, loss, king, mentioned, influence, impact, led, beyond, fully, models, wave, dose, scientific, located, license, mathematical, asked, typical, mostly, office, decision, salt, city, allowing, magnetic, id, went, soon, connection, involves, metal, sun, published, eventually, solve, divided, client, japanese, maintain, attack, syntax, limit, situation, topic, advantage, wish, static, load, ii, tells, fit, genetic, wiki, alone, established, FALSE, h, leading, places, ones, discussed, oxygen, pieces, motion, opposite, actions, indicate, isbn, finding, showing, concepts, handle, tests, sides, college, edits, physics, achieve, safety, thinking, devices, lists, supply, gain, specify, count, ten, got, legal, official, fields, son, atoms, edge, documentation, median, saying, medium, himself, exists, maybe, que, comment, speak, possibly, functional, successful, stable, moves, lost, planning, feedback, chain, o, k, aspects, binding, selection, editing, connect, update, calls, sites, empty, switch, sections, comments, symbol, mix, news, laws, today, settings, costs, yahweh, minimum, ip, commands, animals, unlike, website, liquid, usda, keeping, equations, clinical, animal, despite, fuel, engine, ancient, ends, indicates, skin, obtain, shell, closed, download, success, stay, speech, en, usage, kept, extended, leaving, statements, games, acts, combined, seven, mental, wanted, happens, cold, benefits, molecule, linked, believed, academic, gene, suggest, chinese, leads, widely, none, patient, staff, cycle, decide, saw, html, fast, kinds, height, goals, coming, quick, acids, exchange, damage, python, hope, hands, calculate, sounds, begins, fill, implement, somewhat, playing, tasks, identified, valid, extension, ron, chance, fine, mouse, die, east, gave, mainly, es, becoming, velocity, attached, solutions, spanish, enzyme, detailed, obtained, wants, cm, el, union, host, month, finally, sexual, width, ions, japan, causing, ending, stated, templates, humans, paul, accept, capital, atom, visible, ion, l, companies, famous, eye, aid, assigned, film, eat, atomic, shift, bond, glass, fundamental, continuous, soil, distinct, tax, messages, logic, hence, told, specifically, logical, biological, enable, scope, clean, wait, deep, happen, split, safe, daily, dynamic, bound, ideal, affected, mobile, buy, topics, taught, opinion, balance, log, mhz, u, thanks, meant, suitable, mathematics, talking, expect, follows, yellow, stand, hit, modify, bone, seed, suggests, bits, suppose, calling, exception, chosen, dc, slow, couple, scientists, symptoms, income, female, rna, meeting, exact, efficient, financial, density, modified, demand, yes, waves, whom, disk, join, weeks, division, heavy, soft, ball, quantum, blocks, dependent, axis, naming, math, stock, voice, latin, decided, male, capacity, passing, unable, plane, tag, obvious, association, wall, fix, sufficient, bonds, mechanism, foundation, equipment, composed, putting, knows, holds, iii, technologies, excellent, tissue, documents, suggested, tables, pick, cognitive, highest, club, nice, defines, continued, significantly, missing, thin, flat, namespace, email, impossible, civil, vote, assembly, mac, push, sea, west, amounts, noob, fluid, tab, nodes, identity, self, week, effectively, noted, visit, adds, capable, bold, woman, attested, mm, phone, cook, noun, detail, evaluation, implemented, watch, evaluate, substance, stem, fish, textbook, filled, pull, muscle, bind, classical, played, channel, dog, huge, knowing, indian, involving, essentially, signals, non, installed, displayed, helpful, panel, null, lives, subjects, plate, communicate, asking, ice, spoken, explanation, aspect, died, font, texts, town, keys, wikimedia, policies, attempts, stack, eight, depend, involve, adult, claim, contained, holy, nov, mixed, foot, pain, lessons, modules, tube, basically, flag, spend, io, james, absolute, speaking, kingdom, combine, decisions, asks, nations, gold, deleted, execute, establish, depth, guidelines, spaces, accepted, waste, nouns, sentences, completed, feet, continues, zone, pdf, edges, quantity, genes, meat, milk, spent, applies, lots, unknown, imagine, showed, symbols, divide, situations, holding, custom, explained, ada, possibility, moment, battle, dealing, manage, accessed, vowel, egg, maintenance, eyes, snape, pan, identical, sheet, compound, objective, net, vehicle, kill, sex, conclusion, plans, met, closely, lose, initially, compounds, plays, apple, consistent, dead, motivation, conflict, complicated, execution, enzymes, updated, season, calculated, extent, alpha, usual, q, roman, label, positions, william, invasive, smooth, chicken, tea, or, noise, solving, google, shot, span, cooking, mechanical, sub, def, float, maps, definitions, sees, columns, oncol, wood, assignment, david, weak, lets, subsequent, finished, lua, night, advantages, w, usa, opposed, commons, achieved, faith, stages, embedded, annual, immediate, taste, willing, wheel, dutch, applying, cups, existence, detect, beam, los, goods, thousands, twice, bag, styles, consensus, views, optional, synthesis, entity, biol, expensive, intelligence, computing, puts, usb, radiat, joint, nucleus, looked, valuable, scene, managing, indeed, paid, accounts, walk, chopped, layout, sin, fat, plastic, evolution, tense, touch, boolean, acting, exposed, guess, failed, installation, feed, finds, signs, al, explains, cd, stands, displays, plain, extend, bank, identification, psychology, red, cool, studying, assign, cpu, handling, audience, magnitude, lung, edition, extensive, russian, adjuvant, faces, sensitive, stick, angles, info, seconds, families, assumed, limits, decimal, bodies, connections, bill, institute, seeing, focused, minimal, cast, exit, consist, right, affects, notation, defining, difficulty, successfully, supposed, ph, clicking, outcome, eggs, unix, flash, icon, km, jewish, modeling, deletion, spacing, se, emotional, review, equals, felt, mail, helping, expand, samples, island, saved, compile, stuff, seek, catch, alcohol, conventional, killed, plot, sold, glucose, sc, ahead, ms, feeling, echo, desktop, wind, mention, navigation, claims, copies, fun, composition, tm, van, tags, implies, sauce, passive, indicated, philosophy, jump, du, css, slowly, cheese, viewed, sql, song, expansion, mesh, session, station, packages, standing, stamps, latest, england, tends, fail, adjust, cable, movie, acceptable, paste, obviously, executed, enhance, statistical, knew, outline, examine, asia, delay, communities, convention, subset, sequences, websites, infinite, communications, challenge, wild, escape, passes, meanings, teaspoon, thick, choosing, checking, studied, assist, peace, consisting, stone, admin, efficiency, hole, neck, potentially, san, dialog, signed, helped, sit, moon, automatic, keeps, li, gnu, placing, estimated, equally, lived, dimension, blank, classification, fatty, hidden, facts, belief, lies, yield, oven, sales, dimensions, conduct, const, mapping, employees, specifies, sell, sending, specialized, simplest, bible, finite, databases, investment, bytes, pc, constantly, thousand, selecting, happened, matching, unto, psychological, api, seeds, skill, foss, covenant, clin, multiply, matches, machines, steel, distinguish, welcome, opened, multiplication, channels, chemo, classic, read, reading, tested, vitamin, maintaining, waiting, boot, eating, substances, consequences, falls, wikimanual, functionality, gap, excess, spot, substitute, band, cited, belong, nation, fight, passage, council, collected, scheme, choices, teams, employee, tips, adopted, vision, sodium, dish, enables, conducted, si, explicitly, aim, toxicity, washington, assess, attempting, cat, believes, telling, happy, byte, setup, lay, finish, baby, cities, advance, leaf, instances, utility, inputs, classified, assistance, healthy, estimate, buttons, ultimately, influenced, loaded, fixing, pacific, agent, constitution, thank, judge, listen, mountain, del, il, floating, remove, bug, codes, jobs, odd, television, real, occasionally, absence, walls, juice, emphasis, mount, accessible, magic, oz, intensity, castle, explicit, engage, pts, clause, exam, mouth, sweet, asian, boys, institutions, statistics, upload, collect, tiny, lowest, analyze, objectives, telephone, enjoy, dim, thomas, economy, cash, dough, managed, expectations, imaging, billion, locally, conditional, queen, compute, cooling, police, maintained, diseases, funding, ha, compatible, cube, mechanisms, adjacent, intellectual, inch, fifth, gained, feels, metabolic, magical, publishing, extends, soviet, assuming, china, dedicated, challenges, specification, bases, wet, vice, cooked, discussions, plasma, implementing, thoughts, adequate, newly, spin, matthew, kinetic, noticed, licensed, advice, latex, sends, bus, clock, planet, beneficial, categories, segment, biology, italian, involvement, pointed, joined, heavily, minute, boat, hide, flows, gone, medicine, meaningful, facilitate, holes, votes, from, baking, jan, cutting, suggestions, con, destination, attempted, ocean, textbooks, vowels, vital, facing, identifying, tone, connecting, myself, miles, indicating, spatial, ict, unsigned, paint, definite, interwikis, detection, pot, debate, childhood, flexible, outcomes, pointing, boiling, zealand, des, pixel, tongue, lemon, technological, committee, eliminate, vast, deals, dot, focuses, animation, lock, quantities, jews, checks, hash, wine, pawn, edited, diagnosis, locations, wife, adults, tsp, hypothesis, feelings, immune, chains, url, distinction, combinations, photo, evaluated, achievement, fold, linking, golden, massive, optical, tail, tip, muscles, stops, spelling, win, isolated, packet, radio, adjectives, coding, radiation, compiled, voting, clients, peak, wavelength, competition, masculine, catholic, exceptions, dec, acute, spinal, businesses, besides, complexity, bones, won, substantial, canada, feminine, detected, assumption, boost, iv, beta, smallest, climate, actively, vegetable, movements, analog, nine, citizens, attacks, loops, commission, attend, lie, inches, modes, alphabet, patent, competitive, php, executive, wings, les, essay, committed, laid, binds, gel, sleep, mission, tv, convenient, subpages, suffix, easiest, ad, definitely, doubt, boxes, listing, mysql, cache, expanded, consonant, boy, aros, distant, kids, chess, chief, dance, legs, london, labeled, typing, lymph, backup, republic, lift, copied, dominant, analytical, slide, slope, quote, induction, abuse, genus, lighting, solved, invented, schedule, listening, planned, momentum, sizes, ethical, satisfy, opinions, haskell, blocked, employed, michael, defense, gfdl, ok, implications, fukushima, olive, justice, scan, skip, ease, offset, digits, genome, aside, auto, celestia, publication, sciences, victim, disciples, males, polynomial, dioxide, cycles, yields, und, guides, domestic, shapes, beliefs, installing, absolutely, bias, founded, discipline, temple, confused, sky, combining, ma, foods, pool, enabled, calculation, bowl, unlikely, unusual, weasley, saving, slice, seemed, stability, dialogue, heading, heads, cloud, evolved, metabolism, encoding, belongs, defence, limitations, twelve, selling, ago, calcium, assumes, modification, supplied, basics, qualified, atp, houses, shade, mixing, till, gui, findings, lake, est, meets, max, diffuse, canadian, tbsp, mov, switching, lens, pixels, adapted, stopped, significance, bed, gate, twenty, downloaded, applicable, sitting, feb, chose, ram, india, seeking, hello, chemicals, jun, beat, chocolate, val, physically, fails, consumption, taxes, suit, opens, italy, postage, mood, pitney, gospel, walking, lo, confidence, denoted, manually, dates, agency, filling, falling, steam, decay, tendency, explaining, ought, seat, millions, ended, diet, leg, counting, capabilities, polish, denote, theme, coast, accomplished, effectiveness, eligibility, vos, loose, bool, nuclei, optimal, emission, examination, discussing, candidate, eaten, gases, peptide, te, ultimate, issued, biggest, accused, boil, plenty, login, accomplish, shaped, innovation, minimize, acoustic, automated, illegal, totally, ongoing, whilst, inclusion, funds, role, lab, offence, substitution, watching, adjective, chip, seventh, conscious, viewing, locate, hospital, evil, equality, beef, glpk, duty, mechanics, caught, budget, mi, dimensional, consciousness, calculations, uv, digit, conjunction, notion, intent, flight, attenuation, labels, loan, aims, coefficient, lights, vocal, posted, mutation, for, phenomenon, pipe, closing, displacement, disc, tablespoons, independence, losing, plug, ca, spell, attach, fighting, di, dvd, household, testament, sine, intel, loading, claimed, pump, schema, owned, guideline, placement, initiative, violence, lipid, dual, behave, possess, demands, continuing, designing, dividing, outputs, county, sustainable, pop, sand, utilize, checked, incoming, appointed, honey, tied, manipulate, designs, ages, von, thickness, quiz, possibilities, blended, planets, nmr, chest, mutations,