(Isolation Forest无监督)这个算法是随机森林的推广。 iTree树构造：随机选一个属性，再随机选该特征的一个值，对样本进行二叉划分，重复以上操作。 iTree构建好了后，就可以对数据进行预测啦，预测的过程就是把测试记录在iTree上走一下，看测试记录落在哪个叶子节点。iTree能有效检测异常的假设是：异常点一般都是非常稀有的，在iTree中会很快被划分到叶子节点，因此可以用叶子节点到根节点的路径h(x)长度来判断一条记录x是否是异常点。 
越接近1表示是异常点的可能性高； 越接近0表示是正常点的可能性比较高； 如果大部分的训练样本的s(x,n)都接近于0.5，整个数据没有明显的异常。
Iforest构造：随机采样抽取一部分数据构造每一棵树，保证树的差异性。 与随机森林的区别： 1、随机森林需要采样集样本个数等于训练集个数，IForest也是随机采样，但样本个数小于训练集个数。因为我们的目的是异常点检测，只需要部分的样本我们一般就可以将异常点区别出来了 2、IForest采用随机选择一个划分特征，对划分特征随机选择一个划分阈值，RF划分需要使用信息增益或者信息增益率作为选择属性和阈值的依据。
URL异常检测(IForest)
数据源在这里。 正常请求的数据
/103886/ /rcanimal/ /458010b88d9ce/ /cclogovs/ /using-localization/ /121006_dakotacwpressconf/
恶意请求的数据
/top.php?stuff='uname >q36497765 # /h21y8w52.nsf?<script>cross_site_scripting.nasl</script> /ca000001.pl?action=showcart&hop="><script>alert('vulnerable')</script>&path=acatalog/ /scripts/edit_image.php?dn=1&userfile=/etc/passwd&userfile_name= ;id;
这里就需要有文本向量化的知识，调用sklearn中CountVectorizer
1from sklearn.feature_extraction.text import CountVectorizer 2vectorizer=CountVectorizer() 3corpus=["I come to China to travel", 4 "This is a car polupar in China", 5 "I love tea and Apple ", 6 "The work is to write some papers in science"] 7print(vectorizer.fit_transform(corpus)) 8'''文本的序号,词的序号,词频''' 9 (0, 16) 1 10 (0, 3) 1 11 (0, 15) 2 12print(vectorizer.get_feature_names())#I是停用词不统计,打印分割好的词向量 13['and', 'apple', 'car', 'china', 'come', 'in', 'is', 'love', 'papers', 'polupar', 'science', 'some', 'tea', 'the', 'this', 'to', 'travel', 'work', 'write'] 14 15 16print(vectorizer.fit_transform(corpus).toarray()) //转换成矩阵打印，四行代表四句话，列上的数字代表出现频率，统计词频后的19维特征做为文本分类的输入
TF-IDF是一种统计方法，用以评估一字词对于一个文件集或一个语料库中的其中一份文件的重要程度。 TF（Term Frequency）词频，某个词在文章中出现的次数或频率。 IDF（inverse document frequency）逆文档频率。一个词语“权重”的度量，一个词越常见，IDF越低。 IDF计算公式： 
N代表语料库中文本的总数，而N(x)代表语料库中包含词x的文本总数，有对应的词库 
1from sklearn.feature_extraction.text import TfidfTransformer 2from sklearn.feature_extraction.text import CountVectorizer 3 4corpus=["I come to China to travel", 5 "This is a car polupar in China", 6 "I love tea and Apple ", 7 "The work is to write some papers in science"] 8 9vectorizer=CountVectorizer() 10 11transformer = TfidfTransformer() 12tfidf = transformer.fit_transform(vectorizer.fit_transform(corpus)) 13print(tfidf)
得到（文本的序号,词的序号，TF-IDF值） 
这块知识的相关参考文章： https://www.cnblogs.com/pinard/p/6693230.html https://www.cnblogs.com/pinard/p/6693230.html
1import pandas as pd 2import numpy as np 3import matplotlib.pyplot as plt 4from sklearn.model_selection import train_test_split 5from sklearn.feature_extraction.text import TfidfVectorizer 6from sklearn.ensemble import IsolationForest 7from sklearn.metrics import confusion_matrix 8import itertools 9from sklearn.metrics import accuracy_score 10#data 11good_data=pd.read_csv('goodqueries.txt',names=['url']) 12good_data['label']=0 13data=good_data 14data.head() 15##feature 16vectorizer = TfidfVectorizer(min_df = 0.0, analyzer="char", sublinear_tf=True, ngram_range=(1,3)) #converting data to vectors 17X = vectorizer.fit_transform(data['url'].values.astype('U')) #TF-IDF向量化
分为训练集和测试集
1X_train, X_test, y_train, y_test = train_test_split(X, data['label'].values, test_size=0.2, random_state=42) #splitting data 2print(X_train) #y_test标签都设置为0表示都是正常数据集 3 4 5clf=IsolationForest() 6clf.fit(X_train) 7y_pre = clf.predict(X_test) 8 9 10ny_pre = np.asarray(y_pre) 11ny_pre[ny_pre==1] = 0 ##最后输出1为正常，-1为异常 ==>0是正常，1是异常点 12ny_pre[ny_pre==-1] = 1 13 14ny_test = np.asarray(y_test) #y_test都是0因为只导入了goodqueries.txt数据集 15accuracy_score(ny_test,ny_pre)
URL异常检测(LSTM)
1#!/usr/bin/env python 2# -*- coding: utf-8 -*- 3# @Time : 2019/6/4 9:25 4# @Author : afanti 5 6import sys 7import os 8import json 9import pandas as pd 10import numpy 11import optparse 12from keras.callbacks import TensorBoard 13from keras.models import Sequential 14from keras.layers import LSTM, Dense, Dropout 15from keras.layers.embeddings import Embedding 16from keras.preprocessing import sequence 17from keras.preprocessing.text import Tokenizer 18from collections import OrderedDict 19from keras.models import load_model 20from keras.models import model_from_json 21def model(): 22 dataframe = pd.read_csv('goodqueries.txt', names=['url']) 23 dataframe['label']=0 24 # dataframe.head() 25 dataframe1 = pd.read_csv('badqueries.txt', names=['url']) 26 dataframe1['label']=1 27 # dataframe1.head() 28 dataset=pd.concat([dataframe,dataframe1]) 29 dataset=dataset.sample(frac=1).values 30 X = dataset[:,0] 31 Y = dataset[:,1] 32 for i in range(len(X)): 33 if type(X[i])==float: 34 X[i]=str(X[i]) 35 tokenizer = Tokenizer(filters='\t\n', char_level=True) 36 tokenizer.fit_on_texts(X) 37 X = tokenizer.texts_to_sequences(X) #序列的列表，列表中每个序列对应于一段输入文本 38 word_dict_file = 'build/word-dictionary.json' 39 40 if not os.path.exists(os.path.dirname(word_dict_file)): 41 os.makedirs(os.path.dirname(word_dict_file)) 42 43 with open(word_dict_file, 'w',encoding='utf-8') as outfile: 44 json.dump(tokenizer.word_index, outfile, ensure_ascii=False) #将单词（字符串）映射为它们的排名或者索引 45 46 num_words = len(tokenizer.word_index)+1 #174 47 48 max_log_length = 100 49 train_size = int(len(dataset) * .75) 50 51 # padding 52 X_processed = sequence.pad_sequences(X, maxlen=max_log_length) 53 # 划分样本集 54 X_train, X_test = X_processed[0:train_size], X_processed[train_size:len(X_processed)] 55 Y_train, Y_test = Y[0:train_size], Y[train_size:len(Y)] 56 57 model = Sequential() 58 model.add(Embedding(num_words, 32, input_length=max_log_length)) 59 model.add(Dropout(0.5)) 60 model.add(LSTM(64, recurrent_dropout=0.5)) 61 model.add(Dropout(0.5)) 62 model.add(Dense(1, activation='sigmoid')) 63 model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy']) 64 model.summary() 65 model.fit(X_train, Y_train, validation_split=0.25, epochs=3, batch_size=128) 66 # Evaluate model 67 score, acc = model.evaluate(X_test, Y_test, verbose=1, batch_size=128) 68 print("Model Accuracy: {:0.2f}%".format(acc * 100)) 69 # Save model 70 model.save_weights('securitai-lstm-weights.h5') 71 model.save('securitai-lstm-model.h5') 72 with open('securitai-lstm-model.json', 'w') as outfile: 73 outfile.write(model.to_json()) 74 75 df_black = pd.read_csv('badqueries.txt', names=['url'], nrows=20000) 76 df_black['label'] = 1 77 X_waf = df_black['url'].values.astype('str') 78 Y_waf = df_black['label'].values.astype('str') 79 X_sequences = tokenizer.texts_to_sequences(X_waf) 80 X_processed = sequence.pad_sequences(X_sequences, maxlen=max_log_length) 81 score, acc = model.evaluate(X_processed, Y_waf, verbose=1, batch_size=128) 82 print("Model Accuracy: {:0.2f}%".format(acc * 100))
这里用到了keras的Tokenizer分词器
1from keras.preprocessing.text import Tokenizer 2import keras 3tokenizer = Tokenizer(char_level=True) 4text = ["/javascript/nets.png", "/javascript/legacy.swf"] 5tokenizer.fit_on_texts(text) 6# tokenizer.word_counts #[[2, 5]] False 7tokenizer.texts_to_sequences(["nets swf"]) 8tokenizer.word_index
当char_level=True时，会按char-level将字符频度进行统计，如下图a出现次数最多， 
输出结果，nets swf输出在上面图查找：
#[[11, 12, 6, 3, 3, 17, 18]] True 语义没了
输入数据做了padding操作，这里补成了128维
1# padding 2 X_processed = sequence.pad_sequences(X, maxlen=max_log_length)
相关Tokenizer操作看这里: https://blog.csdn.net/wcy23580/article/details/84885734 https://blog.csdn.net/wcy23580/article/details/84957471 训练模型前Embedding层做一次word embedding，单词嵌入是使用密集的矢量表示来表示单词和文档的一类方法。output_dim这是嵌入单词的向量空间的大小，input_dim这是文本数据中词汇的取值可能数这里是173，input_length：输入文档都由1000个字组成，那么input_length就是1000. 下面我们定义一个词汇表为173的嵌入层（例如从1到173的整数编码的字，包括1到173），一个32维的向量空间，其中将嵌入单词，以及输入文档，每个单词有128个字符 嵌入层自带学习的权重，如果将模型保存到文件中，则将包含嵌入图层的权重。
1model = Sequential() 2model.add(Embedding(input_dim=num_words, output_dim=32, input_length=max_log_length)) 3model.add(Dropout(0.5)) 4model.add(LSTM(64, recurrent_dropout=0.5)) 5model.add(Dropout(0.5)) 6model.add(Dense(1, activation='sigmoid')) 7model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])
总的来说就是char-level+统计向量化+词嵌入+神经网络LSTM
相关参考在这里：https://juejin.im/entry/5acc23f26fb9a028d1416bb3
URL异常检测(LR)
导入数据
1# -*- coding:utf8 -*- 2from sklearn.externals import joblib 3from sklearn.feature_extraction.text import TfidfVectorizer 4import os 5from sklearn.model_selection import train_test_split 6from sklearn.linear_model import LogisticRegression 7from sklearn import metrics 8import urllib.parse 9from sklearn.pipeline import Pipeline 10import matplotlib.pyplot as plt 11 12def loadFile(name): 13 directory = str(os.getcwd()) 14 filepath = os.path.join(directory, name) 15 with open(filepath,'r',encoding="utf8") as f: 16 data = f.readlines() 17 data = list(set(data)) 18 result = [] 19 for d in data: 20 d = str(urllib.parse.unquote(d)) #converting url encoded data to simple string 21 result.append(d) 22 return result 23 24badQueries = loadFile('badqueries.txt') 25validQueries = loadFile('goodqueries.txt') 26 27badQueries = list(set(badQueries)) 28validQueries = list(set(validQueries)) 29allQueries = badQueries + validQueries 30yBad = [1 for i in range(0, len(badQueries))] #labels, 1 for malicious and 0 for clean 31yGood = [0 for i in range(0, len(validQueries))] 32y = yBad + yGood 33queries = allQueries
划分测试集训练集
1X_train, X_test, y_train, y_test = train_test_split(queries, y, test_size=0.2, random_state=42) #splitting data 2badCount = len(badQueries) #44532 3validCount = len(validQueries) #1265974
Pipeline可以将许多算法模型串联起来，比如将特征提取、归一化、分类组织在一起形成一个典型的机器学习问题工作流。 tfidf TfidfVectorizer类参数analyzer定义特征为词（word）或n-gram字符。 sublinear_tf 应用线性缩放TF，例如，使用1+log(tf)覆盖tf。 ngram_range要提取的n-gram的n-values的下限和上限范围class_weight={0:0.9, 1:0.1}，这样类型0的权重为90%，而类型1的权重为10%。
1pipe_lr = Pipeline([('tfidf', TfidfVectorizer(min_df = 0.0, analyzer="char", sublinear_tf=True, ngram_range=(1,3))), 2 ('clf', LogisticRegression(class_weight="balanced")) 3 ]) 4pipe_lr.fit(X_train, y_train) 5 6predicted = pipe_lr.predict(X_test) 7predicted=list(predicted) 8fpr, tpr, _ = metrics.roc_curve(y_test, (pipe_lr.predict_proba(X_test)[:, 1])) 9auc = metrics.auc(fpr, tpr) 10 11print("Bad samples: %d" % badCount) 12print("Good samples: %d" % validCount) 13print("Baseline Constant negative: %.6f" % (validCount / (validCount + badCount))) 14print("------------") 15print("Accuracy: %f" % pipe_lr.score(X_test, y_test)) #checking the accuracy 16print("Precision: %f" % metrics.precision_score(y_test, predicted)) 17print("Recall: %f" % metrics.recall_score(y_test, predicted)) 18print("F1-Score: %f" % metrics.f1_score(y_test, predicted)) 19print("AUC: %f" % auc) 20joblib.dump(pipe_lr,"lr.pkl")
测试
1from urllib.parse import urlparse 2from sklearn.externals import joblib 3lr=joblib.load("lr.pkl") 4def url(url): 5 try: 6 parsed_url=urlparse(url) 7 paths=parsed_url.path+parsed_url.query 8 result=lr.predict([paths]) 9 10 if result==[0]: 11 return False 12 else: 13 return True 14 except Exception as err: 15 #print(str(err)) 16 pass 17 18result=url('http://br-ofertasimperdiveis.epizy.com/examples/jsp/cal/feedsplitter.php?format=../../../../../../../../../../etc/passwd\x00&debug=1') 19result1=url('http://br-ofertasimperdiveis.epizy.com/produto.php?linkcompleto=iphone-6-plus-apple-64gb-cinza') 20result2=url('http://br-ofertasimperdiveis.epizy.com/?q=select * from x')
char-level/word-level+n-gram+tfidf一把梭下来，可以解决很多问题了，包括较长文本和短文本，而安全中的很多关键信息都可以看成是长文本和短文本，比如域名请求，恶意代码，恶意文件。
https://www.freebuf.com/articles/network/131279.html https://github.com/exp-db/AI-Driven-WAF/blob/master/waf.py https://www.kdnuggets.com/2017/02/machine-learning-driven-firewall.html N-Gram模型理解 总参考连接： https://xz.aliyun.com/t/5288#toc-4